Closed dhruv-anand-aintech closed 6 months ago
@dhruv-anand-aintech i believe this is already supported. what error do you see?
here's the class i wrote for it - https://github.com/BerriAI/litellm/blob/e1e3a9797daf229cf8dbccd2ed68cf63892f80ab/litellm/main.py#L102
thanks for pointing me to that. it worked for the gpt 3.5 case. I was trying with a different model:
import instructor
from litellm import LiteLLM
import litellm
client = instructor.patch(LiteLLM())
litellm.set_verbose=True
model_to_be_used = "openrouter/mistralai/mistral-7b-instruct:free"
if not litellm.supports_function_calling(model=model_to_be_used):
litellm.add_function_to_prompt = True
user = client.chat.completions.create(
model=model_to_be_used,
# model="gpt-4-turbo-preview", "gpt-3.5-turbo"
response_model=UserDetail,
messages=[
{"role": "user", "content": f"{input_prompt}"},
],
)
and got this error:
File [~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:593](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:593), in patch.<locals>.new_create_sync(response_model, validation_context, max_retries, *args, **kwargs)
[582](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:582) @wraps(func)
[583](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:583) def new_create_sync(
[584](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:584) response_model: Type[T_Model] = None,
(...)
[588](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:588) **kwargs: T_ParamSpec.kwargs,
[589](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:589) ) -> T_Model:
[590](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:590) response_model, new_kwargs = handle_response_model(
[591](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:591) response_model=response_model, mode=mode, **kwargs
[592](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:592) )
--> [593](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:593) response = retry_sync(
[594](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:594) func=func,
[595](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:595) response_model=response_model,
[596](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:596) validation_context=validation_context,
[597](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:597) max_retries=max_retries,
[598](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:598) args=args,
[599](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:599) kwargs=new_kwargs,
[600](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:600) mode=mode,
[601](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:601) )
[602](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:602) return response
File [~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:408](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:408), in retry_sync(func, response_model, validation_context, args, kwargs, max_retries, strict, mode)
[405](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:405) raise ValueError("max_retries must be an int or a `tenacity.Retrying` object")
[407](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:407) try:
--> [408](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:408) for attempt in max_retries:
[409](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:409) with attempt:
[410](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:410) try:
File [~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:347](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:347), in BaseRetrying.__iter__(self)
[345](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:345) retry_state = RetryCallState(self, fn=None, args=(), kwargs={})
[346](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:346) while True:
--> [347](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:347) do = self.iter(retry_state=retry_state)
[348](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:348) if isinstance(do, DoAttempt):
[349](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:349) yield AttemptManager(retry_state=retry_state)
File [~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:325](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:325), in BaseRetrying.iter(self, retry_state)
[323](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:323) retry_exc = self.retry_error_cls(fut)
[324](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:324) if self.reraise:
--> [325](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:325) raise retry_exc.reraise()
[326](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:326) raise retry_exc from fut.exception()
[328](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:328) if self.wait:
File [~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:158](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:158), in RetryError.reraise(self)
[156](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:156) def reraise(self) -> t.NoReturn:
[157](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:157) if self.last_attempt.failed:
--> [158](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:158) raise self.last_attempt.result()
[159](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:159) raise self
File [~/miniforge3/lib/python3.10/concurrent/futures/_base.py:451](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:451), in Future.result(self, timeout)
[449](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:449) raise CancelledError()
[450](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:450) elif self._state == FINISHED:
--> [451](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:451) return self.__get_result()
[453](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:453) self._condition.wait(timeout)
[455](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:455) if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:
File [~/miniforge3/lib/python3.10/concurrent/futures/_base.py:403](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:403), in Future.__get_result(self)
[401](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:401) if self._exception:
[402](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:402) try:
--> [403](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:403) raise self._exception
[404](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:404) finally:
[405](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:405) # Break a reference cycle with the exception in self._exception
[406](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:406) self = None
File [~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:411](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:411), in retry_sync(func, response_model, validation_context, args, kwargs, max_retries, strict, mode)
[409](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:409) with attempt:
[410](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:410) try:
--> [411](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:411) response = func(*args, **kwargs)
[412](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:412) stream = kwargs.get("stream", False)
[413](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:413) if (
[414](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:414) isinstance(response, ChatCompletion)
[415](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:415) and response.usage is not None
[416](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:416) ):
File [~/miniforge3/lib/python3.10/site-packages/litellm/main.py:131](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:131), in Completions.create(self, messages, model, **kwargs)
[129](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:129) self.params[k] = v
[130](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:130) model = model or self.params.get("model")
--> [131](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:131) response = completion(model=model, messages=messages, **self.params)
[132](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:132) return response
File [~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2625](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2625), in client.<locals>.wrapper(*args, **kwargs)
[2621](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2621) if (
[2622](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2622) liteDebuggerClient and liteDebuggerClient.dashboard_url != None
[2623](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2623) ): # make it easy to get to the debugger logs if you've initialized it
[2624](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2624) e.message += f"\n Check the log in your dashboard - {liteDebuggerClient.dashboard_url}"
-> [2625](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2625) raise e
File [~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2528](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2528), in client.<locals>.wrapper(*args, **kwargs)
[2526](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2526) print_verbose(f"Error while checking max token limit: {str(e)}")
[2527](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2527) # MODEL CALL
-> [2528](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2528) result = original_function(*args, **kwargs)
[2529](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2529) end_time = datetime.datetime.now()
[2530](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2530) if "stream" in kwargs and kwargs["stream"] == True:
File [~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1936](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1936), in completion(model, messages, timeout, temperature, top_p, n, stream, stop, max_tokens, presence_penalty, frequency_penalty, logit_bias, user, response_format, seed, tools, tool_choice, logprobs, top_logprobs, deployment_id, extra_headers, functions, function_call, base_url, api_version, api_key, model_list, **kwargs)
[1933](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1933) return response
[1934](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1934) except Exception as e:
[1935](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1935) ## Map to OpenAI Exception
-> [1936](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1936) raise exception_type(
[1937](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1937) model=model,
[1938](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1938) custom_llm_provider=custom_llm_provider,
[1939](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1939) original_exception=e,
[1940](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1940) completion_kwargs=args,
[1941](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1941) )
File [~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7801](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7801), in exception_type(model, original_exception, custom_llm_provider, completion_kwargs)
[7799](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7799) # don't let an error with mapping interrupt the user from receiving an error from the llm api calls
[7800](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7800) if exception_mapping_worked:
-> [7801](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7801) raise e
[7802](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7802) else:
[7803](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7803) raise original_exception
File [~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7776](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7776), in exception_type(model, original_exception, custom_llm_provider, completion_kwargs)
[7769](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7769) raise APIConnectionError(
[7770](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7770) message=f"{str(original_exception)}",
[7771](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7771) llm_provider=custom_llm_provider,
[7772](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7772) model=model,
[7773](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7773) request=original_exception.request,
[7774](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7774) )
[7775](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7775) else:
-> [7776](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7776) raise APIConnectionError(
[7777](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7777) message=f"{str(original_exception)}",
[7778](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7778) llm_provider=custom_llm_provider,
[7779](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7779) model=model,
[7780](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7780) request=httpx.Request(
[7781](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7781) method="POST", url="https://api.openai.com/v1/"
[7782](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7782) ), # stub the request
[7783](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7783) )
[7784](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7784) except Exception as e:
[7785](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7785) # LOGGING
[7786](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7786) exception_logging(
[7787](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7787) logger_fn=user_logger_fn,
[7788](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7788) additional_args={
(...)
[7792](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7792) exception=e,
[7793](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7793) )
APIConnectionError: 'functions'
Could you check what's going on?
openrouter doesn't have a function calling api as far as i know. the error seems to be that no function is in the response which looks correct.
I have set litellm.add_function_to_prompt = True
as is suggested when a non function calling model is used
yes - and that adds the function in the prompt as a normal message. It'll then return the response in the content.
I guess it's better to describe it as a hack to try and prompt the model to respond with a function call, but it can't be guaranteed. I think we can remove the recommendation in the error message, to prevent this confusion.
Closing the issue for now - but @dhruv-anand-aintech let me know if there's something we can do to help here
I guess Instructor library nukes the other response fields (like token usage) leading to completion_cost not being usable:
{
"name": "AttributeError",
"message": "'InvoiceDetail' object has no attribute 'get'",
"stack": "---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[17], line 2
1 import litellm
----> 2 litellm.completion_cost(invoice_details)
File /opt/homebrew/anaconda3/lib/python3.9/site-packages/litellm/utils.py:3755, in completion_cost(completion_response, model, prompt, messages, completion, total_time, call_type, custom_llm_provider, region_name, size, quality, n)
3753 return _final_cost
3754 except Exception as e:
-> 3755 raise e
File /opt/homebrew/anaconda3/lib/python3.9/site-packages/litellm/utils.py:3646, in completion_cost(completion_response, model, prompt, messages, completion, total_time, call_type, custom_llm_provider, region_name, size, quality, n)
3643 custom_llm_provider = None
3644 if completion_response is not None:
3645 # get input/output tokens from completion_response
-> 3646 prompt_tokens = completion_response.get(\"usage\", {}).get(\"prompt_tokens\", 0)
3647 completion_tokens = completion_response.get(\"usage\", {}).get(
3648 \"completion_tokens\", 0
3649 )
3650 total_time = completion_response.get(\"_response_ms\", 0)
File /opt/homebrew/anaconda3/lib/python3.9/site-packages/pydantic/main.py:767, in BaseModel.__getattr__(self, item)
764 return super().__getattribute__(item) # Raises AttributeError if appropriate
765 else:
766 # this is the current error
--> 767 raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}')
AttributeError: 'InvoiceDetail' object has no attribute 'get'"
}
@jxnl Is it possible to keep these in a separate field in the output of client.chat.completions.create()?
That's awesome. Let me check if that works with completion_cost.
It works!
@krrishdholakia
In this case, I find that my litellm.api_key = ...
not work. I guess this is because the LiteLMM()
call instantiated an object that has no relation to any changes that are made to litellm
afterwards.
If my code relies on these kind of changes, like litellm.success_callbacks
and litellm.api_key
- how would you operate those together with the instructor
patching?
The Feature
I'd like to be able to use the Instructor library for function calling using pydantic classes for any LLM in LiteLLM.
I haven't been able to figure out how to patch that library onto litellm as there is no client object to work with.
https://github.com/jxnl/instructor/pull/116
Motivation, pitch
Function calling is not supported by many more LLMs than when Instructor launched. It makes it much easier to use function calling (by just defining a pydantic class for the expected extraction object).
Twitter / LinkedIn details
https://twitter.com/dhruv___anand