search

BerriAI / litellm

Python SDK, Proxy Server to call 100+ LLM APIs using the OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]
https://docs.litellm.ai/docs/
Other
12.03k stars 1.39k forks source link

[Feature]: Instructor library support #2308

Closed dhruv-anand-aintech closed 6 months ago

dhruv-anand-aintech commented 6 months ago

The Feature

I'd like to be able to use the Instructor library for function calling using pydantic classes for any LLM in LiteLLM.

I haven't been able to figure out how to patch that library onto litellm as there is no client object to work with.

https://github.com/jxnl/instructor/pull/116

Motivation, pitch

Function calling is not supported by many more LLMs than when Instructor launched. It makes it much easier to use function calling (by just defining a pydantic class for the expected extraction object).

Twitter / LinkedIn details

https://twitter.com/dhruv___anand

krrishdholakia commented 6 months ago

@dhruv-anand-aintech i believe this is already supported. what error do you see?

here's the class i wrote for it - https://github.com/BerriAI/litellm/blob/e1e3a9797daf229cf8dbccd2ed68cf63892f80ab/litellm/main.py#L102

dhruv-anand-aintech commented 6 months ago

thanks for pointing me to that. it worked for the gpt 3.5 case. I was trying with a different model:

import instructor
from litellm import LiteLLM
import litellm
client = instructor.patch(LiteLLM())

litellm.set_verbose=True
model_to_be_used = "openrouter/mistralai/mistral-7b-instruct:free"
if not litellm.supports_function_calling(model=model_to_be_used):
    litellm.add_function_to_prompt = True
user = client.chat.completions.create(
    model=model_to_be_used,
    # model="gpt-4-turbo-preview", "gpt-3.5-turbo"
    response_model=UserDetail,
    messages=[
        {"role": "user", "content": f"{input_prompt}"},
    ],
)

and got this error:

File [~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:593](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:593), in patch.<locals>.new_create_sync(response_model, validation_context, max_retries, *args, **kwargs)
    [582](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:582) @wraps(func)
    [583](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:583) def new_create_sync(
    [584](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:584)     response_model: Type[T_Model] = None,
   (...)
    [588](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:588)     **kwargs: T_ParamSpec.kwargs,
    [589](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:589) ) -> T_Model:
    [590](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:590)     response_model, new_kwargs = handle_response_model(
    [591](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:591)         response_model=response_model, mode=mode, **kwargs
    [592](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:592)     )
--> [593](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:593)     response = retry_sync(
    [594](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:594)         func=func,
    [595](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:595)         response_model=response_model,
    [596](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:596)         validation_context=validation_context,
    [597](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:597)         max_retries=max_retries,
    [598](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:598)         args=args,
    [599](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:599)         kwargs=new_kwargs,
    [600](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:600)         mode=mode,
    [601](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:601)     )
    [602](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:602)     return response

File [~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:408](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:408), in retry_sync(func, response_model, validation_context, args, kwargs, max_retries, strict, mode)
    [405](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:405)     raise ValueError("max_retries must be an int or a `tenacity.Retrying` object")
    [407](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:407) try:
--> [408](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:408)     for attempt in max_retries:
    [409](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:409)         with attempt:
    [410](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:410)             try:

File [~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:347](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:347), in BaseRetrying.__iter__(self)
    [345](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:345) retry_state = RetryCallState(self, fn=None, args=(), kwargs={})
    [346](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:346) while True:
--> [347](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:347)     do = self.iter(retry_state=retry_state)
    [348](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:348)     if isinstance(do, DoAttempt):
    [349](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:349)         yield AttemptManager(retry_state=retry_state)

File [~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:325](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:325), in BaseRetrying.iter(self, retry_state)
    [323](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:323)     retry_exc = self.retry_error_cls(fut)
    [324](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:324)     if self.reraise:
--> [325](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:325)         raise retry_exc.reraise()
    [326](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:326)     raise retry_exc from fut.exception()
    [328](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:328) if self.wait:

File [~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:158](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:158), in RetryError.reraise(self)
    [156](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:156) def reraise(self) -> t.NoReturn:
    [157](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:157)     if self.last_attempt.failed:
--> [158](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:158)         raise self.last_attempt.result()
    [159](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/tenacity/__init__.py:159)     raise self

File [~/miniforge3/lib/python3.10/concurrent/futures/_base.py:451](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:451), in Future.result(self, timeout)
    [449](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:449)     raise CancelledError()
    [450](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:450) elif self._state == FINISHED:
--> [451](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:451)     return self.__get_result()
    [453](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:453) self._condition.wait(timeout)
    [455](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:455) if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:

File [~/miniforge3/lib/python3.10/concurrent/futures/_base.py:403](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:403), in Future.__get_result(self)
    [401](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:401) if self._exception:
    [402](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:402)     try:
--> [403](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:403)         raise self._exception
    [404](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:404)     finally:
    [405](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:405)         # Break a reference cycle with the exception in self._exception
    [406](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/concurrent/futures/_base.py:406)         self = None

File [~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:411](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:411), in retry_sync(func, response_model, validation_context, args, kwargs, max_retries, strict, mode)
    [409](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:409) with attempt:
    [410](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:410)     try:
--> [411](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:411)         response = func(*args, **kwargs)
    [412](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:412)         stream = kwargs.get("stream", False)
    [413](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:413)         if (
    [414](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:414)             isinstance(response, ChatCompletion)
    [415](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:415)             and response.usage is not None
    [416](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/instructor/patch.py:416)         ):

File [~/miniforge3/lib/python3.10/site-packages/litellm/main.py:131](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:131), in Completions.create(self, messages, model, **kwargs)
    [129](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:129)     self.params[k] = v
    [130](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:130) model = model or self.params.get("model")
--> [131](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:131) response = completion(model=model, messages=messages, **self.params)
    [132](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:132) return response

File [~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2625](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2625), in client.<locals>.wrapper(*args, **kwargs)
   [2621](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2621)         if (
   [2622](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2622)             liteDebuggerClient and liteDebuggerClient.dashboard_url != None
   [2623](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2623)         ):  # make it easy to get to the debugger logs if you've initialized it
   [2624](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2624)             e.message += f"\n Check the log in your dashboard - {liteDebuggerClient.dashboard_url}"
-> [2625](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2625) raise e

File [~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2528](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2528), in client.<locals>.wrapper(*args, **kwargs)
   [2526](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2526)         print_verbose(f"Error while checking max token limit: {str(e)}")
   [2527](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2527) # MODEL CALL
-> [2528](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2528) result = original_function(*args, **kwargs)
   [2529](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2529) end_time = datetime.datetime.now()
   [2530](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:2530) if "stream" in kwargs and kwargs["stream"] == True:

File [~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1936](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1936), in completion(model, messages, timeout, temperature, top_p, n, stream, stop, max_tokens, presence_penalty, frequency_penalty, logit_bias, user, response_format, seed, tools, tool_choice, logprobs, top_logprobs, deployment_id, extra_headers, functions, function_call, base_url, api_version, api_key, model_list, **kwargs)
   [1933](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1933)     return response
   [1934](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1934) except Exception as e:
   [1935](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1935)     ## Map to OpenAI Exception
-> [1936](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1936)     raise exception_type(
   [1937](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1937)         model=model,
   [1938](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1938)         custom_llm_provider=custom_llm_provider,
   [1939](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1939)         original_exception=e,
   [1940](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1940)         completion_kwargs=args,
   [1941](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/main.py:1941)     )

File [~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7801](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7801), in exception_type(model, original_exception, custom_llm_provider, completion_kwargs)
   [7799](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7799) # don't let an error with mapping interrupt the user from receiving an error from the llm api calls
   [7800](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7800) if exception_mapping_worked:
-> [7801](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7801)     raise e
   [7802](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7802) else:
   [7803](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7803)     raise original_exception

File [~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7776](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7776), in exception_type(model, original_exception, custom_llm_provider, completion_kwargs)
   [7769](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7769)             raise APIConnectionError(
   [7770](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7770)                 message=f"{str(original_exception)}",
   [7771](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7771)                 llm_provider=custom_llm_provider,
   [7772](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7772)                 model=model,
   [7773](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7773)                 request=original_exception.request,
   [7774](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7774)             )
   [7775](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7775)         else:
-> [7776](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7776)             raise APIConnectionError(
   [7777](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7777)                 message=f"{str(original_exception)}",
   [7778](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7778)                 llm_provider=custom_llm_provider,
   [7779](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7779)                 model=model,
   [7780](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7780)                 request=httpx.Request(
   [7781](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7781)                     method="POST", url="https://api.openai.com/v1/"
   [7782](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7782)                 ),  # stub the request
   [7783](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7783)             )
   [7784](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7784) except Exception as e:
   [7785](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7785)     # LOGGING
   [7786](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7786)     exception_logging(
   [7787](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7787)         logger_fn=user_logger_fn,
   [7788](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7788)         additional_args={
   (...)
   [7792](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7792)         exception=e,
   [7793](https://file+.vscode-resource.vscode-cdn.net/Users/dhruvanand/Code/usergpt/src/~/miniforge3/lib/python3.10/site-packages/litellm/utils.py:7793)     )

APIConnectionError: 'functions'

Could you check what's going on?

krrishdholakia commented 6 months ago

openrouter doesn't have a function calling api as far as i know. the error seems to be that no function is in the response which looks correct.

dhruv-anand-aintech commented 6 months ago

I have set litellm.add_function_to_prompt = True as is suggested when a non function calling model is used

krrishdholakia commented 6 months ago

yes - and that adds the function in the prompt as a normal message. It'll then return the response in the content.

I guess it's better to describe it as a hack to try and prompt the model to respond with a function call, but it can't be guaranteed. I think we can remove the recommendation in the error message, to prevent this confusion.

krrishdholakia commented 6 months ago

Closing the issue for now - but @dhruv-anand-aintech let me know if there's something we can do to help here

dhruv-anand-aintech commented 6 months ago

I guess Instructor library nukes the other response fields (like token usage) leading to completion_cost not being usable:

{
    "name": "AttributeError",
    "message": "'InvoiceDetail' object has no attribute 'get'",
    "stack": "---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[17], line 2
      1 import litellm
----> 2 litellm.completion_cost(invoice_details)

File /opt/homebrew/anaconda3/lib/python3.9/site-packages/litellm/utils.py:3755, in completion_cost(completion_response, model, prompt, messages, completion, total_time, call_type, custom_llm_provider, region_name, size, quality, n)
   3753     return _final_cost
   3754 except Exception as e:
-> 3755     raise e

File /opt/homebrew/anaconda3/lib/python3.9/site-packages/litellm/utils.py:3646, in completion_cost(completion_response, model, prompt, messages, completion, total_time, call_type, custom_llm_provider, region_name, size, quality, n)
   3643 custom_llm_provider = None
   3644 if completion_response is not None:
   3645     # get input/output tokens from completion_response
-> 3646     prompt_tokens = completion_response.get(\"usage\", {}).get(\"prompt_tokens\", 0)
   3647     completion_tokens = completion_response.get(\"usage\", {}).get(
   3648         \"completion_tokens\", 0
   3649     )
   3650     total_time = completion_response.get(\"_response_ms\", 0)

File /opt/homebrew/anaconda3/lib/python3.9/site-packages/pydantic/main.py:767, in BaseModel.__getattr__(self, item)
    764     return super().__getattribute__(item)  # Raises AttributeError if appropriate
    765 else:
    766     # this is the current error
--> 767     raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}')

AttributeError: 'InvoiceDetail' object has no attribute 'get'"
}

@jxnl Is it possible to keep these in a separate field in the output of client.chat.completions.create()?

jxnl commented 6 months ago

https://jxnl.github.io/instructor/concepts/raw_response/ ?

dhruv-anand-aintech commented 6 months ago

That's awesome. Let me check if that works with completion_cost.

dhruv-anand-aintech commented 6 months ago

It works!

eliorc commented 5 months ago

@krrishdholakia

In this case, I find that my litellm.api_key = ... not work. I guess this is because the LiteLMM() call instantiated an object that has no relation to any changes that are made to litellm afterwards.

If my code relies on these kind of changes, like litellm.success_callbacks and litellm.api_key - how would you operate those together with the instructor patching?