[Bug]: Completion breaks if response.model is None

josejg commented 2 months ago

What happened?

The following check breaks a custom endpoint I am working with that returns model=None in the response. The codepath seems related to cost, which should gracefully fail if there is insufficient information

https://github.com/BerriAI/litellm/blob/main/litellm/utils.py#L4040

The call looks like

from litellm import completion
x = completion(
    model='mymodel',
    custom_llm_provider="openai",
    messages=[{"role": "user", "content": "Hello!"}],
    api_base='https://my-endpoint',
    api_key='my-api-key',
)

Relevant log output

Exception in thread Thread-6 (success_handler):
Traceback (most recent call last):
  File "/opt/env/lib/python3.11/site-packages/litellm/utils.py", line 1326, in _success_handler_helper_fn
    litellm.completion_cost(
  File "/opt/env/lib/python3.11/site-packages/litellm/utils.py", line 4113, in completion_cost
    raise e
  File "/opt/env/lib/python3.11/site-packages/litellm/utils.py", line 4039, in completion_cost
    raise ValueError(
ValueError: Model is None and does not exist in passed completion_response. Passed completion_response=ModelResponse(id='chatcmpl-e126c39bfd7348d1ad51b9b4d4939377', choices=[Choices(finish_reason='stop', index=0, message=Message(content="Hello! How can I assist you today? I'm here to help with a variety of tasks, such as writing, coding, and providing information on a wide range of topics. I can also provide balanced perspectives on controversial issues. Let me know how I can best assist you.", role='assistant'))], created=1712886969, model=None, object='chat.completion', system_fingerprint=None, usage=Usage(completion_tokens=58, prompt_tokens=187, total_tokens=245)), model=None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/env/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/opt/env/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/env/lib/python3.11/site-packages/litellm/utils.py", line 1363, in success_handler
    start_time, end_time, result = self._success_handler_helper_fn(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/env/lib/python3.11/site-packages/litellm/utils.py", line 1357, in _success_handler_helper_fn
    raise Exception(f"[Non-Blocking] LiteLLM.Success_Call Error: {str(e)}")
Exception: [Non-Blocking] LiteLLM.Success_Call Error: Model is None and does not exist in passed completion_response. Passed completion_response=ModelResponse(id='chatcmpl-e126c39bfd7348d1ad51b9b4d4939377', choices=[Choices(finish_reason='stop', index=0, message=Message(content="Hello! How can I assist you today? I'm here to help with a variety of tasks, such as writing, coding, and providing information on a wide range of topics. I can also provide balanced perspectives on controversial issues. Let me know how I can best assist you.", role='assistant'))], created=1712886969, model=None, object='chat.completion', system_fingerprint=None, usage=Usage(completion_tokens=58, prompt_tokens=187, total_tokens=245)), model=None

Twitter / LinkedIn details

No response

Manouchehri commented 2 months ago

Related to https://github.com/BerriAI/litellm/issues/1881, probably the same fix needed for both tickes.

krrishdholakia commented 2 months ago

Agreed @josejg - will revert once fixed. Aiming to pick this up today/tomorrow.

krrishdholakia commented 2 months ago

Hey @josejg, unable to repro this. I see a non-blocking error in my console, but it's in a separate thread (doesn't block the actual call).

sdk

proxy

Closing as unable to repro. Please bump me @josejg if there's a way to repro the call failing due to this.

Even in your log output i see - .. Exception: [Non-Blocking]

Happy to hop on a call to debug further - https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

BerriAI / litellm