search

BerriAI / litellm

Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]
https://docs.litellm.ai/docs/
Other
13.7k stars 1.61k forks source link

[Bug]: tenacity not included in docker image and required? #2616

Closed kun432 closed 6 months ago

kun432 commented 7 months ago

What happened?

as a proxy using docker compose, seems the latest image does not include tenacity and it gets an error when a request error happen.

litellm_config.yaml:

model_list:
  - model_name: my_completion_model
    litellm_params:
      model: anthropic.claude-3-haiku-20240307-v1:0
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: "ap-northeast-1"  # Haiku not supported in this region yet
litellm_settings:
  num_retries: 2
  set_verbose: True

docker-compose.yml:

version: "3.9"
services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    ports:
      - "8001:8000"
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    command: [ "--config", "/app/config.yaml", "--port", "8000", "--num_workers", "8" ]
    env_file:
      - .env
$ docker compose up
$ curl http://0.0.0.0:8001/chat/completions --header 'Content-Type: application/json' --data @- <<EOF
{
  "model": "anthropic.claude-3-haiku-20240307-v1:0",
  "messages": [{"role": "user", "content": "日本の総理大臣は？"}],
  "max_tokens": 500
}
EOF
{"error":{"message":"tenacity import failed please run `pip install tenacity`. ErrorNo module named 'tenacity'","type":null,"param":null,"code":500}}

log:

litellm-proxy-openai-claude-litellm-1  | INFO:     192.168.32.1:34500 - "POST /chat/completions HTTP/1.1" 500 Internal Server Error

but somehow logs show retrying seems working. I can see the following logs 2 times:

litellm-proxy-openai-claude-litellm-1  | Request Sent from LiteLLM:
litellm-proxy-openai-claude-litellm-1  |
litellm-proxy-openai-claude-litellm-1  |             response = client.invoke_model(
litellm-proxy-openai-claude-litellm-1  |                 body={"messages": [{"role": "user", "content": [{"type": "text", "text": "\u65e5\u672c\u306e\u7dcf\u7406\u5927\u81e3\u306f\uff1f"}]}], "max_tokens": 500, "anthropic_version": "bedrock-2023-05-31"},
litellm-proxy-openai-claude-litellm-1  |                 modelId=anthropic.claude-3-haiku-20240307-v1:0,
litellm-proxy-openai-claude-litellm-1  |                 accept=accept,
litellm-proxy-openai-claude-litellm-1  |                 contentType=contentType
litellm-proxy-openai-claude-litellm-1  |             )

and also this 1 time:

litellm-proxy-openai-claude-litellm-1  | RETRY TRIGGERED!

I don't know if this is a bug or not.

Relevant log output

litellm-proxy-openai-claude-litellm-1  | Request to litellm:
litellm-proxy-openai-claude-litellm-1  | Traceback (most recent call last):
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/llms/bedrock.py", line 894, in completion
litellm-proxy-openai-claude-litellm-1  |     response = client.invoke_model(
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 553, in _api_call
litellm-proxy-openai-claude-litellm-1  |     return self._make_api_call(operation_name, kwargs)
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 1009, in _make_api_call
litellm-proxy-openai-claude-litellm-1  |     raise error_class(parsed_response, operation_name)
litellm-proxy-openai-claude-litellm-1  | botocore.errorfactory.ResourceNotFoundException: An error occurred (ResourceNotFoundException) when calling the InvokeModel operation: Could not resolve the foundation model from the provided model identifier.
litellm-proxy-openai-claude-litellm-1  |
litellm-proxy-openai-claude-litellm-1  | During handling of the above exception, another exception occurred:
litellm-proxy-openai-claude-litellm-1  |
litellm-proxy-openai-claude-litellm-1  | Traceback (most recent call last):
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/main.py", line 1741, in completion
litellm-proxy-openai-claude-litellm-1  |     response = bedrock.completion(
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/llms/bedrock.py", line 1022, in completion
litellm-proxy-openai-claude-litellm-1  |     raise e
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/llms/bedrock.py", line 902, in completion
litellm-proxy-openai-claude-litellm-1  |     raise BedrockError(status_code=500, message=str(e))
litellm-proxy-openai-claude-litellm-1  | litellm.llms.bedrock.BedrockError: An error occurred (ResourceNotFoundException) when calling the InvokeModel operation: Could not resolve the foundation model from the provided model identifier.
litellm-proxy-openai-claude-litellm-1  |
litellm-proxy-openai-claude-litellm-1  | During handling of the above exception, another exception occurred:
litellm-proxy-openai-claude-litellm-1  |
litellm-proxy-openai-claude-litellm-1  | Traceback (most recent call last):
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/utils.py", line 2693, in wrapper
litellm-proxy-openai-claude-litellm-1  |     result = original_function(*args, **kwargs)
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/main.py", line 2058, in completion
litellm-proxy-openai-claude-litellm-1  |     raise exception_type(
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/utils.py", line 8265, in exception_type
litellm-proxy-openai-claude-litellm-1  |     raise e
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/utils.py", line 7377, in exception_type
litellm-proxy-openai-claude-litellm-1  |     raise ServiceUnavailableError(
litellm-proxy-openai-claude-litellm-1  | litellm.exceptions.ServiceUnavailableError: BedrockException - An error occurred (ResourceNotFoundException) when calling the InvokeModel operation: Could not resolve the foundation model from the provided model identifier.
litellm-proxy-openai-claude-litellm-1  |
litellm-proxy-openai-claude-litellm-1  | During handling of the above exception, another exception occurred:
litellm-proxy-openai-claude-litellm-1  |
litellm-proxy-openai-claude-litellm-1  | Traceback (most recent call last):
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/main.py", line 2071, in completion_with_retries
litellm-proxy-openai-claude-litellm-1  |     import tenacity
litellm-proxy-openai-claude-litellm-1  | ModuleNotFoundError: No module named 'tenacity'
litellm-proxy-openai-claude-litellm-1  |
litellm-proxy-openai-claude-litellm-1  | During handling of the above exception, another exception occurred:
litellm-proxy-openai-claude-litellm-1  |
litellm-proxy-openai-claude-litellm-1  | Traceback (most recent call last):
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/main.py", line 288, in acompletion
litellm-proxy-openai-claude-litellm-1  |     response = await loop.run_in_executor(None, func_with_context)  # type: ignore
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
litellm-proxy-openai-claude-litellm-1  |     result = self.fn(*self.args, **self.kwargs)
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/utils.py", line 2763, in wrapper
litellm-proxy-openai-claude-litellm-1  |     return litellm.completion_with_retries(*args, **kwargs)
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/main.py", line 2073, in completion_with_retries
litellm-proxy-openai-claude-litellm-1  |     raise Exception(
litellm-proxy-openai-claude-litellm-1  | Exception: tenacity import failed please run `pip install tenacity`. ErrorNo module named 'tenacity'
litellm-proxy-openai-claude-litellm-1  |
litellm-proxy-openai-claude-litellm-1  | During handling of the above exception, another exception occurred:
litellm-proxy-openai-claude-litellm-1  |
litellm-proxy-openai-claude-litellm-1  | Traceback (most recent call last):
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/proxy/proxy_server.py", line 3055, in chat_completion
litellm-proxy-openai-claude-litellm-1  |     responses = await asyncio.gather(
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/router.py", line 399, in acompletion
litellm-proxy-openai-claude-litellm-1  |     raise e
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/router.py", line 395, in acompletion
litellm-proxy-openai-claude-litellm-1  |     response = await self.async_function_with_fallbacks(**kwargs)
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/router.py", line 1217, in async_function_with_fallbacks
litellm-proxy-openai-claude-litellm-1  |     raise original_exception
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/router.py", line 1140, in async_function_with_fallbacks
litellm-proxy-openai-claude-litellm-1  |     response = await self.async_function_with_retries(*args, **kwargs)
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/router.py", line 1327, in async_function_with_retries
litellm-proxy-openai-claude-litellm-1  |     raise original_exception
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/router.py", line 1234, in async_function_with_retries
litellm-proxy-openai-claude-litellm-1  |     response = await original_function(*args, **kwargs)
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/router.py", line 482, in _acompletion
litellm-proxy-openai-claude-litellm-1  |     raise e
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/router.py", line 458, in _acompletion
litellm-proxy-openai-claude-litellm-1  |     response = await litellm.acompletion(
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/utils.py", line 3248, in wrapper_async
litellm-proxy-openai-claude-litellm-1  |     raise e
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/utils.py", line 3084, in wrapper_async
litellm-proxy-openai-claude-litellm-1  |     result = await original_function(*args, **kwargs)
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/main.py", line 296, in acompletion
litellm-proxy-openai-claude-litellm-1  |     raise exception_type(
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/utils.py", line 8265, in exception_type
litellm-proxy-openai-claude-litellm-1  |     raise e
litellm-proxy-openai-claude-litellm-1  |   File "/usr/local/lib/python3.9/site-packages/litellm/utils.py", line 8240, in exception_type
litellm-proxy-openai-claude-litellm-1  |     raise APIConnectionError(
litellm-proxy-openai-claude-litellm-1  | litellm.exceptions.APIConnectionError: tenacity import failed please run `pip install tenacity`. ErrorNo module named 'tenacity'

Twitter / LinkedIn details

No response

ishaan-jaff commented 7 months ago

looking into this

ishaan-jaff commented 7 months ago

hmm I don't see the error on my local machine

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/litellm/main.py", line 289, in acompletion
    response = await loop.run_in_executor(None, func_with_context)  # type: ignore
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/site-packages/litellm/utils.py", line 2732, in wrapper
    return litellm.completion_with_retries(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/litellm/main.py", line 2091, in completion_with_retries
    return retryer(original_function, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 379, in __call__
    do = self.iter(retry_state=retry_state)
  File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 325, in iter
    raise retry_exc.reraise()
  File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 158, in reraise
    raise self.last_attempt.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 382, in __call__
    result = fn(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/litellm/utils.py", line 2761, in wrapper
    raise e
  File "/usr/local/lib/python3.9/site-packages/litellm/utils.py", line 2662, in wrapper
    result = original_function(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/litellm/main.py", line 2059, in completion
    raise exception_type(
  File "/usr/local/lib/python3.9/site-packages/litellm/utils.py", line 8229, in exception_type
    raise e
  File "/usr/local/lib/python3.9/site-packages/litellm/utils.py", line 7341, in exception_type
    raise ServiceUnavailableError(
litellm.exceptions.ServiceUnavailableError: BedrockException - An error occurred (ResourceNotFoundException) when calling the InvokeModel operation: Could not resolve the foundation model from the provided model identifier.
ishaan-jaff commented 7 months ago

Okay I'm able to repro this on a new server

krrishdholakia commented 7 months ago

I believe this is because the completion call goes through completion_with_retries.

Is this necessary since the router has it's own retry logic? @ishaan-jaff

ishaan-jaff commented 7 months ago

Fix - it should not even go through this logic when on the router/proxy: https://github.com/BerriAI/litellm/pull/2620

ishaan-jaff commented 7 months ago

@kun432 we'd love to get on a call and learn how we can improve litellm proxy for you. Any chance you're free sometime this week? Sharing my cal for your convenience: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Here's my linkedin if you want to message https://www.linkedin.com/in/reffajnaahsi/

kun432 commented 7 months ago

thank you for fix! and sorry for late reply.

this happened only once after server started and seems not a big deal in my end, but I found and just reported. Tried current version, confirmed this error handled correctly like:

{"error":{"message":"BedrockException - BedrockException - An error occurred (ResourceNotFoundException) when calling the InvokeModel operation: Could not resolve the foundation model from the provided model identifier.","type":null,"param":null,"code":500}}

thanks.