Python SDK, Proxy Server to call 100+ LLM APIs using the OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]
When making a call to a gemini model through vertex ai with tools and the returned tools args contains a list the type pf the list is a ReapetedComposite with is part of the proto types and isn't json serializable.
Traceback (most recent call last):
File "/litellm/litellm/proxy/proxy_server.py", line 3668, in chat_completion
responses = await asyncio.gather(
File "/litellm/litellm/router.py", line 470, in acompletion
raise e
File "/litellm/litellm/router.py", line 466, in acompletion
response = await self.async_function_with_fallbacks(**kwargs)
File "/litellm/litellm/router.py", line 1442, in async_function_with_fallbacks
raise original_exception
File "/litellm/litellm/router.py", line 1360, in async_function_with_fallbacks
response = await self.async_function_with_retries(*args, **kwargs)
File "litellm/litellm/router.py", line 1524, in async_function_with_retries
raise original_exception
File "/litellm/litellm/router.py", line 1461, in async_function_with_retries
response = await original_function(*args, **kwargs)
File "/litellm/litellm/router.py", line 589, in _acompletion
raise e
File "/litellm/litellm/router.py", line 573, in _acompletion
response = await _response
File "/litellm/litellm/utils.py", line 3708, in wrapper_async
raise e
File "/litellm/litellm/utils.py", line 3536, in wrapper_async
result = await original_function(*args, **kwargs)
File "/litellm/litellm/main.py", line 345, in acompletion
raise exception_type(
File "/litellm/litellm/utils.py", line 9233, in exception_type
raise e
File "/litellm/litellm/utils.py", line 8461, in exception_type
raise APIError(
litellm.exceptions.APIError: VertexAIException - Object of type RepeatedComposite is not JSON serializable
What happened?
When making a call to a gemini model through vertex ai with tools and the returned tools args contains a list the type pf the list is a ReapetedComposite with is part of the proto types and isn't json serializable.
Looks like the problem is in the way the returned message is serialized to json here: https://github.com/BerriAI/litellm/blob/main/litellm/llms/vertex_ai.py#L872-L874
Relevant log output
Twitter / LinkedIn details
No response