[Bug]: watsonx embeddings don't seem to work with litellm proxy

[alexvanolst]

What happened?

When I try to use a watsonx embedding model with the proxy e.g.

model_list:
  - model_name: slate-125m-english-rtrvr
    litellm_params:
      model: watsonx/ibm/slate-125m-english-rtrvr
      api_key: "os.environ/WATSONX_API_KEY"
litellm_settings:
  drop_params: true

I get an Internal Server Error:

 curl localhost:4000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "input": "Your text string goes here",
    "model": "slate-125m-english-rtrvr"
  }'
Internal Server Error

Relevant log output

INFO:     172.17.0.1:58590 - "POST /chat/completions HTTP/1.1" 200 OK
INFO:     172.17.0.1:58598 - "POST /embeddings HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/fastapi/encoders.py", line 324, in jsonable_encoder
    data = dict(obj)
           ^^^^^^^^^
TypeError: 'coroutine' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/fastapi/encoders.py", line 329, in jsonable_encoder
    data = vars(obj)
           ^^^^^^^^^
TypeError: vars() argument must have __dict__ attribute

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 411, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 296, in app
    content = await serialize_response(
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 180, in serialize_response
    return jsonable_encoder(response_content)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/encoders.py", line 332, in jsonable_encoder
    raise ValueError(errors) from e
ValueError: [TypeError("'coroutine' object is not iterable"), TypeError('vars() argument must have __dict__ attribute')]
14:21:10 - LiteLLM Router:ERROR: router.py:2782 - litellm.proxy.hooks.prompt_injection_detection.py::async_pre_call_hook(): Exception occured - 'coroutine' object is not subscriptable
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 2760, in deployment_callback_on_success
    total_tokens = completion_response["usage"]["total_tokens"]
                   ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
TypeError: 'coroutine' object is not subscriptable

Twitter / LinkedIn details

No response

[polubarev]

Found bug and reported it here https://github.com/BerriAI/litellm/issues/4538

[simonsanvil]

This was fixed in #4586, we can probably close this issue now. @alexvanolst you should target version 1.41.12 at minimum.

cc/ @krrishdholakia @ishaan-jaff