Closed Rizaldy closed 10 months ago
Tried to create new files following the documentation with this code
import openai
openai.api_key = "any-string-here"
openai.api_base = "http://0.0.0.0:8000" # your proxy url
# call openai
response = openai.ChatCompletion.create(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hey"}])
print(response)
and the error is below
(litellm_env) aldi@Rizaldy-MacBook-Pro litellm % /Users/aldi/miniconda3/envs/litellm_env/bin/python /Users/aldi/Documents/Python/projects-3/litellm
/test.py
Traceback (most recent call last):
File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/openai/api_requestor.py", line 413, in handle_error_response
error_data = resp["error"]
~~~~^^^^^^^^^
TypeError: string indices must be integers, not 'str'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/aldi/Documents/Python/projects-3/litellm/test.py", line 7, in <module>
response = openai.ChatCompletion.create(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hey"}])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/openai/api_resources/chat_completion.py", line 25, in create
return super().create(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 155, in create
response, _, api_key = requestor.request(
^^^^^^^^^^^^^^^^^^
File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/openai/api_requestor.py", line 299, in request
resp, got_stream = self._interpret_response(result, stream)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/openai/api_requestor.py", line 710, in _interpret_response
self._interpret_response_line(
File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/openai/api_requestor.py", line 775, in _interpret_response_line
raise self.handle_error_response(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/openai/api_requestor.py", line 415, in handle_error_response
raise error.APIError(
openai.error.APIError: Invalid response object from API: 'Internal Server Error' (HTTP response code was 500)
@Rizaldy the last error you reported I was having when I was passing "NONE" as the API key.
When I replaced it for "hi", it worked.
But I ended up with the same "AttributeError: 'Router' object has no attribute 'model_names'" problem, which I'm currently running into.
@Rizaldy forget it. The error keeps coming while running litellm as a proxy server.
hey @elhombrejd @Rizaldy can you run pip install --upgrade litellm
This should be resolved in the latest version of litellm.
Please re-open if you still see the issue in v0.11.1
@krrishdholakia I think am having this issue with v0.11.1 I am trying to run local ollama with litellm.
(autogen) a:other a$ pip show litellm
Name: litellm
Version: 0.11.1
(autogen) a :other a$ litellm --model ollama/mistral --api_base http://localhost:11434 --temperature 0.3 --max_tokens 2048
#------------------------------------------------------------#
# #
# 'I don't like how this works...' #
# https://github.com/BerriAI/litellm/issues/new #
# #
#------------------------------------------------------------#
Thank you for using LiteLLM! - Krrish & Ishaan
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
Docs: https://docs.litellm.ai/docs/proxy_server
ollama called
LiteLLM: Test your local endpoint with: "litellm --test" [In a new terminal tab]
LiteLLM: View available endpoints for this server on: http://0.0.0.0:8000
LiteLLM: Self-host your proxy using the following: https://docs.litellm.ai/docs/proxy_server#deploy-proxy
`INFO: Started server process [55605]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
'Router' object has no attribute 'model_names'
LiteLLM.Exception: 'Router' object has no attribute 'model_names'
INFO: 127.0.0.1:53025 - "POST /chat/completions HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/litellm/proxy/llm.py", line 138, in litellm_completion
if model_router and data["model"] in model_router.get_model_names():
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/litellm/router.py", line 100, in get_model_names
return self.model_names
AttributeError: 'Router' object has no attribute 'model_names'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
return await self.app(scope, receive, send)
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/fastapi/applications.py", line 1115, in __call__
await super().__call__(scope, receive, send)
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
await self.middleware_stack(scope, receive, send)
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
raise exc
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
await self.app(scope, receive, _send)
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/starlette/middleware/cors.py", line 83, in __call__
await self.app(scope, receive, send)
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
raise exc
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
await self.app(scope, receive, sender)
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
raise e
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
await self.app(scope, receive, send)
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
await route.handle(scope, receive, send)
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
response = await func(request)
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/fastapi/routing.py", line 274, in app
raw_response = await run_endpoint_function(
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/litellm/proxy/proxy_server.py", line 515, in chat_completion
return litellm_completion(data, type="chat_completion", user_model=user_model,
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry
ret = target(*args, **kwargs)
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry
ret = target(*args, **kwargs)
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/litellm/proxy/llm.py", line 148, in litellm_completion
handle_llm_exception(e=e, user_api_base=user_api_base)
File "/Users/opt/anaconda3/envs/autogen/lib/python3.10/site-packages/litellm/proxy/llm.py", line 92, in handle_llm_exception
raise UnknownLLMError from e
litellm.proxy.llm.UnknownLLMError
hey @Rizaldy @kim333 thanks for raising this. I just pushed the fix here - https://github.com/BerriAI/litellm/commit/cfed9fff74598c43f150ad5b8b29e4cc05ca989c. It should be out in v0.12.2
in 10-15 minutes.
can confirm it works with ollama mistral
@krrishdholakia It looks like version 0.12.2
is not yet available in poetry nor on pypi.
Same here for me. I am waiting the 0.12.2
release!
@reactivetype @kim333 @Rizaldy thanks for raising this. working on it now.
Fix is live in a dev release v0.12.4.dev2
We're working on a main release.
I arrived here with almost the same problem as the poster, but using local ollama. I installed 0.12.4.dev2, but the litellm server is just not answering. No CPU or GPU usage.
Normally, this would seem like a separate bug, but I'm not sure it's related specifically to the dev release or this was a problem before.
Command:
litellm --model ollama/wizardcoder:13b-python --api_base http://localhost:8000
Test command fails:
litellm --test --host 127.0.0.1
Side note, Control-C doesn't kill the server. I have to kill -9 it.
thanks for raising it @eleqtrizit
I'm on v0.12.4
locally and i can confirm this works:
What happened?
I was trying to make the OpenAI Proxy Server on my local to work, following the documentation
litellm --model together_ai/WizardLM/WizardCoder-Python-34B-V1.0
On my terminal looks fineDocs: https://docs.litellm.ai/docs/proxy_server
LiteLLM: Test your local endpoint with: "litellm --test" [In a new terminal tab]
LiteLLM: View available endpoints for this server on: http://0.0.0.0:8000
LiteLLM: Self-host your proxy using the following: https://docs.litellm.ai/docs/proxy_server#deploy-proxy
INFO: Started server process [87766] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
'Router' object has no attribute 'model_names' LiteLLM.Exception: 'Router' object has no attribute 'model_names' INFO: 127.0.0.1:59454 - "POST /chat/completions HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/litellm/proxy/llm.py", line 138, in litellm_completion if model_router and data["model"] in model_router.get_model_names(): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/litellm/router.py", line 100, in get_model_names return self.model_names ^^^^^^^^^^^^^^^^ AttributeError: 'Router' object has no attribute 'model_names'
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in call return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/fastapi/applications.py", line 1115, in call await super().call(scope, receive, send) File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/starlette/applications.py", line 122, in call await self.middleware_stack(scope, receive, send) File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in call raise exc File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in call await self.app(scope, receive, _send) File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/starlette/middleware/cors.py", line 83, in call await self.app(scope, receive, send) File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in call raise exc File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in call await self.app(scope, receive, sender) File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in call raise e File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in call await self.app(scope, receive, send) File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/starlette/routing.py", line 718, in call await route.handle(scope, receive, send) File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle await self.app(scope, receive, send) File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/starlette/routing.py", line 66, in app response = await func(request) ^^^^^^^^^^^^^^^^^^^ File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/fastapi/routing.py", line 274, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function return await dependant.call(values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/litellm/proxy/proxy_server.py", line 515, in chat_completion return litellm_completion(data, type="chat_completion", user_model=user_model, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/backoff/_sync.py", line 105, in retry ret = target(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/litellm/proxy/llm.py", line 148, in litellm_completion handle_llm_exception(e=e, user_api_base=user_api_base) File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/litellm/proxy/llm.py", line 92, in handle_llm_exception raise UnknownLLMError from e litellm.proxy.llm.UnknownLLMError
Twitter / LinkedIn details
No response