[Questions]: Call ollama docker image return `404 page not found`

Ivanw commented 1 year ago

What happened?

I followed this https://docs.litellm.ai/docs/providers/ollama#making-a-request-to-this-server write a test code(just copy them), after executing I get 404 page not found.

In docker container, I can execute successful,

python3 test.py

In my browser access http://localhost:8000 or use curl http://localhost:8000, response successful with Ollama is running.

Relevant log output

log of docker image ollama:

[GIN] 2023/10/21 - 04:38:58 | 404 |       24.05µs |    192.168.65.1 | POST     "/chat/completions"

http://0.0.0.0:8000
LiteLLM: response from proxy with streaming
Traceback (most recent call last):
  File "/Users/ivan/Project/auto-gen/autogen/lib/python3.11/site-packages/openai/api_requestor.py", line 413, in handle_error_response
    error_data = resp["error"]
                 ~~~~^^^^^^^^^
TypeError: string indices must be integers, not 'str'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ivan/Project/auto-gen/litellm-openai.py", line 10, in <module>
    response = openai.ChatCompletion.create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ivan/Project/auto-gen/autogen/lib/python3.11/site-packages/openai/api_resources/chat_completion.py", line 25, in create
    return super().create(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ivan/Project/auto-gen/autogen/lib/python3.11/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 155, in create
    response, _, api_key = requestor.request(
                           ^^^^^^^^^^^^^^^^^^
  File "/Users/ivan/Project/auto-gen/autogen/lib/python3.11/site-packages/openai/api_requestor.py", line 299, in request
    resp, got_stream = self._interpret_response(result, stream)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ivan/Project/auto-gen/autogen/lib/python3.11/site-packages/openai/api_requestor.py", line 710, in _interpret_response
    self._interpret_response_line(
  File "/Users/ivan/Project/auto-gen/autogen/lib/python3.11/site-packages/openai/api_requestor.py", line 775, in _interpret_response_line
    raise self.handle_error_response(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ivan/Project/auto-gen/autogen/lib/python3.11/site-packages/openai/api_requestor.py", line 415, in handle_error_response
    raise error.APIError(
openai.error.APIError: Invalid response object from API: '404 page not found' (HTTP response code was 404)
(autogen) ivan@IvandeiMac auto-gen % /Users/ivan/Project/auto-gen/autogen/bin/python /Users/ivan/Project/auto-gen/litellm-openai.py
http://0.0.0.0:8000
LiteLLM: response from proxy with streaming
Traceback (most recent call last):
  File "/Users/ivan/Project/auto-gen/autogen/lib/python3.11/site-packages/openai/api_requestor.py", line 413, in handle_error_response
    error_data = resp["error"]
                 ~~~~^^^^^^^^^
TypeError: string indices must be integers, not 'str'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ivan/Project/auto-gen/litellm-openai.py", line 10, in <module>
    response = openai.ChatCompletion.create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ivan/Project/auto-gen/autogen/lib/python3.11/site-packages/openai/api_resources/chat_completion.py", line 25, in create
    return super().create(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ivan/Project/auto-gen/autogen/lib/python3.11/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 155, in create
    response, _, api_key = requestor.request(
                           ^^^^^^^^^^^^^^^^^^
  File "/Users/ivan/Project/auto-gen/autogen/lib/python3.11/site-packages/openai/api_requestor.py", line 299, in request
    resp, got_stream = self._interpret_response(result, stream)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ivan/Project/auto-gen/autogen/lib/python3.11/site-packages/openai/api_requestor.py", line 710, in _interpret_response
    self._interpret_response_line(
  File "/Users/ivan/Project/auto-gen/autogen/lib/python3.11/site-packages/openai/api_requestor.py", line 775, in _interpret_response_line
    raise self.handle_error_response(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ivan/Project/auto-gen/autogen/lib/python3.11/site-packages/openai/api_requestor.py", line 415, in handle_error_response
    raise error.APIError(
openai.error.APIError: Invalid response object from API: '404 page not found' (HTTP response code was 404)

Twitter / LinkedIn details

No response

krrishdholakia commented 1 year ago

@Ivanw can you share a code snippet we can use for repro?

Ivanw commented 1 year ago

@krrishdholakia I just use the exact same as the sample code:

import openai

api_base = f"http://0.0.0.0:8000" # base url for server

openai.api_base = api_base
openai.api_key = "temp-key"
print(openai.api_base)

print(f'LiteLLM: response from proxy with streaming')
response = openai.ChatCompletion.create(
    model="ollama/llama2", 
    messages = [
        {
            "role": "user",
            "content": "this is a test request, acknowledge that you got it"
        }
    ],
    stream=True
)

for chunk in response:
    print(f'LiteLLM: streaming response from proxy {chunk}')

krrishdholakia commented 1 year ago

The error would imply there's some issue on the server. Does it work when you make a curl request? @Ivanw

krrishdholakia commented 1 year ago

bump @Ivanw

ishaan-jaff commented 1 year ago

Hi @Ivanw was this issue resolved ?

I'd love to hop on a call and ensure this gets resolved for you sharing my cal for your convenience: https://calendly.com/ishaan-berri/30min?month=2023-11

wac81 commented 8 months ago

same error: curl --location 'http://0.0.0.0:8000/v1/chat/completions' --header 'Content-Type: application/json' --data ' { "model": "ollama/gemma:7b-instruct-q5_K_M", "messages": [ { "role": "user", "content": "what llm are you" } ], } '

config:

model_name: gpt-3.5-turbo litellm_params: model: ollama/gemma:7b-instruct-q5_K_M api_key: noneed api_base: http://127.0.0.1:11434/v1

model change to gpt-3.5-turbo , same error

BerriAI / litellm