search

BerriAI / litellm

Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]
https://docs.litellm.ai/docs/
Other
13.45k stars 1.57k forks source link

[Bug]: Together AI / proxy calls - yi-chat #1082

Closed krrishdholakia closed 10 months ago

krrishdholakia commented 10 months ago

What happened?

Hi guys, I try to update to 1.11.1 and when I run litellm with together.ai I got this error (SS) for some reason the completion is looking at Huggingface not together_ai. I use config.yaml as in SS.

the curl is:

curl --location 'http://0.0.0.0:8000/chat/completions' \
    --header 'Content-Type: application/json' \
    --data ' {
    "model": "together_ai/zero-one-ai/Yi-34B-Chat", #This already tried with `Yi-34B-Chat` but still had the same error
    "messages": [
        {
        "role": "user",
        "content": "what llm are you"
        }
    ]
    }'

image

image

Relevant log output

No response

Twitter / LinkedIn details

No response

Rizaldy commented 10 months ago

Thanks @krrishdholakia This is more information as I test Together.ai model. All tested using curl only plug the model name that already on config.yaml. And before updating the litellm is working but I forget what version it is.

curl --location 'http://0.0.0.0:8000/chat/completions' \
    --header 'Content-Type: application/json' \
    --data ' {
    "model": "Yi-34B-Chat",                  
    "messages": [
        {
        "role": "user",
        "content": "what llm are you"
        }
    ]
    }'

Yi-34B-Chat

  File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/litellm/utils.py", line 4973, in exception_type
    raise e
  File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/litellm/utils.py", line 4763, in exception_type
    raise APIError(
litellm.exceptions.APIError: TogetherAIException - TogetherAIException - {"model":"zero-one-ai/Yi-34B-Chat","request_id":"833f00fd29ee880b-SIN","error":"Cannot read properties of undefined (reading 'status')"}
INFO:     127.0.0.1:64934 - "POST /chat/completions HTTP/1.1" 500 Internal Server Error

Solar-70B

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/litellm/llms/prompt_templates/factory.py", line 286, in format_prompt_togetherai
    prompt = hf_chat_template(model=None, messages=messages, chat_template=chat_template)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/litellm/llms/prompt_templates/factory.py", line 231, in hf_chat_template
    raise Exception("Error rendering template")
Exception: Error rendering template
INFO:     127.0.0.1:65263 - "POST /chat/completions HTTP/1.1" 500 Internal Server Error

CodeLlama-34b-Python, This one is had error asking for API key. The api key is already put inside config.yaml and tested with Mistral model it works

  File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/litellm/llms/together_ai.py", line 118, in completion
    prompt = prompt_factory(model=model, messages=messages, api_key=api_key, custom_llm_provider="together_ai") # api key required to query together ai model list
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/litellm/llms/prompt_templates/factory.py", line 373, in prompt_factory
    return format_prompt_togetherai(messages=messages, prompt_format=prompt_format, chat_template=chat_template)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/litellm/llms/prompt_templates/factory.py", line 283, in format_prompt_togetherai
    human_prompt, assistant_prompt = prompt_format.split('{prompt}')
                                     ^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'split'
INFO:     127.0.0.1:65288 - "POST /chat/completions HTTP/1.1" 500 Internal Server Error

WizardLM-70B-V1.0

    return format_prompt_togetherai(messages=messages, prompt_format=prompt_format, chat_template=chat_template)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/litellm/llms/prompt_templates/factory.py", line 286, in format_prompt_togetherai
    prompt = hf_chat_template(model=None, messages=messages, chat_template=chat_template)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/litellm/llms/prompt_templates/factory.py", line 231, in hf_chat_template
    raise Exception("Error rendering template")
Exception: Error rendering template
INFO:     127.0.0.1:65311 - "POST /chat/completions HTTP/1.1" 500 Internal Server Error

NexusRaven-V2-13B

                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/litellm/llms/together_ai.py", line 118, in completion
    prompt = prompt_factory(model=model, messages=messages, api_key=api_key, custom_llm_provider="together_ai") # api key required to query together ai model list
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/litellm/llms/prompt_templates/factory.py", line 372, in prompt_factory
    prompt_format, chat_template = get_model_info(token=api_key, model=model)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aldi/miniconda3/envs/litellm_env/lib/python3.11/site-packages/litellm/llms/prompt_templates/factory.py", line 277, in get_model_info
    return m['config'].get('prompt_format', None), m['config'].get('chat_template', None)
           ~^^^^^^^^^^
KeyError: 'config'
INFO:     127.0.0.1:65350 - "POST /chat/completions HTTP/1.1" 500 Internal Server Error
krrishdholakia commented 10 months ago

Hi @Rizaldy, i'm unable to repro your issue.

Screenshot 2023-12-12 at 5 26 19 PM
krrishdholakia commented 10 months ago

To deal with this though, I've added a try-except block around the GET request for the together ai template - https://github.com/BerriAI/litellm/commit/82d28a8825f7f61e4f0028b6268b564af54debf2

Should be live in v1.13.1