[Feature Request]: Support for DeepInfra

[shauryr]

Feature Description

DeepInfra is pay per use service like OpenAI but way cheaper - 50% cheaper than gpt3.5-turbo for llama 2

Right now they support openai python api.

So I tried to test my luck, what happens if you use it with llama-index.

llm = OpenAI(model="meta-llama/Llama-2-70b-chat-hf")
service_context = ServiceContext.from_defaults(llm=llm)

You get this trace of error -

-> service_context = ServiceContext.from_defaults(llm=llm)

[llama_index/indices/service_context.py:168]
prompt_helper = prompt_helper or _get_default_prompt_helper(
--> 168     llm_metadata=llm_predictor.metadata,
    169     context_window=context_window,
    170     num_output=num_output,
    171 )

[llama_index/llm_predictor/base.py:119]
--> 119     return self._llm.metadata

[llama_index/llms/openai.py:113]
--> 113         context_window=openai_modelname_to_contextsize(self._get_model_name()),

[llama_index/llms/openai_utils.py:192]
--> 192     raise ValueError(
ValueError: Unknown model: meta-llama/Llama-2-70b-chat-hf. Please provide a valid OpenAI model name.Known models are: gpt-4, gpt-4-32k, gpt-4-0613, gpt-4-32k-0613, gpt-4-0314, gpt-4-32k-0314, gpt-3.5-turbo, gpt-3.5-turbo-16k, gpt-3.5-turbo-0613, gpt-3.5-turbo-16k-0613, gpt-3.5-turbo-0301, text-davinci-003, text-davinci-002, gpt-3.5-turbo-instruct, text-ada-001, text-babbage-001, text-curie-001, ada, babbage, curie, davinci, gpt-35-turbo-16k, gpt-35-turbo

Reason

DeepInfra seems like it is fairly new to the game.

Value of Feature

Their product is great! and it is much cheaper than gpt-4 with similar performance. As they add more models to their endpoints I imagine more and more people start using them as a better and cheaper alternative to OAI's GPT.

[dosubot[bot]]

🤖

Thank you for your feature request and for providing a detailed description of the issue you're facing. I understand that you're looking to integrate DeepInfra with LlamaIndex due to its cost-effectiveness and similar performance to gpt-4.

Based on the error message you're receiving, it seems that LlamaIndex currently does not recognize the model name meta-llama/Llama-2-70b-chat-hf as it's not in the list of known models. This is likely because LlamaIndex has not been updated to support DeepInfra yet.

To add support for DeepInfra, we would need to extend the current implementation of LlamaIndex. This would involve creating a new class, similar to how we have classes for other APIs like OpenAI. This new class would handle the interaction with the DeepInfra API.

Here's a high-level overview of the steps:

1. Import the necessary DeepInfra libraries.

2. Create a new class, say DeepInfraLLM, that inherits from the LLM base class. This class will implement the methods required to interact with the DeepInfra API.

3. In the __init__ method of this class, initialize the connection to the DeepInfra API.

4. Implement the chat, stream_chat, complete, and stream_complete methods to interact with the DeepInfra API. These methods will be similar to the ones in the OpenAI class but will use DeepInfra's API.

Please note that the exact implementation will depend on the specifics of the DeepInfra API, which is not provided in the context. Therefore, the above steps are a general guide and may need to be adjusted based on the actual API.

As for the error you're encountering, it's happening because the model name you're trying to use is not recognized by LlamaIndex. To fix this, we would need to add the model name meta-llama/Llama-2-70b-chat-hf to the list of known models in the openai_utils.py file.

I hope this gives you a good starting point for integrating DeepInfra with LlamaIndex. If you have any further questions or need more clarification, please let me know.

Sources

• [Feature Request]: Fastchat RESTful OpenAI API server drop-in for Llama 2
• llama_index/vector_stores/deeplake.py
• llama_index/llms/init.py
• llama_index/llms/openai.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[logan-markewich]

@shauryr you maybe need to change the api_base parameter to point to deepinfra?

[shauryr]

yup I did that above this.

[prince-paulraj]

I am interested in this as well, llama_index with deepInfra - llama 2 model...

The following code snippet from deepInfra working fine but want the integration from llama_index. Can we get this feature ASAP?

`

Access Llma2 model from DeepInfra

import openai import os

openai.api_key = "#" openai.api_base = "https://api.deepinfra.com/v1/openai"

chat_completion = openai.ChatCompletion.create( model="meta-llama/Llama-2-70b-chat-hf", messages=[{"role": "user", "content": "Hello"}], )

print(chat_completion.choices[0].message.content) `

[nerdai]

Closing this as I believe this can be implemented using OpenAILike https://github.com/run-llama/llama_index/blob/main/llama_index/llms/openai_like.py