replace openai base with proxy w/ translation

[krrishdholakia]

There’s a lot of different LLM deployment providers. How do I easily replace my OpenAI base with their url as a proxy? - https://github.com/petals-infra/chat.petals.dev/issues/20, https://www.banana.dev/, etc.

```python
def translate_function(model, messages, max_tokens):
    prompt = " ".join(message["content"] for message in messages)
    max_new_tokens = max_tokens
    return {"model": model, "prompt": prompt, "max_new_tokens": max_new_tokens} 

openai.api_base = litellm.translate_api_call(custom_api_base, translate_function)
```

[krrishdholakia]

Tracking 20 repos that might find this helpful:

• https://github.com/shroominic/codeinterpreter-api/issues/82
• https://github.com/GAIR-NLP/factool/issues/8
• https://github.com/danswer-ai/danswer/issues/240
• https://github.com/openai/openai-python/issues/471
• https://github.com/bentoml/OpenLLM
• https://github.com/basetenlabs/truss
• https://github.com/pytorch/serve
• https://github.com/apot-group/ml-models-in-production
• https://github.com/premAI-io/prem-app
• https://github.com/InternLM/lmdeploy
• https://github.com/homanp/superagent
• https://github.com/tensorchord/openmodelz
• https://github.com/baileytec-labs/llama-on-lambda
• https://github.com/ludwig-ai/ludwig
• https://github.com/facebookresearch/llama-recipes/blob/main/docs/inference.md
• https://github.com/THUDM/SwissArmyTransformer

[ishaan-jaff]

updates:

• baseten client/url doesn't work - they have a bunch of auth errors. Reached out to their support for help

[ishaan-jaff]

How would this be possible ? When you set a url for OpenAI base it expects a server

openai.api_base = "https://localhost:500"

The proposed interface there is a function :

openai.api_base = litellm.translate_api_call(custom_api_base, translate_function)

Open to suggestions on this

[ishaan-jaff]

Tried doing something like this:

def custom_function():
  print("custom function called")

openai.api_base = custom_function

but openai calls @app.route('/chat/completions', methods=["POST"])

[ishaan-jaff]

closing this