For python developers, litellm maybe a good choice. That would be great we can do it in a runtime and any developers can use LLMs in own language way rather than packaging HTTP/gRPC calls by themselves.
Now we have many LLM APIs like OpenAI, Azure AI, Cohere, LLaMA, AWS bedrock, Kserve, OpenVINO and so on, migrate from one to the other still need many code changes.
I propose that we can do it in a Cloud Native LLM runtime, then developers can migrate from one to the other only by config.
zanetworker
commented
5 months ago 
daixiang0
For python developers, litellm maybe a good choice. That would be great we can do it in a runtime and any developers can use LLMs in own language way rather than packaging HTTP/gRPC calls by themselves.
Now we have many LLM APIs like OpenAI, Azure AI, Cohere, LLaMA, AWS bedrock, Kserve, OpenVINO and so on, migrate from one to the other still need many code changes.
I propose that we can do it in a Cloud Native LLM runtime, then developers can migrate from one to the other only by config.