search

BerriAI / litellm

Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]
https://docs.litellm.ai/docs/
Other
12.57k stars 1.46k forks source link

[Feature]: Add support for Nvidia AI Foundation endpoints #1266

Open ishaan-jaff opened 9 months ago

ishaan-jaff commented 9 months ago

The Feature

NVIDIA AI Foundation Endpoints give users easy access to NVIDIA hosted API endpoints for NVIDIA AI Foundation Models like Mixtral 8x7B, Llama 2, Stable Diffusion, etc. These models, hosted on the NVIDIA NGC catalog, are optimized, tested, and hosted on the NVIDIA AI platform, making them fast and easy to evaluate, further customize, and seamlessly run at peak performance on any accelerated stack.

Motivation, pitch

new provider with self hosted LLMs

Twitter / LinkedIn details

No response

krrishdholakia commented 9 months ago

I think this means you can self-host

Screenshot 2023-12-29 at 11 12 05 AM
krrishdholakia commented 9 months ago

is this already openai-compatible?

Screenshot 2023-12-29 at 11 13 42 AM
krrishdholakia commented 9 months ago
Screenshot 2023-12-29 at 11 14 18 AM
krrishdholakia commented 9 months ago

Nvm - you have to poll an endpoint for a response - similar to replicate.

Screenshot 2023-12-29 at 11 15 03 AM