search

amaiya / onprem

A tool for running on-premises large language models with non-public data
https://amaiya.github.io/onprem
Apache License 2.0
684 stars 32 forks source link

Use OnPrem.LLM with OpenAI-compatible REST APIs #61

Closed amaiya closed 3 months ago

amaiya commented 3 months ago

Many LLM tools can serve LLMs through OpenAI-compatible REST APIs. Examples include vLLM, OpenLLM, and Ollama. This issue is to allow OnPrem.LLM to easily be used with these served LLMs.