Allow to override endpoints by configuration

continuedev / continue

⏩ Continue is the leading open-source AI code assistant. You can connect any models and any context to build custom autocomplete and chat experiences inside VS Code and JetBrains

https://docs.continue.dev/

Apache License 2.0

19.76k stars 1.74k forks source link

Allow to override endpoints by configuration #2711

Open rmax opened 1 month ago

rmax commented 1 month ago

Validations

[X] I believe this is a way to improve. I'll try to join the Continue Discord for questions
[X] I'm not able to find an open issue that requests the same enhancement

Problem

I’m trying to use CodeLlama deployed on databricks but the endpoint is /invocations rather than /completions, using OpenAI-compatible provided.

Solution

It would be useful if we could override the completion endpoint by configuration.

RomneyDa commented 1 month ago

Comment from duplicate #2763 https://discord.com/channels/1108621136150929458/1300903286060220469

Can't currently do fully custom endpoints that aren't compatible with e.g. openai, ollama without adding a provider in the codebase. e.g. company.com/embeddings, company.com/conversation

baseUrl and apiUrl are nice but assume the endpoints match with e.g. ollama or openai

Solution Add chatUrl, embeddingsUrl etc. or some other paradigm that would allow this

RomneyDa commented 1 month ago

Falls under meta llm-info + config #2769