Custom AI Completion Endpoint

[tarasglek]

Lets extend the work in https://github.com/tarasglek/chatcraft.org/pull/498 to enable a custom provider.

Eg I would provide a url like "https://taras-free_open_router.web.val.run/api/v1" or "https://api.[deepinfra.com](https://deepinfra.com/docs/advanced/function_calling)/v1/openai" and a token and be able to talk to a custom model.

Chatcraft require that [url]/completion and [url]/models are openai-compatible. Note this should be done in such a way that we could have multiple custom providers, eg deepinfra and anyscale.

[humphd]

So to clarify a bit (tell me if I'm right, correct me):

• we create a new provider, CustomProvider, where the API URL is unknown and will be provided by the user
• we expect that URL to follow the same pattern as other OpenAI compatible endpoints (e.g., just like OpenRouter, which pretends to be OpenAI). We'll have to think about validation of the URL so you don't use a bad one (e.g., maybe what we have now is enough, validating the URL by getting models?)

Using this would require a UI update as well, to allow the user to enter a custom URL when they choose the custom provider.

@kliu57 this feels like something you'd be good at, interested?

[humphd]

Also, re-reading your comment above, it sounds like we'll want to be able to have multiple "custom providers" in our list (e.g., you might add 2 or 3 of your own custom providers). So our settings would need to be able to keep track of them all and let the user choose between them.

[tarasglek]

Your clarification makes sense. We already validate via models endpoint, I think that should be enough here.

[kliu57]

@humphd sure I can take this

[kliu57]

@humphd @tarasglek What do you think of my design idea for adding/removing custom providers? For the adding part, I got the design from Microsoft Azure portal.

I could have them input the key at the time of inputting the new url, or have them input the key afterwards. This design flowchart above has them inputting the key afterwards.

[tarasglek]

I like borrowing existing designs that work

[humphd]

@kliu57 I think this is going to be hard to implement and maintain, because you're going to require custom UI components.

I would simplify this. We are going to have an unknown number of entries, each with multiple values. This perfectly describes a table:

[ ]	Provider	API URL	API Key
[ ]	OpenAI	https://openai.com/v1/	**
[ ]	FreeProvider	https://free-chatcraft.ai.deno.dev/api/v1

Beneath this, you put Delete and Add buttons. The Delete button would operate on any items that have checkmarks beside them. The Add button would create a new row. Only the API key would be editable in each row.

[kliu57]

@humphd Do you mean something like this?

Also what do you mean only the api key would be editable in the row? When I press add and create a new row I would need to enter the api URL and the api key as well right? not just the api key

[humphd]

Consider this UI pattern from AWS. Here I'm looking at Docker images in a container registry:

Notice how I have a checkbox beside each one, so that I can select (and then delete) any of the rows. The header also has a checkbox so that I can select them all (we may not need that).

I wouldn't use a drop-down at all, just show the table.

In terms of "API Key is editable," you are going to need to be able to update your API key, but not the provider name, for example.

[kliu57]

@humphd if I get rid of the dropdown and just have the table, how would the user select whcih one is their current provider? Currently they utilize the dropdown to select whcich provider they would like to use as the current provider.

[humphd]

Two options:

1) We can add another column to "set as default" (e.g., a button) 2) Remove the idea of setting the "default" provider in the settings completely and use the work that @Rachit1313 is doing to the Ask menu, adding the providers in a list with a checkmark, to indicate it.

[kliu57]

@humphd I think better to add the other column so people could add their new provider and just use it right away