Issue 341 Manage LLMs

[stephan-buckmaster]

Ok, this actually works!

The ordinary case is to create a new language model, that is not in the built-in list, but uses the "regular services"

This can be accomplished as follows:

1. Go to settings, Language Models
2. Select "Add New"
3. Fill in Name gpt-3.5-turbo-16k and Description, "My gpt-3.5-turbo-16k", and Save
4. Select "Add new assisant"
5. Under Model, select your model from step 3
6. Leave APIService as Built-in
7. Save
8. Now you can converse with this new model (Note: I got some flakiness, and also choosing other models from https://platform.openai.com/docs/models/ resulted in errors "You have no access to this model")

I don't have access to Anthropic so can't test that. I presume it should go just the same, as long as the name starts with "claude."

Here are steps to connect to another service. Let's say you have a ollama server running at http://1.2.3.4:8888. Not going to show to set that up, except that using the proxy from https://github.com/stephan-buckmaster/ruby-bearer-auth-proxy should make it easy to add authentication)

1. Go to settings, Language Models
2. Select "Add New"
3. Fill in Name llama3 -- or whatever model you know is available there, and Description, "Local llama3", and Save
4. Select "API Services"
5. Select "Add New"
6. Enter http://1.2.3.4:8888 under URL , Description "My local 1.2.3.4," and access token as needed
7. Select Driver as Open AI
8. Save
9. Select "Add new assisant"
10. Under Model, select your model from step 3 (Local llama3 ?)
11. Under API Service select your service from step 7 (My local 1.2.3.4)
12. Save

Now you can start chatting with that assistant.

Notes:

1. If no ollama server or similar is available, I have published a mock service at https://github.com/stephan-buckmaster/lm_qa_api and reverse-proxy at https://github.com/stephan-buckmaster/ruby-bearer-auth-proxy (again this doesn't support Anthropic API style)
2. It may seem a bit scattered, having 3 setups for one assistant, but this is actually very flexible. It seems wrong to ask user to reenter the same over and over for example, if all of this was on the Assistant level.
3. It may be needed to add a slash at the end of the URL for the API Service. This looks like a library thing though
4. In dev mode, I find errors from the connection may show up in the conversation. Not very nice for reguar users.
5. Not happy about the display of Language Models/API services as "cards." A table doesn't seem good for mobile though.
6. Can add selectors "All" / "Mine" to the Language Models listing.
7. Soft-deletes supported in the db-schema for Language Models/API Services, but not implemented
8. Expecting UI changes, so no proper tests added
9. It seems Description in Language Model, and API Service would be better named Title, to be used for the drop-downs in the assistant form

[krschacht]

Very nice! I’m starting to check this out.

[krschacht]

Hi @stephan-buckmaster, I’m excited to get this PR mergers in. I looked through it more closely today and I noticed that some things I cleaned up in your last PR seem to be reverted in this one. I think it’s because this wasn’t branched off of main. You had probably created this branch off of your old PR before I did some revisions and merged it in.

Can you cherry-pick those commits that I added onto your other branch back onto this branch? You’ve probably done that before but just in case:

1) look at the commit history for that last PR and note the commit hashes that I applied (and maybe your last couple too). Copy them in the order they were committed to that last branch 2) while on this branch, just do git cherry-pick [hash]

I suspect most of those should apply cleanly. If it does, that should resolve a lot of the merge conflicts that this branch is showing.

Or, if you try and and get a lot of merge conflicts, then another possibility would be to create a brand new branch off of main. Then look through the git history of this new branch of yours and cherry-pick the relevant changes back onto the new branch. But that assumes that the significant changes from this PR are in a handful of clean commits.

Or, actually as I’m writing this to you I just thought of a third possibility. And now that I think of it, this might be the easiest of all. Whenever I find myself with a branch that (a) has a lot of good changes I want to get in, but (b) has a bunch of changes I don’t want to include, then I have a technique for this.

My best technique is:

• On your local machine, make sure you switch to main and pull down the latest changes
• Then switch to this branch
• And from there create a new branch just so we don’t mess anything up: git branch -b new-manage-llms That should create & switch you to the new branch.
• Then on this new branch, uncommit & unstage everything you’ve done with: git reset --mixed main
• Do git status and you’ll see what I mean. All of your changes are still there on the branch but they’re all showing as uncommitted. Now it’s really easy to click through each file and re-commit the changes you actually want to submit. Make sure you have a good git editor (like in VS code) so you can select lines within a file and just stage & commit those lines because you may not want to commit whole files.

Before you do this, skim the commits I added to your last diff just to note the things I did so that you preserve those (or maybe you intentionally changed some of that, which is fine, but then you’ll be committing only what you intend).

[stephan-buckmaster]

So sorry about that, I thought I had already done that. So I'll go make a new branch, and go cherry-picking.

How do you find the user interface here?

[stephan-buckmaster]

Your reset-mixed suggestion showed large differences, both in files and lines. I was surprised. Replaying the actual commits I know I want looks like the better option.

[krschacht]

On your question of the UI, I just spent a little while going through it closely and thinking about it. There is something I'm finding confusing about separating out "API Service" and "Language Model." I think we might be treating them as a bit too disconnected from each other. For example, when I go to create a new Assistant I now select the Language Model and API Service as two separate drop-downs. So right now I can select "GPT-4o" and the local interface I just selected — but that's not true. GPT-4o must be used with a specific interface. I think it's probably more accurate for Assistants to require you to select a Language Model, and when editing (or creating) a Language Model you indicate that what API service it uses.

So, for example, if I run a Llamafile locally, there is a new API service I'm creating and a single new Language Model that uses it. Or, if I add Groq as a new API service then there are multiple new Language Models that I can create which are associated with this same API service. So I'd propose:

• [ ] Assistants has_one LanguageModel which has_one APIService

Regarding visual UI. It was really easy to use when I tried it. The suggestions I have are just to make it a bit more consistent with the other UI. The specific changes I'd propose are:

• [ ] In the left sidebar there is a hover effect and "selected" state when you're on one of those sections. It's working for all the other ones but not for "Language Models" and "API Services"
• [ ] When I click language models (and API service) and see all of them listed, I think we should do that just as a text list rather than these cards. And I think the order of the list should be the same as what we currently have in the drop-down when editing an Assistant. Card views like this are better for images where you're visually scanning, but when you have to read text to "pick" what you want to click on, it's easier for the eye to track down a list than to jump between cards.
• [ ] Let's keep the New button at the top of the list (as you have it) but let's style that button the same as the primary action button (e.g. Save) beneath the edit assistant form. And when you show a back button let's style that the same as the secondary action button (e.g. Cancel) beneath the edit assistant form. I've been meaning to turn those into real components that we're reusing, but for now you can just continue to copy & paste the set of tailwind classes. (and no blue links, just use the same style)
• [ ] You have separate View and Edit pages but I don't think you need those. With the Assistants, viewing is done on the form. I think that works and is slightly simpler.

[stephan-buckmaster]

Thanks for the feedback.

It's a good point that it looks heavy, that language models and API services are separated. Indeed I had them combined initially. Then I looked at the database schema and it wasn't normalized; then I used the app the next day and it was annoying to look up the port and access token. So once one has a API server setup I think it's natural to have that reflected in one database table, and one UI section. Its often effort to locate the access-token (again)

On the other hand, by having the langauge model interface like this, users can do the "best-of" thing that we put in in the migration ("Best OpenAI Model") by just using best-of in their title, and by updating that one language_models table record, all its assistants will be updated.

So both tables and forms are useful. Although they look split at first.

So I suggest that separation part is actually ok. In the future there may be more other settings to administer (tools, precanned prompts, user-groups,???), when the new items wouldn't get as much weight as they currently have. (I feel the assistants should not be as prominent, get that much space)

In terms of the display you suggest, I'll work on item 1,3.

Item 2: So you're saying instead of cards, just have in effect, a paragraph, per entry. Then an Edit link if applicable. Item 4: The "system" entries (have user_id=NULL) wouldn't be editable by any user, in my mind. So they wouldn't get shown in a form.

[stephan-buckmaster]

Replaced with new pull-request: https://github.com/AllYourBot/hostedgpt/pull/389