Remote server demo

[shavit]

I tried it with a remote server and it works pretty well.

The app requires a model to be loaded first before using it, although it doesn't need a model if configured with a remote server. But for UX it is good to have the chat always available with a fallback model.

Maybe should be a boolean / chekbox value to indicate that the user uses a remote server, and enable / disable remote server config in the settings.

• [x] Onboarding without a model.
• [x] Settings UX.
• [x] Switch between servers during conversation.

Closes https://github.com/psugihara/FreeChat/issues/28

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
free-chat	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 24, 2024 1:10am

[psugihara]

sweet!

I agree it seems fine to punt on any onboarding changes since this is really an advanced use-case.

Let me think about the UI a little bit. I think my goals would be:

• explain how to run the server / what the server should be (e.g. the command for running server on port X)
• make it clear that this is not required / that the app runs it's own server

Another consideration is that we start the server with the path to the model. The model in the Intelligence section of Settings is no longer active when you have a server connected, so that's a bit misleading.

One solution could be putting your Server tab contents into the Intelligence > Model section and showing it rather than the current contents if you select a special option from the model dropdown "Custom server". What do you think of that?

[shavit]

Do you know what users think about the current onboarding?

It seems a good UX to download a default model and make the app ready, the way it works right now. Except maybe the case where users want to delete the default model, they will have the welcome dialog.

If a selects a remote location then the model and advanced settings sections are dimmed. The host and port will be inline, and maybe TLS checkbox for the scheme (http/https), and on save the app will try to construct the URL and ping the server or get the /health endpoint.

[psugihara]

Yeah, I think current onboarding is good and should not be changed for this feature.

I like the remote model language and keeping it in the Intelligence section. I'm thinking something like this could work in the model picker:

Selecting that could replace the instruction text and show the host and port fields you have.

The new instruction text could be something like:

"If you have access to a powerful server, you may want to run your model there. Enter the host and port to connect to a remote llama.cpp server. Instructions for running the server can be found here."

What do you think?

[shavit]

All those settings are for the local machine, not just the model. The remote server options should be separate in some way, in a different section.

Also the remote server is for users who are familiar with llama.cpp, and the link isn't helpful for those who need instructions.

My initial idea was based on Xcode's preferences

[psugihara]

My problem with having it as its own item in the nav bar is it's a bit confusing that some of the intelligence stuff would not be used (model) but some will be (system prompt, temperature, prompt template). I also think it's a bit too prominent for this very advanced feature.

the remote server is for users who are familiar with llama.cpp and the link isn't helpful

I use the server.cpp README to run/configure the server. Maybe there's something better? I would not assume people know how to use server, it changes week to week and is kind of arcane to set up. Without the llama.cpp reference I think many power users would think it's supposed to be an Open AI API compatible server since that's more common.

All those settings are for the local machine, not just the model.

System prompt and temperature will be used if a remote model is picked. Maybe my description was confusing, here's a quick mock of what I mean it would look like when remote model is selected...

It could have a little status indicator in there too showing that the connection is good with the /health endpoint as you mention or something else.

[shavit]

~It is simple to configure the URL for the remote server, but the app wouldn't work without the selection of the local model. The server and agent are dependent on the local model file path.~

This leads to another minor issue, when a user deletes all models the welcome sheet pops, and load custom model option will open a second EditModels window.

[psugihara]

Nice fix. One thing that occurred to me is that you probably want to be able to toggle the template format when using a remote model. I can think of 2 solutions off the top of my head:

• Create a dummy Model object for the Remote model selection to store the template format
• Store an @AppStorage key specifically for the template format when using remote server

[shavit]

Since the settings are already in the key-value, then the second option is more straightforward. But if there are requests to save the settings with the models, then it worth to add different settings with prompts in a table.

[psugihara]

looking good! I think this is good progressive disclosure for advanced users.

I made a few minor tweaks here but I think you have to sync them into your branch for them to show up here.

One weird state I hit... I tried testing on a local server but it wasn't running so I restarted it but the "Apply" button stays disabled so I couldn't re-test. Maybe when it hits the error state it should not be disabled.

[shavit]

Yes, it enables the button only if there are changes. The saved connection retries, but there is no indication for that. I added a retry to the connection error label. It will also enable the Apply button too.

[psugihara]

awesome. looks good! thank you 🙏