Allow self-hosted LLMs

[anjok]

Given privacy and cost concerns it would be nice if we could use self-hosted LLMs.

I downloaded LM Studio, installed "lmstudio-community/Meta-Llama-3-8B-Instruct-GGUF" and started the server at port 1234.

This is the minimal fix needed to have Vector use it:

diff --git a/chipper/pkg/wirepod/ttr/kgsim.go b/chipper/pkg/wirepod/ttr/kgsim.go
index 818e4cf..3f34921 100644
--- a/chipper/pkg/wirepod/ttr/kgsim.go
+++ b/chipper/pkg/wirepod/ttr/kgsim.go
@@ -82,7 +82,8 @@ func StreamingKGSim(req interface{}, esn string, transcribedText string) (string
                        vars.WriteConfigToDisk()
                }
                conf := openai.DefaultConfig(vars.APIConfig.Knowledge.Key)
-               conf.BaseURL = "https://api.together.xyz/v1"
+               // conf.BaseURL = "https://api.together.xyz/v1"
+               conf.BaseURL = "http://localhost:1234/v1"
                c = openai.NewClientWithConfig(conf)
        } else if vars.APIConfig.Knowledge.Provider == "openai" {
                c = openai.NewClient(vars.APIConfig.Knowledge.Key)

To actually have it user configurable, it wouldn't require much more work, but more than I can currently put in:

• add "url":"..." key in apiConfig file. Having it written out
• change the UI to be able to edit this url for "togetherai" - or use some other provider like "compatible" and copy the together stuff each time it's used - or rename "together " to "compatible" and provide its url as the default

It was a ton of work to get this to actually run as I don't think the builds on MacOS are correct, but I just may be doing something wrong.... why is it using /root in the builds and why is sudo needed to actually build? Also, there's no default brew user. And finally it couldn't find my vosk_api.h file and I had to symlink/copy a lot of stuff around.

[anjok]

Discord discussion: https://discord.com/channels/527874754342944770/1245142221318258708/1245142221318258708

[LordJuice1]

I'm interested in this as well. I'll have to put time aside to mess with it on my Pi. But, I would love to see this added

[kercre123]

A "Custom" option for Knowledge Graph has been added in the latest release, which lets you enter a custom endpoint.