Features request: Converting llms.CallOptions to local-llm script arguments

[EvilFreelancer]

Hi!

I have a suggestion regarding the options we currently have.

In localllm.go file, we define the following strings:

const (
    // The name of the environment variable that contains the path to the local LLM binary.
    localLLMBinVarName = "LOCAL_LLM_BIN"
    // The name of the environment variable that contains the CLI arguments to pass to the local LLM binary.
    localLLMArgsVarName = "LOCAL_LLM_ARGS"
)

My idea is to make these environment variables optional. It would be better to have the ability to change them programmatically. For example, if the bin and args values are not set in the New function, we can use the values from the environment variables. If neither is set, we can check and trigger an error.

As a result, we would have something like this:

func New(opts ...Option) (*LLM, error) {
    options := &options{
        bin: os.Getenv(localLLMBinVarName),
        args: os.Getenv(localLLMArgsVarName),
    }

    for _, opt := range opts {
        opt(options)
    }

        path, err := exec.LookPath(options.bin)
    if err != nil {
        return nil, errors.Join(ErrMissingBin, err)
    }

    c, err := localclient.New(path, options.args)
    return &LLM{
        client: c,
    }, err
}

Another idea is to implement dynamic generation of the arguments array based on the options passed through the llms.CallOptions struct.

Before the execution stage, we can append these options to the args array. This would result in a string like the following:

/path/to/executable/file --top_k="1" --top_p="10" --seed="42" --prompt="prompt"

What are your thoughts on these ideas?

[edocevol]

Looks great!

[EvilFreelancer]

Looks great!

It's almost implemented btw, i've just can't find spare time for creating a PR :)

[EvilFreelancer]

Solution implemented