jeremychone / rust-genai

Rust multiprovider generative AI client (Ollama, OpenAi, Anthropic, Groq, Gemini, Cohere, ...)
Apache License 2.0
164 stars 35 forks source link

Allow default model and api key #7

Closed stargazing-dino closed 1 month ago

stargazing-dino commented 1 month ago

Allows something like this:

let client = Client::builder()
    .with_default_model("gpt-3.5-turbo")
    .with_default_api_key("my_key")
    .build();

so users don't have to pass it at every call site of exec_chat

jeremychone commented 1 month ago

@stargazing-dino First, can you squash those commits into one, so that it's faster for me to review? Perhaps creating a new branch and doing a fresh PR with the squashed commits might be good.

Second, I need to think about this. I do not guarantee I will take it as is.

I will look at the code, but there are other dimensions. For example, the API Key might be more by AdapterKind, and then I also need to add a custom endpoint. So, I need to think about all of this together. But your commit can help me clarify one requirement at least.

Thanks.

jeremychone commented 1 month ago

@stargazing-dino Sorry, this does not really fit the API model in mind. I do think it is important to split the model and even adapters from the client.

One way to accomplish this behavior would be to implement an application struct that would have the client, the appropriate default, and re-expose the exec in a custom way.

I will close this PR.

jeremychone commented 1 month ago

By the way, there are two features I am going to implement that will allow you to accomplish what you described.

1) Model alias - the Client/ClientConfig will have aliases:

client_builder
  .append_alias("big-one", "gpt-4o")
  .append_alias("small-one", "llama-3.1-70b-versatile")

So, the developer can also set an alias for "" and that will work as well.

2) AuthResolver refactor to be at the Client level - Now that we have ModelInfo with adapter_kind and model_name, I am going to refactor the AuthResolver to be at the ClientConfig level (so, no need for AdapterConfig for now and probably later as well). This will simplify the API and implementation a lot.

    let client = Client::builder()
        .with_sync_auth_resolver(|model_info: ModelInfo| { ... })
        .build();

This new AuthResolver model is much simpler, same flexibility, one entry point for all.