Settings Customized options for generation.

[dezoito]

Greetings,

I am trying to set inference options such as temperature and top_k when using the completion generation API.

Something like:

// Declare the options I want
let generation_options = GenerationOptions {
    mirostat: Some(5),
    mirostat_eta: Some(0.1),
    mirostat_tau: Some(0.2),
    num_ctx: None,
    .....
};

// Build the request, passing the options
let req = GenerationRequest::new(params.model, params.prompt).options(generation_options);

// Make the request.
let res = match ollama.generate(req).await {
    Ok(value) => value,
    Err(err) => {
        println!("Error: {}", err.to_string());
        return Err(Error::StringError(err.to_string()));
    }
};

I currently have two issues, though:

1- Is that the correct way to declare and pass those options?

2- When trying to create the generation_options instance, Rust warns that field is private for all fields (even though they are declared as public here)

In that case, what would be the correct to fix that? (I don't want to re-define the GenerationOptions struct within my project, if that can be avoided).

Thank you! I Appreciate the effort you've put into this project.

[dezoito]

For reference, I eventually solved the problem by using the syntax below:

    let options = GenerationOptions::default()
        .temperature(0.8)
        .repeat_penalty(1.5);