Implement multi-model support via JSON/YAML config

floneum / floneum

Instant, controllable, local pre-trained AI models in Rust

http://floneum.com/kalosm

Apache License 2.0

1.53k stars 71 forks source link

Implement multi-model support via JSON/YAML config #265

Open Liberxue opened 2 months ago

Liberxue commented 2 months ago

Hey

I have a suggestion that I believe could enhance its flexibility and ease of use:

Would it be possible to implement example floneum/models/xxx/src/source.rs using JSON or YAML configuration files to support multiple models? The benefits I envision are:

Easier addition of new models
Simplified parameter adjustments
Increased flexibility for users

If this aligns with the project's goals, I'd be willing to contribute by submitting a Pull Request to implement this feature.

ealmloff commented 2 months ago

We could pull all of this information from the .gguf file. For example, the llama 8b chat model defines the .gguf link, tokenizer and chat format, but if you look at the data defined in the .gguf file for llama 8b chat it has all that info. We just need to turn it into a form we can use. The trickiest bit is the chat format. It is defined in jinja which has embedded functions. It looks like https://docs.rs/minijinja/latest/minijinja/ might work for rust?