Open JuliaMerz opened 10 months ago
Alright, I finally dug into this properly. I got some debugs on the configured sampler, and was able to reconstruct our default implementation from a string.
-s "repetition:penalty=1.3:last_n=64 topk:k=40:min_keep=1 topp:p=0.95:min_keep=1 temperature:0.8"
creates the same configured sampler as our default.
I'll submit a small PR to throw this into a comment just for future users trying to learn about the code. I'll add it to the CLI help as well.
https://github.com/rustformers/llm/blob/18b2a7d37e56220487e851a45badc46bf9dcb9d3/crates/llm-base/src/samplers.rs#L95
We have a default for ConfiguredSamplers, as well as a FromStr implementation for users to give in a sampler string. However, at the moment we do not have that default represented anywhere. It would be useful to write the default string down somewhere so users can edit and use it themselves.
An LLM-based conversion of the default leads to this (which matches the structure in the default code correctly):
However, the default (and therefore this string) contains both mirostat samplers, but sets
mirostat1
andmirostat2
to false:https://github.com/rustformers/llm/blob/18b2a7d37e56220487e851a45badc46bf9dcb9d3/crates/llm-base/src/samplers.rs#L167-L169
So there's something going on that I'm missing, or there's something off about the current default implementation.