llama: check significance of sampling params `min_keep`

alpaca-core / ac-local

Alpaca Core local inference SDK

MIT License

1 stars 0 forks source link

llama: check significance of sampling params `min_keep` #15

Closed iboB closed 1 month ago

iboB commented 3 months ago

sparams.min_keep overrides the minimum tokens kept for the different sampling strategies (for example overriding top_k, or adding tokens with prob less than top_p).

It looks strange and doesn't seem to be used. Should we keep it in the interface?

It's currently disabled

pminev commented 2 months ago

Feature (min_keep option) PR intro.

pminev commented 2 months ago

Firstly min_keep was always 1. Then in this PR was introduces usage of n_probs instead.

After a while it seems to be wrong to use n_probs and it was removed here. However it was still needed and it was added again but with new option (commit).

iboB commented 1 month ago

It was better documented and improved in upstream