search

KoboldAI / KoboldAI-Client

For GGUF support, see KoboldCPP: https://github.com/LostRuins/koboldcpp
https://koboldai.com
GNU Affero General Public License v3.0
3.52k stars 761 forks source link

Feature request: Top-A Sampling #115

Closed shinkarom closed 2 years ago

shinkarom commented 2 years ago

https://naidb.miraheze.org/wiki/NovelAI_Features defines Top-A Sampling as

Top-A considers the probability of the most likely Token, and sets a limit based on its percentage. After this, remaining tokens are compared to this limit. If their probability is too low, they are removed from the pool.

The calculation is as follows: limit = max(token_probs)^2 * A.

Increasing A results in a stricter limit. Lowering A results in a looser limit.

This means that if the top token has a moderate likelihood of appearing, the pool of possibilities will be large. On the other hand, if the top token has a very high likelihood of appearing, then the pool will be 1-3 tokens at most. This ensures that structure remains solid, and focuses creative output in areas where it is actually wanted.

henk717 commented 2 years ago

It has been merged in the United version of KoboldAI which you can use to test it. You can find this in the version dropdown on colab or in your KoboldAI Updater.

shinkarom commented 2 years ago

Top-A sampling seems a real gamechanger. Thank you very much.