search

Lightning-AI / litgpt

Pretrain, finetune, deploy 20+ LLMs on your own data. Uses state-of-the-art techniques: flash attention, FSDP, 4-bit, LoRA, and more.
https://lightning.ai
Apache License 2.0
6.85k stars 726 forks source link

Feature/top p sampling #1360

Closed belerico closed 2 weeks ago

belerico commented 3 weeks ago

This PR adds the nucleus-sampling (aka top-p sampling) as specified from https://arxiv.org/abs/1904.09751. In top-p sampling the next token is chosen from the smallest set of tokens with a cumulative probability greater than top-p, i.e. by selecting the highest probability tokens whose cumulative probability exceeds the top-p threshold.

rasbt commented 2 weeks ago

Thanks for all the updates and fixes. It looks all great to me now. ~The only thing is perhaps adding one more unit test, but I can take care of that to make it easier.~ [done]

belerico commented 2 weeks ago

Thanks for all the updates and fixes. It looks all great to me now. ~The only thing is perhaps adding one more unit test, but I can take care of that to make it easier.~ [done]

Thank you @rasbt: i had missed your comment

rasbt commented 2 weeks ago

No worries at all, I also thought it was probably quicker to just add instead of explain 😅