search

HomebrewNLP / Olmax

HomebrewNLP in JAX flavour for maintable TPU-Training
BSD 2-Clause "Simplified" License
45 stars 6 forks source link

Alternative Sampling Methods #42

Closed ClashLuke closed 2 years ago

ClashLuke commented 2 years ago

This PR primarily implements alternative methods such as Top-P-X and Top-A from RWKV. I'll evaluate whether they perform better than the baseline, and if they do, merge this PR or perhaps add even more options to the inference code. Once we're confident that our sampling performs well, we can take a stab at alternative data distributions (#5, #9) and automated evaluation (#21).

ClashLuke commented 2 years ago

https://github.com/HomebrewNLP/HomebrewNLP-Jax/pull/42/commits/a67d2e0784498b0fff46c18d86d0be88ad4c57db works fine

ClashLuke commented 2 years ago

..and so does https://github.com/HomebrewNLP/HomebrewNLP-Jax/pull/42/commits/4fdc94b2c0f686a8928ac7781618a3ea41e2bb6e grafik