feat: add top-A sampling

turboderp / exllamav2

A fast inference library for running LLMs locally on modern consumer-class GPUs

MIT License

3.28k stars 243 forks source link

feat: add top-A sampling #239

Closed AlpinDale closed 7 months ago

AlpinDale commented 7 months ago

This PR adds support for Top-A sampling (invented by BlinkDL from the RWKV team). Based on the min_p implementation here, since min_p is just linear top_a.