sampling: fused speculative sampling kernels - Githubissues

flashinfer-ai / flashinfer

FlashInfer: Kernel Library for LLM Serving

https://flashinfer.ai

Apache License 2.0

768 stars 64 forks source link

sampling: fused speculative sampling kernels #259

Closed yzh119 closed 1 month ago

yzh119 commented 1 month ago

~~[ ] Fused chain verify sampling~~ (left for next pr).
[x] Fused tree verify sampling
[x] Renorm top-p, top-k