Speculative Sampling

[manisnesan]

Speculative sampling is a technique used in machine learning and natural language processing. It involves generating multiple possible outputs or continuations of a sequence using a draft model, and then scoring them in parallel using a larger model. This allows for faster decoding and can lead to speedups in large language models. Speculative sampling is used to accelerate transformer decoding and improve the efficiency of language models.

• https://jaykmody.com/blog/speculative-sampling/
• https://www.arxiv-vanity.com/papers/2302.01318/
• https://youtube.com/watch?v=q6oiidmVnwE
• https://aclanthology.org/2021.findings-emnlp.407
• https://medium.com/syncedreview/deepminds-speculative-sampling-achieves-2-2-5x-decoding-speedups-in-large-language-models-aede69823575
• https://paperswithcode.com/paper/speculative-sampling-in-variational
• https://openreview.net/pdf?id=C9NEblP8vS https://github.com/ggerganov/llama.cpp/issues/2030
• https://news.ycombinator.com/item?id=35118704

[manisnesan]

johnowhitaker video on this topic