search

hemingkx / Spec-Bench

Spec-Bench: A Comprehensive Benchmark and Unified Evaluation Platform for Speculative Decoding (ACL 2024 Findings)
https://sites.google.com/view/spec-bench
Apache License 2.0
107 stars 11 forks source link

PaSS methodology. #2

Closed jivanph closed 4 months ago

jivanph commented 4 months ago

Thank you so much for your work in establishing this benchmark!

I wanted to ask if you would include PaSS (https://arxiv.org/pdf/2311.13581.pdf) to your benchmark analysis, since it is included in your references, and it allows nucleus sampling, greedy sampling, but it doe require the training of extra token embeddings.

hemingkx commented 4 months ago

Thank you for your suggestion!

Unfortunately, we have not been able to find an open-source implementation of PaSS, nor have we found the pretrained weights for the token embeddings required to reproduce this method. We are keen to include PaSS in Spec-Bench if these resources become available.

jivanph commented 4 months ago

Thank you for your response. This makes a lot of sense.