FasterDecoding / REST

REST: Retrieval-Based Speculative Decoding, NAACL 2024
Apache License 2.0
166 stars 10 forks source link

How to adjust the maximum token lengths when drafting? #18

Open zomss opened 1 month ago

zomss commented 1 month ago

I am trying to adjust draft tokens by length, explicitly aiming to retrieve tokens with a size greater than four by changing the “long” parameter when calling the search function. However, the draft retriever still returns tokens with lengths shorter than 4.

How can I ensure tokens with lengths longer than four are returned?

zhenyuhe00 commented 1 month ago

Hi, I wonder if you rebuilt the file to install Draftretriver after changing the code.

zomss commented 1 month ago

Thanks. I revise the code for draftretriever, and then it is working.

Siegfried-qgf commented 2 weeks ago

I am trying to adjust draft tokens by length, explicitly aiming to retrieve tokens with a size greater than four by changing the “long” parameter when calling the search function. However, the draft retriever still returns tokens with lengths shorter than 4.

How can I ensure tokens with lengths longer than four are returned?

So how can I get more retrieve tokens in a retrieve sequence? I change the long parameter and It also doesn‘t work