search

pytorch-labs / gpt-fast

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.
BSD 3-Clause "New" or "Revised" License
5.35k stars 484 forks source link

How is llama-7b trained, what is the verification accuracy? #86

Closed frankxyy closed 4 months ago

frankxyy commented 5 months ago

Hi, I am wondering about the training process of the small model and the verification accuracy. As it has large effects on the decoding effectiveness. Thank you!

Chillee commented 4 months ago

The specdec results here are just using out of the box llama-7B.

frankxyy commented 4 months ago

The specdec results here are just using out of the box llama-7B.

So is there an average speculative decoding length that can pass the verification process?