search

pytorch-labs / gpt-fast

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.
BSD 3-Clause "New" or "Revised" License
5.65k stars 513 forks source link

How is llama-7b trained, what is the verification accuracy? #86

Closed frankxyy closed 8 months ago

frankxyy commented 9 months ago

Hi, I am wondering about the training process of the small model and the verification accuracy. As it has large effects on the decoding effectiveness. Thank you!

Chillee commented 8 months ago

The specdec results here are just using out of the box llama-7B.

frankxyy commented 8 months ago

The specdec results here are just using out of the box llama-7B.

So is there an average speculative decoding length that can pass the verification process?