search

feifeibear / LLMSpeculativeSampling

Fast inference from large lauguage models via speculative decoding
415 stars 46 forks source link

Refactor the KV Cache logic #7

Closed feifeibear closed 9 months ago