why inference with kv cache have problems?

evo-design / evo

Biological foundation modeling from molecular to genome scale

Apache License 2.0

1.14k stars 137 forks source link

why inference with kv cache have problems? #82

Open objectnoppppppp opened 2 months ago

objectnoppppppp commented 2 months ago

I have fintune Evo with lora and then inference with cache on, i find that the inference with cache on was not right because i find that it's right when i turn the cache off. I don't know what's wrong with the cache.