search

bigscience-workshop / petals

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
https://petals.dev
MIT License
8.89k stars 490 forks source link

DynamicCache and Beam Search #571

Open artek0chumak opened 2 months ago

artek0chumak commented 2 months ago

Make sure that RemotePastKeyValues will reorder beams when the method reorder_cache is called. Related: #570