ImportError: cannot import name 'BatchedKvCache' from 'punica'

punica-ai / punica

Serving multiple LoRA finetuned LLM as one

https://arxiv.org/abs/2310.18547

Apache License 2.0

883 stars 40 forks source link

ImportError: cannot import name 'BatchedKvCache' from 'punica' #36

Open VisionwWu opened 5 months ago

VisionwWu commented 5 months ago

Everything go well when I install punica from binary package. However, it shows "ImportError: cannot import name 'BatchedKvCache' from 'punica'" when I run "python -m benchmarks.bench_textgen_lora --system punica --batch-size 32". How can I fix this?

VisionwWu commented 5 months ago

oh，I have solved it