search

thunlp / InfLLM

The code of our paper "InfLLM: Unveiling the Intrinsic Capacity of LLMs for Understanding Extremely Long Sequences with Training-Free Memory"
MIT License
269 stars 21 forks source link

显存占用问题 #11

Closed cat-sun closed 5 months ago

cat-sun commented 5 months ago

qwen1.5-14b模型用一张80g的A100爆显存了，这个需要这么大的显存吗

guyan364 commented 5 months ago

可以尝试调小 topk, cache size(大于等于 topk), chunk size (例如2048/4096）