slow inference with peft model

AI4Finance-Foundation / FinGPT

FinGPT: Open-Source Financial Large Language Models! Revolutionize 🔥 We release the trained model on HuggingFace.

https://ai4finance.org

MIT License

13.84k stars 1.92k forks source link

slow inference with peft model #107

Open wwh0411 opened 11 months ago

wwh0411 commented 11 months ago

I use the code from fingpt benckmark to evaluate my peft model on datasets：fpb/fiqa/tfns respectively. However the inference speed is very slow, and increasing batch size doesn't help with this issue. It seems that when i increase the batch size, the inference speed per iteration increases consequently. Could u please help with this issue?

BruceYanghy commented 11 months ago

Which GPU device are you using?

wwh0411 commented 11 months ago

I‘m testing the model on 2 RTX 3090s

Weiyao-Li commented 7 months ago

Could you try google colab with A100 or TPU. Did you make any progress or still having any issue now?