search

AI4Finance-Foundation / FinGPT

FinGPT: Open-Source Financial Large Language Models! Revolutionize 🔥 We release the trained model on HuggingFace.
https://ai4finance.org
MIT License
14.1k stars 1.95k forks source link

slow inference with peft model #107

Open wwh0411 opened 1 year ago

wwh0411 commented 1 year ago

I use the code from fingpt benckmark to evaluate my peft model on datasets：fpb/fiqa/tfns respectively. However the inference speed is very slow, and increasing batch size doesn't help with this issue. It seems that when i increase the batch size, the inference speed per iteration increases consequently. Could u please help with this issue?

BruceYanghy commented 1 year ago

Which GPU device are you using?

wwh0411 commented 1 year ago

I‘m testing the model on 2 RTX 3090s

Weiyao-Li commented 8 months ago

Could you try google colab with A100 or TPU. Did you make any progress or still having any issue now?