search

FreedomIntelligence / LLMZoo

⚡LLM Zoo is a project that provides data, models, and evaluation benchmark for large language models.⚡
Apache License 2.0
2.93k stars 201 forks source link

Is is possible to run the pre-trained model with rtx 2060 6g? #15

Open dream-desktop opened 1 year ago

dream-desktop commented 1 year ago

My computer has the rtx 2060 6G, the results say my gpu is lack, is there any way to run the phoenix model. thank you.

zhjohnchan commented 1 year ago

Hi @dream-desktop,

Thanks for your attention. Currently, 6G is not sufficient for running Phoenix. We will work on the Quantization version.

Or you can try Phoenix using cpu:

python -m llmzoo.deploy.cli --model-name FreedomIntelligence/phoenix-inst-chat-7b --device cpu

which requires ~30GB of CPU memory.

Best, Zhihong

wabyking commented 1 year ago

Please check if this helps: https://gist.github.com/rain-1/8cc12b4b334052a21af8029aa9c4fafc @dream-desktop

wabyking commented 1 year ago

It is for LLaMA based models