Open ucsdzehualiu opened 4 months ago
If you are loading the 8B model in bfloat16 then it won't fit in 1 GPU as it took 15.5 GB of memory. The quantized versions 8bit/4bit can fit in 1 T4
Here is the memory requirement for bfloat16 in kaggle notebook
And here is the requirement for 4 bit quantized model using nf4.
any idea how to load the quantized version?
as titled,what is the minimun hardware requirement for 8b and 70b