Can we run 4bit Model on RTX4090 with 25GB memory?

InternLM / InternLM-XComposer

InternLM-XComposer2 is a groundbreaking vision-language large model (VLLM) excelling in free-form text-image composition and comprehension.

1.92k stars 121 forks source link

Open marks202309 opened 2 months ago

marks202309 commented 2 months ago

also is it possible that we can do some improvements on gradio_demo_chat.py

like sd-webui, models are not hard-coded and can be selected from UI.

LoFiApostasy commented 2 months ago

Here is the memory footprint on my 3090 using the 4bit model.