search

gordonhu608 / MQT-LLaVA

Matryoshka Query Transformer for Large Vision-Language Models
Apache License 2.0
80 stars 11 forks source link

Run it without CUDA? #3

Open Priya-Panchal opened 2 weeks ago

Priya-Panchal commented 2 weeks ago

Hi there,

Your work is awesome. I'm trying to install it on my system for my learning purpose. However, my system cannot support GPU. Can I install it without CUDA?

Your help will be greatly appreciated.

gordonhu608 commented 2 weeks ago

Thank you for your interest in our work. In this case, you can use cpu device to run our model. You may want to hardcode this line of code to device='cpu'. https://github.com/gordonhu608/MQT-LLaVA/blob/92c35d063912fd5e21d64713b710dfb6bdbdf95a/llava/model/builder.py#L26 Then you can run our model according to the README here https://github.com/gordonhu608/MQT-LLaVA/blob/92c35d063912fd5e21d64713b710dfb6bdbdf95a/README.md?plain=1#L74-L96. Then you may able to run it on cpu device. Notice also make sure when installing environment dependencies such as pytorch, you might want to make sure you install a version without requiring gpu in case it will cause trouble for your system which doesn't support cuda.

Priya-Panchal commented 2 weeks ago

Hi @gordonhu608

Thank you so much for your response. That error was resolved but now I'm getting the error below related to CUDA

AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\accelerate\utils\modeling.py", line 286, in set_module_tensor_to_device and torch.device(device).type == "cuda" ^^^^^^^^^^^^^^^^^^^^ RuntimeError: Expected one of cpu, cuda, ipu, xpu, mkldnn, opengl, opencl, ideep, hip, ve, fpga, ort, xla, lazy, vulkan, mps, meta, hpu, mtia, privateuseone device type at start of device string: CPU

gordonhu608 commented 2 weeks ago

The package 'accelerate' needs gpu. So if you are not doing gpu training, don't install it.