search

JenniferHo97 / llm_factoscope

LLM Factoscope: Uncovering LLMs’ Factual Discernment through Measuring Inner States
MIT License
7 stars 1 forks source link

Fail to use GPU! #1

Open Tan-Hexiang opened 2 months ago

Tan-Hexiang commented 2 months ago

When I am running 'construct_data.py', I find the code is running at cpu which leads to very slow processing. However can I run this code at GPU?

I have carefully check the availability of the gpu in my device and the environments. Could you help me fix this question?

Tan-Hexiang commented 2 months ago

@JenniferHo97

JenniferHo97 commented 2 months ago

Our code is designed to run on the GPU and does so automatically in our experimental environment. You can explicitly move the model and data to the GPU by calling model.to(cuda') or tensor.to(cuda') in your code. Additionally, you can test if the GPU is being used when generating data with other LLMs to ensure the GPU is functioning correctly.

Tan-Hexiang commented 2 months ago

Our code is designed to run on the GPU and does so automatically in our experimental environment. You can explicitly move the model and data to the GPU by calling model.to(cuda') or tensor.to(cuda') in your code. Additionally, you can test if the GPU is being used when generating data with other LLMs to ensure the GPU is functioning correctly.

I tried using Hugging Face to call the GPU, which worked, indicating that the GPU on the machine is available. However, construct_data.py does not automatically use the GPU. I suspect it might be an issue with Ecco. I noticed the code uses this link. My understanding is that Ecco should automatically place the model and input_text on the GPU, but this doesn’t seem to work for me. Some important package versions I’m using are: ecco==0.1.2 transformers==4.44.2

Could you share the full pip requirements for your environment? Maybe some version mismatch is causing this bug.

I also tried explicitly using model.to('cuda'), but it seems that the Ecco model this link does not support this approach.

JenniferHo97 commented 2 months ago

Thank you for the update. Please use the Ecco library provided in our repository. You can place the ecco folder directly in your project. We have already implemented model.to('cuda') in our version of the library, which should resolve the GPU utilization problem you're encountering.

Tan-Hexiang commented 2 months ago

Thank you very much for your timely and detailed response! I have just successfully run the code on the GPU. Also, thsnks for sharing the code for this work, which is very meaningful and interesting.