search

timothygrant80 / cisTEM

Other
32 stars 27 forks source link

How do I request that cisTEM utilizes multiple GPU's? #432

Closed amytsai0128 closed 2 years ago

amytsai0128 commented 2 years ago

Hi all, I'm interested in using cisTEM template matching with NVIDIA A100's to improve the speed. Using the GUI, I'm not sure where to specify how many GPU's the job can use. I noticed in the elife paper that multiple gpu's can be used, can someone help me out in directing how to do this? Please let me know, thank you and I apologize if this is really obvious in the documentation.

jojoelfe commented 2 years ago

Hi, this might not be documented yet. You can definitively use multiple GPUs spread out over several machines. The general idea is that in the run profile you want to generate one command per GPU. For example if you run on a single machine with two GPUs you would have two commands:

If you want to spread out over several machines you can use ssh like this

We found that in general you will want to use 2-4 threads per command. Check the gpu utilization while match_template is running and increase the number of thread if the utilization is below 100%.

Hope this helps!

jojoelfe commented 2 years ago

In case you want to setup many GPUs, I have this script to help with the typing:

https://github.com/jojoelfe/pycistem/blob/406b3dcc1fea0f4d64b91e7f007ffd13dd158efb/tests/generate_runprofile.py

amytsai0128 commented 2 years ago

Thanks for responding! So just making sure...if the command currently outputted by the gui looks like this:

/local1/workdir1/git/cisTEM_alpha/match_template 128.84.180.31,172.17.0.1,192.168.122.1,127.0.0.1 3001 3142016207418677 2&

Would I be running the export command right before this? Thank you for such a fast response!

jojoelfe commented 2 years ago

Yes, that then should read export CUDA_VISIBLE_DEVICES=0 && /local1/workdir1/git/cisTEM_alpha/match_template 128.84.180.31,172.17.0.1,192.168.122.1,127.0.0.1 3001 3142016207418677 2&

Below is a screenshot how it should look in the settings pane:

image

amytsai0128 commented 2 years ago

Thank you! Forgive me, it's my first time running cisTEM. I want to run this in the background using tmux/screen - but the moment the GUI is closed out/I put my laptop to sleep, the process it's running gets cancelled too. Is it possible for me to run match_templates directly on command line without the use of the GUI, and to also make it so that it can utilize the GPU's like we just talked about earlier? I was able to set up the run profile like you mentioned, but am wondering if there's a workaround to let cisTEM run in the background.

bHimes commented 2 years ago

Please see the cistem FAQ where this is explained.

bHimes commented 2 years ago

Please see the cistem FAQ where this is explained.

While this FAQ suggests x2go as an alternative, I would advise nomachine as it is easier for many to manage.

jojoelfe commented 2 years ago

No, at the moment the only way to use multiple GPUs is through the GUI and as you noticed the GUI needs to stay open.

We normally run the GUI on one of our workstations that is always on, and then use something like NoMachine to check on it with a laptop.

I'm going to close this issue, since the run profile part is resolved. Feel free to open another one if you have other questions.