search

LBEM-CH / focus

High Throughput Electron Microscopy Image Processing Software
http://www.focus-em.org
GNU General Public License v2.0
35 stars 14 forks source link

multu GPU #218

Open Eladbic opened 6 years ago

Eladbic commented 6 years ago

Focus v 1.1.0 I have 4 1080 TI GPU in my system. When I run 1 job (unpack, motioncor2, gCTF, Iciness, pick) everything work perfect. When I try to run 4 parallel job with "gpu in system" setting cycle over GPUs = "yes" I get an error and I can see in nvidia-smi that all jobs are on one GPU (gpu 0).

Any idea?

Thanks,

Eladbic commented 6 years ago

I'm using openmpi/2.1.2/bin/mpirun do I need different mpi?

sthennin commented 6 years ago

Hi, On our system with four 1080 Ti GPUs, we have the highest throughput when NOT cycling over the GPUs, but always using all four simultaneously for each job. That means that you need a MotionCor2 version where you can set the RAM allocation to zero (the newest version can do that). You can then use as settings in the PREFERENCES window under the "System" tab:

How many GPUs to use = "4" Which GPU to use = "0 1 2 3" Cycle over GPUs = "No" Run only one MotionCor2 instance at a time on the system = "No"

Hope this helps,

Henning.

rdrighetto commented 6 years ago

Just to add to Henning's comment, running MotionCor2 (regardless of the parallelization scheme used) has nothing to do with OpenMPI so that's nothing to worry about. I believe the behavior of "cycle over GPUs" has changed recently, please let us know more details about the error if for some reason you need to use this option.

Best wishes,

-- Ricardo Diogo Righetto

2018-04-16 15:51 GMT+02:00 sthennin notifications@github.com:

Hi, On our system with four 1080 Ti GPUs, we have the highest throughput when NOT cycling over the GPUs, but always using all four simultaneously for each job. That means that you need a MotionCor2 version where you can set the RAM allocation to zero (the newest version can do that). You can then use as settings in the PREFERENCES window under the "System" tab:

How many GPUs to use = "4" Which GPU to use = "0 1 2 3" Cycle over GPUs = "No" Run only one MotionCor2 instance at a time on the system = "No"

Hope this helps,

Henning.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/C-CINA/focus/issues/218#issuecomment-381605549, or mute the thread https://github.com/notifications/unsubscribe-auth/AHVvf4t6XTITvy1qnXH1lg1fKnA_KKM8ks5tpKH7gaJpZM4TQwOz .

Eladbic commented 6 years ago

Thanks for your reply,

I'm running the most update motioncor2 (1.0.5 cuda8.0) but I don't have "Run only one MotionCor2 instance at a time on the system = "No" option and it the GPU in the system tab look like this:

also in the "which GPU to use" I can insert just one number.

I'm using focus version 1.1.0 on CentOS 6

Thanks for the help,

On Mon, Apr 16, 2018 at 8:51 AM, sthennin notifications@github.com wrote:

Hi, On our system with four 1080 Ti GPUs, we have the highest throughput when NOT cycling over the GPUs, but always using all four simultaneously for each job. That means that you need a MotionCor2 version where you can set the RAM allocation to zero (the newest version can do that). You can then use as settings in the PREFERENCES window under the "System" tab:

How many GPUs to use = "4" Which GPU to use = "0 1 2 3" Cycle over GPUs = "No" Run only one MotionCor2 instance at a time on the system = "No"

Hope this helps,

Henning.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FC-CINA%2Ffocus%2Fissues%2F218%23issuecomment-381605549&data=02%7C01%7Celad.binshtein%40vanderbilt.edu%7C771c80c578cd43eb421008d5a3a13830%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636594835199275122&sdata=Ixv6Z%2FTVUGQ4gpNkh5SILMyfdErvyJm3ic2ajIHodrE%3D&reserved=0, or mute the thread https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAcBV5gcrqAbulM5PRupJpa3HHAjoQYVYks5tpKH8gaJpZM4TQwOz&data=02%7C01%7Celad.binshtein%40vanderbilt.edu%7C771c80c578cd43eb421008d5a3a13830%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636594835199275122&sdata=%2FEqgOpGvaylGKKBgVeZkxQ2dcTdS%2Bh0yuLfkZSE6S5Y%3D&reserved=0 .

--

Elad Binshtein, Ph.D. Cryo EM specialist - staff scientist Center for Structure Biology (CSB) MCN Room 1207 Vanderbilt University Nashville, TN Office: +1-615-322-4671 Mobile: +1-615-481-4408 https://www.linkedin.com/in/elad-binshtein-6bb89428/ E-Mail: elad.binshtein@vanderbilt.edu eladbi@gmail.com

sthennin commented 6 years ago

... I see. But you should be able to enter more than one number in "which GPU to use". Simply separate them by a space, or maybe try "0123". Which version of FOCUS are you using? Maybe, a "git pull" and re-compilation is helping?

This is how the tab should look like:

screen shot 2018-04-16 at 18 44 38
Eladbic commented 6 years ago

I'm running FOCUS 1.1.0 from sbgrid. In "which GPU to use" I can enter just one number with max of 2 digit with no space. I will try to install a local FOCUS to see if it behave different.

Thanks,

On Mon, Apr 16, 2018 at 11:48 AM, sthennin notifications@github.com wrote:

... I see. But you should be able to enter more than one number in "which GPU to use". Simply separate them by a space, or maybe try "0123". Which version of FOCUS are you using? Maybe, a "git pull" and re-compilation is helping?

This is how the tab should look like: [image: screen shot 2018-04-16 at 18 44 38] https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fuser-images.githubusercontent.com%2F6606048%2F38823283-43520552-41a6-11e8-85e7-b2cd91dff8b3.png&data=02%7C01%7Celad.binshtein%40vanderbilt.edu%7Cd85efa4781c24086fcdc08d5a3b9d48f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636594940920636038&sdata=tff0LYOb6Ap5c2J6P0wsmzs2bPLkg8Gs98ubVhfW%2F54%3D&reserved=0

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FC-CINA%2Ffocus%2Fissues%2F218%23issuecomment-381671791&data=02%7C01%7Celad.binshtein%40vanderbilt.edu%7Cd85efa4781c24086fcdc08d5a3b9d48f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636594940920636038&sdata=%2BToYEboZzYj0TfwyrGdOgB%2B1drwbIjK3RnhcqjZPvKg%3D&reserved=0, or mute the thread https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAcBV5niBJurURoqJrmBUiB_xR59_hLXkks5tpMtGgaJpZM4TQwOz&data=02%7C01%7Celad.binshtein%40vanderbilt.edu%7Cd85efa4781c24086fcdc08d5a3b9d48f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636594940920636038&sdata=rdE%2Ba2a9G4IaP8vKalns2wHUMWEwUYm%2B7EPxEJOK1nU%3D&reserved=0 .

--

Elad Binshtein, Ph.D. Cryo EM specialist - staff scientist Center for Structure Biology (CSB) MCN Room 1207 Vanderbilt University Nashville, TN Office: +1-615-322-4671 Mobile: +1-615-481-4408 https://www.linkedin.com/in/elad-binshtein-6bb89428/ E-Mail: elad.binshtein@vanderbilt.edu eladbi@gmail.com

Eladbic commented 6 years ago

Hi I have install FOCUS 1.1.0 nightly revision 37778f8 (centos7) and new the system tab have the extra option and "which GPU to use" can get any number. I can work in parallel and run all the script and but motioncor2 (v 1.0.5) that have this error:

Traceback (most recent call last): File "/opt/focus/scripts/proc/apply_dose_filter.py", line 4, in from mrcz import ioMRC ImportError: No module named mrcz

Did I missed anything in the install?

Thanks,

On Tue, Apr 17, 2018 at 12:38 PM, Elad Binshtein < elad.binshtein@vanderbilt.edu> wrote:

I'm running FOCUS 1.1.0 from sbgrid. In "which GPU to use" I can enter just one number with max of 2 digit with no space. I will try to install a local FOCUS to see if it behave different.

Thanks,

On Mon, Apr 16, 2018 at 11:48 AM, sthennin notifications@github.com wrote:

... I see. But you should be able to enter more than one number in "which GPU to use". Simply separate them by a space, or maybe try "0123". Which version of FOCUS are you using? Maybe, a "git pull" and re-compilation is helping?

This is how the tab should look like: [image: screen shot 2018-04-16 at 18 44 38] https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fuser-images.githubusercontent.com%2F6606048%2F38823283-43520552-41a6-11e8-85e7-b2cd91dff8b3.png&data=02%7C01%7Celad.binshtein%40vanderbilt.edu%7Cd85efa4781c24086fcdc08d5a3b9d48f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636594940920636038&sdata=tff0LYOb6Ap5c2J6P0wsmzs2bPLkg8Gs98ubVhfW%2F54%3D&reserved=0

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FC-CINA%2Ffocus%2Fissues%2F218%23issuecomment-381671791&data=02%7C01%7Celad.binshtein%40vanderbilt.edu%7Cd85efa4781c24086fcdc08d5a3b9d48f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636594940920636038&sdata=%2BToYEboZzYj0TfwyrGdOgB%2B1drwbIjK3RnhcqjZPvKg%3D&reserved=0, or mute the thread https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAcBV5niBJurURoqJrmBUiB_xR59_hLXkks5tpMtGgaJpZM4TQwOz&data=02%7C01%7Celad.binshtein%40vanderbilt.edu%7Cd85efa4781c24086fcdc08d5a3b9d48f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636594940920636038&sdata=rdE%2Ba2a9G4IaP8vKalns2wHUMWEwUYm%2B7EPxEJOK1nU%3D&reserved=0 .

--

Elad Binshtein, Ph.D. Cryo EM specialist - staff scientist Center for Structure Biology (CSB) MCN Room 1207 Vanderbilt University Nashville, TN Office: +1-615-322-4671 Mobile: +1-615-481-4408 https://www.linkedin.com/in/elad-binshtein-6bb89428/ E-Mail: elad.binshtein@vanderbilt.edu eladbi@gmail.com

--

Elad Binshtein, Ph.D. Cryo EM specialist - staff scientist Center for Structure Biology (CSB) MCN Room 1207 Vanderbilt University Nashville, TN Office: +1-615-322-4671 Mobile: +1-615-481-4408 https://www.linkedin.com/in/elad-binshtein-6bb89428/ E-Mail: elad.binshtein@vanderbilt.edu eladbi@gmail.com

rdrighetto commented 6 years ago

Hi,

You need to install the Python module mrcz. In a terminal, type:

pip install mrcz

(the same procedure can be used to solve other Python dependencies should they arise)

Just make sure that the 'pip' command corresponds to the same Python distribution used in Focus (that is specified in the Preferences >> Software tab, also shown at the beginning of the output when the script is run).

Hope this helps,

-- Ricardo Diogo Righetto

2018-04-19 16:36 GMT+02:00 Elad Binshtein notifications@github.com:

Hi I have install FOCUS 1.1.0 nightly revision 37778f8 (centos7) and new the system tab have the extra option and "which GPU to use" can get any number. I can work in parallel and run all the script and but motioncor2 (v 1.0.5) that have this error:

Traceback (most recent call last): File "/opt/focus/scripts/proc/apply_dose_filter.py", line 4, in from mrcz import ioMRC ImportError: No module named mrcz

Did I missed anything in the install?

Thanks,

On Tue, Apr 17, 2018 at 12:38 PM, Elad Binshtein < elad.binshtein@vanderbilt.edu> wrote:

I'm running FOCUS 1.1.0 from sbgrid. In "which GPU to use" I can enter just one number with max of 2 digit with no space. I will try to install a local FOCUS to see if it behave different.

Thanks,

On Mon, Apr 16, 2018 at 11:48 AM, sthennin notifications@github.com wrote:

... I see. But you should be able to enter more than one number in "which GPU to use". Simply separate them by a space, or maybe try "0123". Which version of FOCUS are you using? Maybe, a "git pull" and re-compilation is helping?

This is how the tab should look like: [image: screen shot 2018-04-16 at 18 44 38] https://na01.safelinks.protection.outlook.com/?url= https%3A%2F%2Fuser-images.githubusercontent.com% 2F6606048%2F38823283-43520552-41a6-11e8-85e7-b2cd91dff8b3. png&data=02%7C01%7Celad.binshtein%40vanderbilt.edu% 7Cd85efa4781c24086fcdc08d5a3b9d48f%7Cba5a7f39e3be4ab3b45067fa80fa ecad%7C0%7C0%7C636594940920636038&sdata=tff0LYOb6Ap5c2J6P0wsmzs2bPLkg8 Gs98ubVhfW%2F54%3D&reserved=0

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://na01.safelinks.protection.outlook.com/?url= https%3A%2F%2Fgithub.com%2FC-CINA%2Ffocus%2Fissues%2F218% 23issuecomment-381671791&data=02%7C01%7Celad.binshtein%40vanderbilt.edu% 7Cd85efa4781c24086fcdc08d5a3b9d48f%7Cba5a7f39e3be4ab3b45067fa80fa ecad%7C0%7C0%7C636594940920636038&sdata=%2BToYEboZzYj0TfwyrGdOgB% 2B1drwbIjK3RnhcqjZPvKg%3D&reserved=0, or mute the thread https://na01.safelinks.protection.outlook.com/?url= https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe- auth%2FAcBV5niBJurURoqJrmBUiB_xR59_hLXkks5tpMtGgaJpZM4TQwOz& data=02%7C01%7Celad.binshtein%40vanderbilt.edu% 7Cd85efa4781c24086fcdc08d5a3b9d48f%7Cba5a7f39e3be4ab3b45067fa80fa ecad%7C0%7C0%7C636594940920636038&sdata=rdE%2Ba2a9G4IaP8vKalns2wHUMWEwUYm% 2B7EPxEJOK1nU%3D&reserved=0 .

--

Elad Binshtein, Ph.D. Cryo EM specialist - staff scientist Center for Structure Biology (CSB) MCN Room 1207 Vanderbilt University Nashville, TN Office: +1-615-322-4671 Mobile: +1-615-481-4408 https://www.linkedin.com/in/elad-binshtein-6bb89428/ E-Mail: elad.binshtein@vanderbilt.edu eladbi@gmail.com

--

Elad Binshtein, Ph.D. Cryo EM specialist - staff scientist Center for Structure Biology (CSB) MCN Room 1207 Vanderbilt University Nashville, TN Office: +1-615-322-4671 Mobile: +1-615-481-4408 https://www.linkedin.com/in/elad-binshtein-6bb89428/ E-Mail: elad.binshtein@vanderbilt.edu eladbi@gmail.com

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/C-CINA/focus/issues/218#issuecomment-382760462, or mute the thread https://github.com/notifications/unsubscribe-auth/AHVvf1uRgTrZq5uKv9P_A40Su2_2p9wDks5tqKDUgaJpZM4TQwOz .