Coordinate nipype threads with ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS

[gdevenyi]

Several of the commands and scripts inside RABIES use ITK tools, which honour ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS.

Should figure out how to make nipype export this variable during job running so that CPUs are allocated properly.

[gdevenyi]

Need to pass it via env= in Popen https://docs.python.org/3/library/os.html

[Gab-D-G]

will need to pass the right number to env= via subprocess within run_command() https://github.com/CoBrALab/RABIES/blob/2d3f0b281053b41328c8fae9b4112317fe40f555/rabies/utils.py#L354. Can update os.environ dictionary to include ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS key.

[geowk]

Hello,

Has this been resolved?

Recently discovered this using Rabies 0.4.8 during preprocessing. The despike, autobox commands uses 8 threads as designated with --local_threads option. However, when the antsMotionCorr starts, it uses all threads available on our computer system, halting other computation working on the system.

-g

[Gab-D-G]

Hi, this is still unresolved. If you think you can help fixing this issue, we'd definitely welcome contributions (we've got some guidelines in this regard https://rabies.readthedocs.io/en/latest/contributing.html).

[geowk]

Hello,

We were using linux and running rabies with a singularity container rabies 0.4.8. when this problem occurs. Also noticed it with AFNI's command 3dDespike.

We upgraded to 0.5.1 and the work around solution seems to add the line to a user's .bashrc file:

export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=1

That seems to limit each thread used to cap at around 100%. For example, if you added --local_threads 8 as an option in your rabies command, 8 threads will be used at 100%.

Without that line in a user's .bashrc and running rabies with the singularity container, rabies 0.5.1, it will use all threads available on your system with commands like antsMotionCorr, antsRegistration, etc.

For AFNI's command 3dDespike.

Including this line in the .bashrc file prevents the use of all threads available on your system.

export OMP_NUM_THREADS=1

-g

[Gab-D-G]

The .bashrc solution will impose a fixed number of thread to 1 for all operations. This is not ideal.

As indicated above, we want to modify the following function so that it can incorporate ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS: https://github.com/CoBrALab/RABIES/blob/2d3f0b281053b41328c8fae9b4112317fe40f555/rabies/utils.py#L354 This function manages execution of shell commands inside nipype nodes, so it should inherit the appropriate number of threads we want to allocate for a given command.

Here's approximately how this would work:

itk_threads_num = '1'
os.environ['ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS'] = itk_threads_num

process = subprocess.run(
    command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
    check=True,
    shell=True,
    env=os.environ,
    )

Regarding AFNI's despike, it looks like the number of threads can be set with the num_threads parameter https://nipype.readthedocs.io/en/latest/api/generated/nipype.interfaces.afni.preprocess.html#despike. If this doesn't work, it seems like it can also inherit os.environ in a similar manner.

This would be set here https://github.com/CoBrALab/RABIES/blob/2d3f0b281053b41328c8fae9b4112317fe40f555/rabies/preprocess_pkg/bold_main_wf.py#L198