ivadomed / MetricsReloaded

A fork of the Project-MONAI/MetricsReloaded repo
Apache License 2.0
3 stars 0 forks source link

Speed up metrics computation by parallelizing across subjects #11

Closed naga-karthik closed 2 months ago

naga-karthik commented 2 months ago

This PR is an initial attempt at parallelizing the computation of metrics across subjects using multiprocessing. The usage is similar to that of sct_run_batch -- an additional arg -jobs has been added to process subjects in parallel.

I even tried parallelizing across labels (i.e. nested parallelization with 1st parallelism across subjects and 2nd level of parallelism across labels for each subject). BUT, this resulted in an error AssertionError: daemonic processes are not allowed to have children -- which essentially means that child processes are not allowed to create further child processes.

Hence, in this version, parallelization is only happening at one level. If we want to parallelize the computation across multiple subjects, we should avoid parallelizing within each subject's computation (i.e. no parallelization within the labels).

@valosekj could you please give this PR a try and see if you see some speed ups?

Related: https://github.com/ivadomed/MetricsReloaded/issues/10

valosekj commented 2 months ago

I fixed the printing issues in my recent commits:

image

I also fixed the failing CI in https://github.com/ivadomed/MetricsReloaded/pull/11/commits/615ed078d132d24232ef9b7831fa87e8b43ca5b7. I do not know how it is possible that the CI worked for main, though.

valosekj commented 2 months ago

I compared main with your branch on site-003 on romane and I found out that the speed is actually very similar (22 minutes vs 18 minutes).

single CPU processing: main: 22m13.9s Screenshot 2024-06-10 at 12 12 10 PM
parallel processing: nk/faster-metrics-computation `-jobs 11`: 18m9.5s Screenshot 2024-06-10 at 2 13 59 PM