Issues with MSMT-CSD - Githubissues

dipy / dipy

DIPY is the paragon 3D/4D+ imaging library in Python. Contains generic methods for spatial normalization, signal processing, machine learning, statistical analysis and visualization of medical images. Additionally, it contains specialized methods for computational anatomy including diffusion, perfusion and structural imaging.

https://dipy.org

Other

703 stars 437 forks source link

Issues with MSMT-CSD #2336

Open ShreyasFadnavis opened 3 years ago

ShreyasFadnavis commented 3 years ago

Needs to be speeded up at the QP front for fitting procedure cvxpy.Minimize needs to change. Currently runs via OSQP I assume? Need to follow-up with CVXPY if warmstart allows initialization for the parameters?
@karanphil could you also take a look at the version of CVXPY in DIPY? Maybe the version we have makes it slower?
We also have people reporting memory issues -- This needs to be investigated deeper. @karanphil did you face any issues similar to this one?

karanphil commented 3 years ago

[ ] Needs to be speeded up at the QP front for fitting procedure cvxpy.Minimize needs to change. Currently runs via OSQP I assume? Need to follow-up with CVXPY if warmstart allows initialization for the parameters?

[ ] @karanphil could you also take a look at the version of CVXPY in DIPY? Maybe the version we have makes it slower?

[ ] We also have people reporting memory issues -- This needs to be investigated deeper. @karanphil did you face any issues similar to this one?

Hi @ShreyasFadnavis, I did not look deeply into the CVXPY part of the msmt-CSD code when I did my modifications on Dipy, but yes, it normaly runs via OSQP. Also, I never had memory issues. As for the CVXPY version, did you have people telling you that the msmt-CSD process was slower with different versions? I always used the same version, so I didn't see such thing. I am very busy right now, sorry if I don't provide answers quickly.

ShreyasFadnavis commented 3 years ago

Hi @karanphil -- Same, I have never had memory issues either :) Can you report the version of CVXPY that you are working with? Maybe this is something we need to investigate further!

p.s. Don't worry about the slow responses!

karanphil commented 3 years ago

Hi @karanphil -- Same, I have never had memory issues either :) Can you report the version of CVXPY that you are working with? Maybe this is something we need to investigate further!

p.s. Don't worry about the slow responses!

I use CVXPY 1.0.31!

ShreyasFadnavis commented 3 years ago

@karanphil thanks! -- @skoudoro, I know that CVXPY is an optional dependency, but is there a way to enforce a version on an optional dependency?

skoudoro commented 3 years ago

I would not recommend enforcing a version.

However, we can block some versions so if you identify which version or range of version is problematic, we can do that.

ps: cvxpy changed their matrices operation convention from version 1.1

kodiweera commented 3 years ago

On multiprocessing of the msmt-csd model:

from pathos.multiprocessing import ProcessPool

vol_shape = dwi.shape[:-1] n_voxels = np.prod(vol_shape) voxel_by_dir = dwi.reshape(n_voxels, dwi.shape[-1])

pool = ProcessPool(nodes=6) # number of cpus. 60 min on 1 cpu will be 10 min on 6 cpu! results = pool.map(mcsd_model.fit, voxel_by_dir)

You can bring the 'results' back to 4D as follow for the interested coefficient, in this case white matter shm coefficients:

wm_shm_coeff = [results[i].shm_coeff for i in range(0,(np.array(results)).shape[0])] wm_shm_2d = np.array(wm_shm_coeff) wm_shm_4d = wm_shm_2d.reshape(xdim, ydim,zdim,#ofcoeff)

Please note that you can use pathos to multiprocess any model function, not just mcsd_model.fit.