PyDesigner terminated with partial output - did it run out of memory?

[AvantiShri]

Describe the Issue

I ran pydesigner (resuming from a partial run), and it terminated (without any messages) at the following stage:

$ docker run -it --rm -v /home/avanti.shrikumar/repos/av_pku_analysis/prototyping/STI60510710_non_compliant/niftis:/input -v /home/avanti.shrikumar/repos/av_pku_analysis/prototyping/STI60510710_non_compliant/pydesigner_out:/output dmri/neurodock pydesigner --verbose --resume --standard --nthreads 2 --output /output /input/dmri_main/5d434276-0b88-4a8b-a5a8-086a7269d1ba_conv.nii.gz /input/dmri_flipped/1ec06ae4-b0fa-4a57-bac6-0af699ff4f64_conv.nii.gz
Insufficient information in .json file to determine Fourier status. Assuming DWI is partial Fourier
[WARNING] Given DWI is partial fourier, overriding --degibbs; no unringing correction will be done to avoid artifacts.Use the "--adv" flag to run forced corrections.
Using 2 threads.
Plotting motion...
dwiextract: [100%] extracting volumes
mrmath: [100%] preloading data for "/output/B0_ALL.mif"
mrmath: [100%] computing mean along axis 3...
mrcalc: [100%] computing: (finite (/output/B0_MEAN.mif) ? /output/B0_MEAN.mif : 0)
dwiextract: [100%] extracting volumes
mrmath: [100%] preloading data for "/output/b1000_ALL.mif"
mrmath: [100%] computing mean along axis 3...
mrcalc: [100%] computing: (finite (/output/b1000_MEAN.mif) ? /output/b1000_MEAN.mif : 0)
mrcalc: [100%] computing: ((log ((/output/S0.mif / /output/S1000.mif)) / 1) > 2)
dwiextract: [100%] extracting volumes
mrmath: [100%] preloading data for "/output/B0_ALL.mif"
mrmath: [100%] computing mean along axis 3...
mrcalc: [100%] computing: (finite (/output/B0_MEAN.mif) ? /output/B0_MEAN.mif : 0)
mrcalc: [WARNING] existing output files will be overwritten
mrcalc: [100%] computing: (/output/brain_mask.nii * /output/csf_mask.nii)
Computing SNR: dwi_raw.nii
   * B0...
   * B1000...
   * B3000...
Computing SNR: dwi_preprocessed.nii
   * B0...
   * B1000...
   * B3000...
Image dwi_preprocessed.nii loaded successfully
Processing with 2 workers...
Protocol(s) detected: DTI, DKI
IRLLS Noise Estimation: [100%]
IRLLS Outlier Detection: [100%]
/output/metrics_qc/fitting/outliers_irlls.nii
Plotting outliers...
Constrained Tensor Fit: [0%]/usr/local/lib/python3.7/dist-packages/cvxpy/problems/problem.py:1386: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
  "Solution may be inaccurate. Try another solver, "
Constrained Tensor Fit: [1%]/usr/local/lib/python3.7/dist-packages/cvxpy/problems/problem.py:1386: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information.
  "Solution may be inaccurate. Try another solver, "
Constrained Tensor Fit: [100%]
DTI Parameters: [100%]
DTI ODF: [100%]

(as an aside, I had to delete the files csf_mask.nii, S0.mif and S1000.mif to get it to run, because otherwise it was throwing an error saying it refused to overrwrite those files).

The output files under "metrics" are as follows: ti_ad.nii dti_fa.nii dti_fe.nii dti_md.nii dti_rd.nii dti_trace.nii DT.nii KT.nii

I understand that there are supposed to be more. I was doing this test on a very lightweight amazon workspace instance with only limited memory - is it possible that pydesigner ran out of memory and that's why it terminated? How much memory should I expect the steps to take?

Thanks, Avanti

[gonzoBlackMamba]

Hi @AvantiShri

Yes, every run you restart there are some files it will refuse to overwrite so you must delete. Apologies.

Regarding the outputs. I did a test run myself using Docker on a DKI set and it failed at the DTI ODF as yours seems to have done as well.

@TheJaeger is there a prior version where the ODF is not generated. It fails in docker with 'tuple index out of range' (see screenshot)

-Hunter

[bmol239]

Hi there, I am also having this issue: DTI ODF: [100%] Traceback (most recent call last): File "/home/bmol239/.local/bin/pydesigner", line 8, in sys.exit(main()) File "/home/bmol239/.local/lib/python3.8/site-packages/designer/pydesigner.py", line 965, in main dp.fit_regime( File "/home/bmol239/.local/lib/python3.8/site-packages/designer/fitting/dwipy.py", line 2784, in fit_regime dti_odfs = dtimodel.dtiodf() File "/home/bmol239/.local/lib/python3.8/site-packages/designer/tractography/odf.py", line 375, in dtiodf odf = vectorize(odf, self.mask_img) File "/home/bmol239/.local/lib/python3.8/site-packages/designer/system/utils.py", line 35, in vectorize img.shape[2]), IndexError: tuple index out of range