Adapt old mt_sat workflow to new data organization

[agahkarakuzu]

Limited to mt_sat for now

[jcohenadad]

Tested fd047c10dd3fea53a2f0d4b0f47b9a4705c4ca1a and got this error:

  .command.sh: line 7:   361 Killed                  octave --no-gui --eval "addpath(genpath('qMRWrappers')); mt_sat_wrapper('sub-01_ses-003_acq-MTon_MTS_aligned.nii.gz','sub-01_ses-003_acq-MToff_MTS_aligned.nii.gz','sub-01_ses-003_acq-T1w_MTS.nii.gz','sub-01_ses-003_acq-MTon_MTS.json','sub-01_ses-003_acq-MToff_MTS.json','sub-01_ses-003_acq-T1w_MTS.json','mask','sub-01_ses-003_acq-T1w_mask.nii.gz','b1map','sub-01_ses-003_B1plusmap_filtered.nii.gz','b1factor',0.4,'qmrlab_path','null', 'sid','sub-01_ses-003'); exit();"

Full output

```console julien-macbook:~/code/anat-processing $ nextflow run neuromod-process-anat.nf --bids ~/data/neuromod/ -with-report report.html N E X T F L O W ~ version 20.10.0 Launching `neuromod-process-anat.nf` [adoring_mcnulty] - revision: ce7b3c51be WARN: There's no process matching config selector: mtsat_align_inputs WARN: There's no process matching config selector: publish_outputs WARN: There's no process matching config selector: extract_brain -- Did you mean: Extract_Brain? Using Octave in Docker or (if local) from the sys path. Input: /Users/julien/data/neuromod/ qMRflow: MTsat pipeline ======================= ## ## ######## ###### ### ######## ### ### ## ## ## ## ## ## #### #### ## ## ## ## ## ## ### ## ## ###### ## ## ## ## ## ## ## ######### ## ## ## ## ## ## ## ## ## ## ## ## ###### ## ## ## Start time: 2021-04-06T11:27:06.521301-04:00 DATA ==== BIDS option has been enabled. WARN: qMRI protocols will be read from sidecar .json files. OPTIONS ======= [GLOBAL] --------------- Selected platform: octave BET enabled: true B1+ correction enabled: true [ANTs Registration] ------------------- Dimensionality: 3 Metric: MI Weight: 1 Number of bins: 32 Sampling type: Regular Sampling percentage: 0.25 Transform: Rigid[0.1] Convergence: [1000x500x250x100,1e-6,10] Shrink factors: 8x4x2x1 Smoothing sigmas: 3x2x1x0vox [FSL BET] --------------- Enabled: true Fractional intensity threshold: 0.45 Robust brain center estimation: true [qMRLab mt_sat] --------------- WARN: Acquisition protocols will be read from sidecar .json files (BIDS). B1+ correction has been ENABLED. WARN: Process will be skipped for participants missing a B1map file. B1 correction factor: 0.4 ======================= executor > local (92) [97/5a7506] process > Align_Input_Volumes (sub-04_ses-003) [100%] 20 of 20 ✔ [34/28cdc4] process > Extract_Brain (sub-04_ses-003) [100%] 20 of 20 ✔ [1c/2038bf] process > B1_Align (sub-04_ses-003) [100%] 19 of 19 ✔ [5f/324cfd] process > B1_Smooth_With_Mask (sub-03_ses-002) [ 84%] 16 of 19 [- ] process > B1_Smooth_Without_Mask - [be/a93349] process > Fit_MTsat_With_B1map_With_Bet (sub-02_ses-003) [ 0%] 0 of 16 [- ] process > Fit_MTsat_With_B1map_Without_Bet - [- ] process > Fit_MTsat_Without_B1map_With_Bet - [- ] process > Fit_MTsat_Without_B1map_Without_Bet - Error executing process > 'Fit_MTsat_With_B1map_With_Bet (sub-01_ses-003)' Caused by: Process `Fit_MTsat_With_B1map_With_Bet (sub-01_ses-003)` terminated with an error exit status (137) Command executed: git clone https://github.com/qMRLab/qMRWrappers.git cd qMRWrappers sh init_qmrlab_wrapper.sh v1.0.0 cd .. octave --no-gui --eval "addpath(genpath('qMRWrappers')); mt_sat_wrapper('sub-01_ses-003_acq-MTon_MTS_aligned.nii.gz','sub-01_ses-003_acq-MToff_MTS_aligned.nii.gz','sub-01_ses-003_acq-T1w_MTS.nii.gz','sub-01_ses-003_acq-MTon_MTS.json','sub-01_ses-003_acq-MToff_MTS.json','sub-01_ses-003_acq-T1w_MTS.json','mask','sub-01_ses-003_acq-T1w_mask.nii.gz','b1map','sub-01_ses-003_B1plusmap_filtered.nii.gz','b1factor',0.4,'qmrlab_path','null', 'sid','sub-01_ses-003'); exit();" Command exit status: 137 Command output: Checked out v1.0.0 ============================= qMRLab version: v1.0.0 ans = 1 ans = 1 ans = 1 ans = 1 ans = 1 =============== qMRLab::Fit ====================== Operation has been completed: mt_sat ================================================== ----------------------------- Saving fit results... ans = 1 Command error: Cloning into 'qMRWrappers'... Note: checking out 'v1.0.0'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: executor > local (92) [97/5a7506] process > Align_Input_Volumes (sub-04_ses-003) [100%] 20 of 20 ✔ [34/28cdc4] process > Extract_Brain (sub-04_ses-003) [100%] 20 of 20 ✔ [1c/2038bf] process > B1_Align (sub-04_ses-003) [100%] 19 of 19 ✔ [75/41c16b] process > B1_Smooth_With_Mask (sub-04_ses-003) [100%] 16 of 16 [- ] process > B1_Smooth_Without_Mask - [38/d28d81] process > Fit_MTsat_With_B1map_With_Bet (sub-06_ses-003) [ 9%] 1 of 11, failed: 1 [- ] process > Fit_MTsat_With_B1map_Without_Bet - [- ] process > Fit_MTsat_Without_B1map_With_Bet - [- ] process > Fit_MTsat_Without_B1map_Without_Bet - Error executing process > 'Fit_MTsat_With_B1map_With_Bet (sub-01_ses-003)' Caused by: Process `Fit_MTsat_With_B1map_With_Bet (sub-01_ses-003)` terminated with an error exit status (137) Command executed: git clone https://github.com/qMRLab/qMRWrappers.git cd qMRWrappers sh init_qmrlab_wrapper.sh v1.0.0 cd .. octave --no-gui --eval "addpath(genpath('qMRWrappers')); mt_sat_wrapper('sub-01_ses-003_acq-MTon_MTS_aligned.nii.gz','sub-01_ses-003_acq-MToff_MTS_aligned.nii.gz','sub-01_ses-003_acq-T1w_MTS.nii.gz','sub-01_ses-003_acq-MTon_MTS.json','sub-01_ses-003_acq-MToff_MTS.json','sub-01_ses-003_acq-T1w_MTS.json','mask','sub-01_ses-003_acq-T1w_mask.nii.gz','b1map','sub-01_ses-003_B1plusmap_filtered.nii.gz','b1factor',0.4,'qmrlab_path','null', 'sid','sub-01_ses-003'); exit();" Command exit status: 137 Command output: Checked out v1.0.0 ============================= qMRLab version: v1.0.0 ans = 1 ans = 1 ans = 1 ans = 1 ans = 1 =============== qMRLab::Fit ====================== Operation has been completed: mt_sat ================================================== ----------------------------- Saving fit results... ans = 1 Command error: Cloning into 'qMRWrappers'... Note: checking out 'v1.0.0'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b HEAD is now at 44f53ac... ms-->s octave: X11 DISPLAY environment variable not set octave: disabling GUI features warning: function /root/work/qMRLab/src/Common/pulse/sinc.m shadows a core library function warning: called from /root/.octaverc at line 2 column 3 warning: function /root/work/qMRLab/External/NODDI_toolbox_v1.0/ParforProgMonv2/example.m shadows a core library function warning: fopen: 'qMRWrappers/version.txt' found by searching load path warning: fopen: 'qMRWrappers/version.txt' found by searching load path warning: load: '/root/work/qMRLab/docs/iqmr_gethelp.mat' found by searching load path warning: fopen: 'qMRWrappers/version.txt' found by searching load path .command.sh: line 7: 361 Killed octave --no-gui --eval "addpath(genpath('qMRWrappers')); mt_sat_wrapper('sub-01_ses-003_acq-MTon_MTS_aligned.nii.gz','sub-01_ses-003_acq-MToff_MTS_aligned.nii.gz','sub-01_ses-003_acq-T1w_MTS.nii.gz','sub-01_ses-003_acq-MTon_MTS.json','sub-01_ses-003_acq-MToff_MTS.json','sub-01_ses-003_acq-T1w_MTS.json','mask','sub-01_ses-003_acq-T1w_mask.nii.gz','b1map','sub-01_ses-003_B1plusmap_filtered.nii.gz','b1factor',0.4,'qmrlab_path','null', 'sid','sub-01_ses-003'); exit();" Work dir: /Users/julien/code/anat-processing/work/b2/9580dd0ac33baf73421401b2b6b26c Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh` ```

[agahkarakuzu]

@jcohenadad it was most likely a temporary issue with the git clone. Can you try again by passing -resume to your nextflow call and share nextflow.log if the issue persists?

In the new version, I am omitting git operations within processes to avoid such interruptions.

[jcohenadad]

@jcohenadad it was most likely a temporary issue with the git clone. Can you try again by passing -resume to your nextflow call and share nextflow.log if the issue persists?

In the new version, I am omitting git operations within processes to avoid such interruptions.

i encountered the same problem: nexflow.log

[agahkarakuzu]

Thank you @jcohenadad ! Looks like your computer was out of memory. I limited the RAM usage for the failing process. I will think of an elastic configuration depending on the host machine capacity. The bottleneck would be max RAM allocated to your docker containers if you have much more available physically.

[jcohenadad]

Tested 4625b8362c451f7a7e2fc4decffe303a5cbfe6a5 and it now finished successfully 🎉

Completed at: 06-Apr.-2021 12:52:35
Duration    : 1m 43s
CPU hours   : 1.8 (90.4% cached)
Succeeded   : 22
Cached      : 75

Thank you for the fix @agahkarakuzu

[agahkarakuzu]

Cool! I am merging this PR to proceed with adding MTR and MP2RAGE in the process using DSL2 modules.

[jcohenadad]

@agahkarakuzu i suggest to delete merged branch (just did it)-- to avoid ending up with 500 stale branches after a few years...

[agahkarakuzu]

Thanks, I went ahead and checked Automatically delete head branches to ensure that this happens systematically.