search

CoBrALab / MAGeTbrain

Multiple Automatically Generated Templates brain segmentation algorithm
Other
48 stars 25 forks source link

Error in mincblur #52

Closed maximedieudonne closed 4 years ago

maximedieudonne commented 4 years ago

Hi, thanks a lot for your project, it is very interesting! however i didn't succes to run the pipeline, maybe you could help. I have installed all the dependencies mincstuff, minctool kit V2 pyminc, perl gnu parralel. I had to remove manualy some @array in the file 'parallel' because of the version of perl. I work on my personal laptop, and i just want to test with one image. I have only one template too, this is just a simple test.

my command is : mb run register -q parallel I get this errors in my log : Error in mincblur in file /home/nistmni/src/mni_autoreg/mincblur/mincblur.c, line 301 problems reading `/tmp/bestlinreg_maget-gbigNVBa/s_brain3_t1_resampled.mnc'. Error in mincblur in file /home/nistmni/src/mni_autoreg/mincblur/mincblur.c, line 301 problems reading `/tmp/bestlinreg_maget-ynSG4ZBf/s_brain1_t1_resampled.mnc'.

What I have missed ?

Thanks a lot,

Maxime Dieudonné

Here is the end of the log and the entire log, i don't know if it could help: end of the lof : Traceback (most recent call last): File "/home/maxime/hpc/src/MAGeTbrain/bin/mb", line 792, in main() File "/home/maxime/hpc/src/MAGeTbrain/bin/mb", line 220, in main options.func(options) File "/home/maxime/hpc/src/MAGeTbrain/bin/mb", line 329, in command_run p.run() File "/home/maxime/hpc/src/MAGeTbrain/bin/mb", line 608, in run self.queue.run(stages=self._get_stage_order()) File "/home/maxime/hpc/src/MAGeTbrain/bin/mb", line 508, in run self.parallel(self.commands[stage]) File "/home/maxime/hpc/src/MAGeTbrain/bin/mb", line 514, in parallel self.execute(command, input='\n'.join(commands)) File "/home/maxime/hpc/src/MAGeTbrain/bin/mb", line 492, in execute raise Exception("Returns %i :: %s" % (proc.returncode, command)) Exception: Returns 5 :: parallel -j8

All the log (i remove intermediate lines of the dilatation processus, reading volumen processus to be more digest) : 5 atlases, 1 templates, 1 subjects found ++ mktemp -d

Error in mincblur in file /home/nistmni/src/mni_autoreg/mincblur/mincblur.c, line 301 problems reading `/tmp/bestlinreg_maget-gbigNVBa/s_brain3_t1_resampled.mnc'. Died at /home/maxime/hpc/src/MAGeTbrain/bin/bestlinreg_maget line 373.** Reading Volume: 15% done. (147/1000) Time: 20 sec out of approx 2.3 min Reading Volume: 100% done. (1000/1000) Time: 2 min out of approx 2 min Reading Volume: DONE in 2 min Binarise: ..................................................................... Dilation: 1% done. (6/509) Time: 21.4 sec out of approx 32.4 min Dilation: 100% done. (509/509) Time: 12 min out of approx 12 min Dilation: DONE in 12.1 min Dilation: 2% done. (12/509) Time: 20.1 sec out of approx 14.4 min Dilation: 100% done. (509/509) Time: 10.3 min out of approx 10.3 min Dilation: DONE in 10.4 min Dilation: 3% done. (13/509) Time: 20.4 sec out of approx 13.6 min Dilation: 100% done. (509/509) Time: 10.3 min out of approx 10.3 min Dilation: DONE in 10.3 min Dilation: 3% done. (17/509) Time: 20.5 sec out of approx 10.2 min Dilation: 100% done. (509/509) Time: 12 min out of approx 12 min Dilation: DONE in 12.2 min Dilation: 1% done. (7/509) Time: 20.7 sec out of approx 26.7 min Dilation: 100% done. (509/509) Time: 11.3 min out of approx 11.3 min Dilation: DONE in 11.6 min Dilation: 1% done. (3/509) Time: 22.2 sec out of approx 66.7 min Dilation: 100% done. (509/509) Time: 14.7 min out of approx 14.7 min Dilation: DONE in 14.8 min Dilation: 1% done. (6/509) Time: 20.6 sec out of approx 31.2 min Dilation: 100% done. (509/509) Time: 11.9 min out of approx 11.9 min Dilation: DONE in 12.1 min Dilation: 2% done. (12/509) Time: 21 sec out of approx 15.1 min Dilation: 100% done. (509/509) Time: 10.3 min out of approx 10.3 min Dilation: DONE in 10.7 min Dilation: 3% done. (15/509) Time: 20.2 sec out of approx 11.7 min Dilation: 100% done. (509/509) Time: 8.7 min out of approx 8.7 min Dilation: DONE in 9 min Dilation: 4% done. (19/509) Time: 20.1 sec out of approx 9 min Dilation: 100% done. (509/509) Time: 7.2 min out of approx 7.2 min Dilation: DONE in 7.4 min Outputting Volume: ............................................................ -+-------------------------[0]------------------------- | steps: 8 8 8 | blur_fwhm: 16 | simplex: 32 | source: /tmp/bestlinreg_maget-AXrlKyCT/s_brain3_t1_16_blur.mnc | target: /tmp/bestlinreg_maget-AXrlKyCT/t_sub-testanat_T1w_template_16_blur.mnc | xfm: /tmp/bestlinreg_maget-AXrlKyCT/s_brain3_t1_0.xfm -+-----------------------------------------------------

Reading Volume: ............................................................... Blurring volume: .............................................................. Making byte volume... Outputting Volume: ............................................................ Blurring volume: .............................................................. Making byte volume... Reading Volume: ............................................................... COG of v1: -2.193283 27.297710 23.153629 COG of v2: -0.982889 5.888766 -10.524686 [trans] = 1.210394 -21.408943 -33.678314 Only center & trans required, now returning from init_transformation WARNING: source volume not UNSIGNED BYTE, will do conversion now. Converting: ................................................................... WARNING: target volume not UNSIGNED BYTE, will do conversion now. Initial objective function val = -0.04291377 Final objective function value = -0.09415729 -+-------------------------[1]------------------------- | steps: 4 4 4 | blur_fwhm: 8 | simplex: 16 | source: /tmp/bestlinreg_maget-AXrlKyCT/s_brain3_t1_8_blur.mnc | target: /tmp/bestlinreg_maget-AXrlKyCT/t_sub-testanat_T1w_template_8_blur.mnc | xfm: /tmp/bestlinreg_maget-AXrlKyCT/s_brain3_t1_1.xfm -+-----------------------------------------------------

Reading Volume: ............................................................... Blurring volume: .............................................................. Making byte volume... Outputting Volume: ............................................................ Blurring volume: .............................................................. Making byte volume... Reading Volume: ............................................................... WARNING: source volume not UNSIGNED BYTE, will do conversion now. Converting: ................................................................... WARNING: target volume not UNSIGNED BYTE, will do conversion now. Initial objective function val = -0.07157136 Final objective function value = -0.07515074 -+-------------------------[2]------------------------- | steps: 4 4 4 | blur_fwhm: 8 | simplex: 16 | source: /tmp/bestlinreg_maget-AXrlKyCT/s_brain3_t1_8_blur.mnc | target: /tmp/bestlinreg_maget-AXrlKyCT/t_sub-testanat_T1w_template_8_blur.mnc | xfm: /tmp/bestlinreg_maget-AXrlKyCT/s_brain3_t1_2.xfm -+-----------------------------------------------------

Reading Volume: ............................................................... WARNING: source volume not UNSIGNED BYTE, will do conversion now. Converting: ................................................................... WARNING: target volume not UNSIGNED BYTE, will do conversion now. Initial objective function val = -0.07515074 Final objective function value = -0.07805241 -+-------------------------[3]------------------------- | steps: 4 4 4 | blur_fwhm: 8 | simplex: 16 | source: /tmp/bestlinreg_maget-AXrlKyCT/s_brain3_t1_8_blur.mnc | target: /tmp/bestlinreg_maget-AXrlKyCT/t_sub-testanat_T1w_template_8_blur.mnc | xfm: /tmp/bestlinreg_maget-AXrlKyCT/s_brain3_t1_3.xfm -+-----------------------------------------------------

Reading Volume: ............................................................... WARNING: source volume not UNSIGNED BYTE, will do conversion now. Converting: ................................................................... WARNING: target volume not UNSIGNED BYTE, will do conversion now. Initial objective function val = -0.07805241 Final objective function value = -0.07807405 -+-------------------------[4]------------------------- | steps: 4 4 4 | blur_fwhm: 4 | simplex: 8 | source: /tmp/bestlinreg_maget-AXrlKyCT/s_brain3_t1_4_blur.mnc | target: /tmp/bestlinreg_maget-AXrlKyCT/t_sub-testanat_T1w_template_4_blur.mnc | xfm: /tmp/bestlinreg_maget-AXrlKyCT/s_brain3_t1_4.xfm -+-----------------------------------------------------

Reading Volume: ............................................................... Blurring volume: .............................................................. Making byte volume... Outputting Volume: ............................................................ Making byte volume... Reading Volume: ............................................................... WARNING: source volume not UNSIGNED BYTE, will do conversion now. Converting: ................................................................... WARNING: target volume not UNSIGNED BYTE, will do conversion now. Initial objective function val = -0.07131222 Final objective function value = -0.07217486 -+-------------------------[5]------------------------- | steps: 2 2 2 | blur_fwhm: 2 | simplex: 4 | source: /tmp/bestlinreg_maget-AXrlKyCT/s_brain3_t1_2_blur.mnc | target: /tmp/bestlinreg_maget-AXrlKyCT/t_sub-testanat_T1w_template_2_blur.mnc | xfm: /tmp/bestlinreg_maget-AXrlKyCT/s_brain3_t1_5.xfm -+-----------------------------------------------------

Reading Volume: ............................................................... Blurring volume: .............................................................. Making byte volume... Outputting Volume: ............................................................ Making byte volume... Reading Volume: ............................................................... WARNING: source volume not UNSIGNED BYTE, will do conversion now. Converting: ................................................................... WARNING: target volume not UNSIGNED BYTE, will do conversion now. Initial objective function val = -0.06622965 Final objective function value = -0.06717904 -+-------------------------[6]------------------------- | steps: 1 1 1 | blur_fwhm: 1 | simplex: 2 | source: /tmp/bestlinreg_maget-AXrlKyCT/s_brain3_t1_1_blur.mnc | target: /tmp/bestlinreg_maget-AXrlKyCT/t_sub-testanat_T1w_template_1_blur.mnc | xfm: /tmp/bestlinreg_maget-AXrlKyCT/s_brain3_t1_6.xfm -+-----------------------------------------------------

Reading Volume: ............................................................... Blurring volume: .............................................................. Making byte volume... Outputting Volume: ............................................................ Blurring volume: .............................................................. Making byte volume... Reading Volume: ............................................................... WARNING: source volume not UNSIGNED BYTE, will do conversion now. Converting: ................................................................... WARNING: target volume not UNSIGNED BYTE, will do conversion now. Initial objective function val = -0.06463195 Final objective function value = -0.06473929 -+-------------------------[1]------------------------- | steps: 2 2 2 | blur_fwhm: 8 | simplex: 1 | source: /tmp/bestlinreg_maget-gbigNVBa/s_brain3_t1_8_blur.mnc | target: /tmp/bestlinreg_maget-gbigNVBa/t_sub-testanat_T1w_template_8_blur.mnc | xfm: /tmp/bestlinreg_maget-gbigNVBa/s_brain3_t1_1.xfm -+-----------------------------------------------------

++ mktemp -d

Error in mincblur in file /home/nistmni/src/mni_autoreg/mincblur/mincblur.c, line 301 problems reading `/tmp/bestlinreg_maget-ynSG4ZBf/s_brain1_t1_resampled.mnc'. Died at /home/maxime/hpc/src/MAGeTbrain/bin/bestlinreg_maget line 373. Reading Volume: 15% done. (153/1000) Time: 20.1 sec out of approx 2.2 min Reading Volume: DONE in 107 sec Dilation: 2% done. (8/489) Time: 20 sec out of approx 21 min Dilation: DONE in 11.2 min Dilation: 3% done. (16/489) Time: 20.1 sec out of approx 10.3 min Dilation: DONE in 9.1 min Dilation: 3% done. (15/489) Time: 20.1 sec out of approx 11 min Dilation: DONE in 8.9 min Dilation: 4% done. (19/489) Time: 20.7 sec out of approx 8.9 min Dilation: DONE in 10.2 min Dilation: 2% done. (10/489) Time: 21.2 sec out of approx 17.8 min Dilation: 100% done. (489/489) Time: 9.7 min out of approx 9.7 min Dilation: DONE in 9.8 min Dilation: 3% done. (13/489) Time: 20 sec out of approx 12.7 min Dilation: 100% done. (489/489) Time: 10.8 min out of approx 10.8 min Dilation: DONE in 11 min Dilation: 0% done. (2/489) Time: 21.3 sec out of approx 86.7 min Dilation: DONE in 12.3 min Dilation: 0% done. (2/489) Time: 33 sec out of approx 2.2 hrs Dilation: DONE in 10.8 min Dilation: DONE in 9.7 min Dilation: 3% done. (13/489) Time: 20.6 sec out of approx 13.1 min Dilation: DONE in 7.9 min Outputting Volume: ............................................................ -+-------------------------[0]------------------------- | steps: 8 8 8 | blur_fwhm: 16 | simplex: 32 | source: /tmp/bestlinreg_maget-X6DwT9LO/s_brain1_t1_16_blur.mnc | target: /tmp/bestlinreg_maget-X6DwT9LO/t_sub-testanat_T1w_template_16_blur.mnc | xfm: /tmp/bestlinreg_maget-X6DwT9LO/s_brain1_t1_0.xfm -+-----------------------------------------------------

Reading Volume: ............................................................... Blurring volume: .............................................................. Making byte volume... Outputting Volume: ............................................................ Blurring volume: .............................................................. Making byte volume... Reading Volume: 28% done. (275/1000) Time: 20.2 sec out of approx 73.1 sec Reading Volume: 64% done. (636/1000) Time: 40.2 sec out of approx 63.2 sec Reading Volume: 99% done. (985/1000) Time: 60.2 sec out of approx 61.1 sec Reading Volume: DONE in 61.1 sec COG of v1: -1.727876 21.059248 48.846256 COG of v2: -0.982889 5.888766 -10.524686 [trans] = 0.744988 -15.170482 -59.370941 Only center & trans required, now returning from init_transformation WARNING: source volume not UNSIGNED BYTE, will do conversion now. Converting: ................................................................... WARNING: target volume not UNSIGNED BYTE, will do conversion now. Initial objective function val = -0.05883462 Final objective function value = -0.12040356 -+-------------------------[1]------------------------- | steps: 4 4 4 | blur_fwhm: 8 | simplex: 16 | source: /tmp/bestlinreg_maget-X6DwT9LO/s_brain1_t1_8_blur.mnc | target: /tmp/bestlinreg_maget-X6DwT9LO/t_sub-testanat_T1w_template_8_blur.mnc | xfm: /tmp/bestlinreg_maget-X6DwT9LO/s_brain1_t1_1.xfm -+-----------------------------------------------------

Reading Volume: ............................................................... Blurring volume: .............................................................. Making byte volume... Outputting Volume: ............................................................ Making byte volume... Reading Volume: ............................................................... WARNING: source volume not UNSIGNED BYTE, will do conversion now. Converting: ................................................................... WARNING: target volume not UNSIGNED BYTE, will do conversion now. Initial objective function val = -0.07693087 Final objective function value = -0.08857998 -+-------------------------[2]------------------------- | steps: 4 4 4 | blur_fwhm: 8 | simplex: 16 | source: /tmp/bestlinreg_maget-X6DwT9LO/s_brain1_t1_8_blur.mnc | target: /tmp/bestlinreg_maget-X6DwT9LO/t_sub-testanat_T1w_template_8_blur.mnc | xfm: /tmp/bestlinreg_maget-X6DwT9LO/s_brain1_t1_2.xfm -+-----------------------------------------------------

Reading Volume: ............................................................... WARNING: source volume not UNSIGNED BYTE, will do conversion now. Converting: ................................................................... WARNING: target volume not UNSIGNED BYTE, will do conversion now. Initial objective function val = -0.08857998 Final objective function value = -0.09135900 -+-------------------------[3]------------------------- | steps: 4 4 4 | blur_fwhm: 8 | simplex: 16 | source: /tmp/bestlinreg_maget-X6DwT9LO/s_brain1_t1_8_blur.mnc | target: /tmp/bestlinreg_maget-X6DwT9LO/t_sub-testanat_T1w_template_8_blur.mnc | xfm: /tmp/bestlinreg_maget-X6DwT9LO/s_brain1_t1_3.xfm -+-----------------------------------------------------

Reading Volume: ............................................................... WARNING: source volume not UNSIGNED BYTE, will do conversion now. Converting: ................................................................... WARNING: target volume not UNSIGNED BYTE, will do conversion now. Initial objective function val = -0.09135900 Final objective function value = -0.09142546 -+-------------------------[4]------------------------- | steps: 4 4 4 | blur_fwhm: 4 | simplex: 8 | source: /tmp/bestlinreg_maget-X6DwT9LO/s_brain1_t1_4_blur.mnc | target: /tmp/bestlinreg_maget-X6DwT9LO/t_sub-testanat_T1w_template_4_blur.mnc | xfm: /tmp/bestlinreg_maget-X6DwT9LO/s_brain1_t1_4.xfm -+-----------------------------------------------------

Reading Volume: ............................................................... Blurring volume: .............................................................. Making byte volume... Outputting Volume: ............................................................ Blurring volume: .............................................................. Making byte volume... Reading Volume: ............................................................... WARNING: source volume not UNSIGNED BYTE, will do conversion now. Converting: ................................................................... WARNING: target volume not UNSIGNED BYTE, will do conversion now. Initial objective function val = -0.07915619 Final objective function value = -0.08026140 -+-------------------------[5]------------------------- | steps: 2 2 2 | blur_fwhm: 2 | simplex: 4 | source: /tmp/bestlinreg_maget-X6DwT9LO/s_brain1_t1_2_blur.mnc | target: /tmp/bestlinreg_maget-X6DwT9LO/t_sub-testanat_T1w_template_2_blur.mnc | xfm: /tmp/bestlinreg_maget-X6DwT9LO/s_brain1_t1_5.xfm -+-----------------------------------------------------

Reading Volume: ............................................................... Blurring volume: .............................................................. Making byte volume... Outputting Volume: ............................................................ Blurring volume: .............................................................. Making byte volume... Reading Volume: ............................................................... WARNING: source volume not UNSIGNED BYTE, will do conversion now. Converting: ................................................................... WARNING: target volume not UNSIGNED BYTE, will do conversion now. Initial objective function val = -0.07476038 Final objective function value = -0.07527920 -+-------------------------[6]------------------------- | steps: 1 1 1 | blur_fwhm: 1 | simplex: 2 | source: /tmp/bestlinreg_maget-X6DwT9LO/s_brain1_t1_1_blur.mnc | target: /tmp/bestlinreg_maget-X6DwT9LO/t_sub-testanat_T1w_template_1_blur.mnc | xfm: /tmp/bestlinreg_maget-X6DwT9LO/s_brain1_t1_6.xfm -+-----------------------------------------------------

Reading Volume: ............................................................... Blurring volume: .............................................................. Making byte volume... Outputting Volume: ............................................................ Blurring volume: .............................................................. Making byte volume... Reading Volume: ............................................................... WARNING: source volume not UNSIGNED BYTE, will do conversion now. Converting: ................................................................... WARNING: target volume not UNSIGNED BYTE, will do conversion now. Initial objective function val = -0.07284146 Final objective function value = -0.07311182 -+-------------------------[1]------------------------- | steps: 2 2 2 | blur_fwhm: 8 | simplex: 1 | source: /tmp/bestlinreg_maget-ynSG4ZBf/s_brain1_t1_8_blur.mnc | target: /tmp/bestlinreg_maget-ynSG4ZBf/t_sub-testanat_T1w_template_8_blur.mnc | xfm: /tmp/bestlinreg_maget-ynSG4ZBf/s_brain1_t1_1.xfm -+-----------------------------------------------------

Traceback (most recent call last): File "/home/maxime/hpc/src/MAGeTbrain/bin/mb", line 792, in main() File "/home/maxime/hpc/src/MAGeTbrain/bin/mb", line 220, in main options.func(options) File "/home/maxime/hpc/src/MAGeTbrain/bin/mb", line 329, in command_run p.run() File "/home/maxime/hpc/src/MAGeTbrain/bin/mb", line 608, in run self.queue.run(stages=self._get_stage_order()) File "/home/maxime/hpc/src/MAGeTbrain/bin/mb", line 508, in run self.parallel(self.commands[stage]) File "/home/maxime/hpc/src/MAGeTbrain/bin/mb", line 514, in parallel self.execute(command, input='\n'.join(commands)) File "/home/maxime/hpc/src/MAGeTbrain/bin/mb", line 492, in execute raise Exception("Returns %i :: %s" % (proc.returncode, command)) Exception: Returns 5 :: parallel -j8

gdevenyi commented 4 years ago

Hi,

mincblur is a core component of the minc-toolkit-v2 package, as is bestlinreg. Based on my reading of the logs, it looks like perhaps you've run out of disk space or memory. MAGeT is very computationally expensive and intended for use on clusters. You may be able to get things to not overload by adjusting the parallelism settings found under mb run --help however, you can expect this to take weeks to run for a real dataset.

maximedieudonne commented 4 years ago

thank you a lot for you answer, I work on a ubuntu virtual machine, I have check that i have 14 Go available before running Maget but still the same problem, what are the clue in the log correlated to a space memory issue ? ok, I check the parallelism setting and i try an other run. yes, thank you for the advice, it's a home test before be able to migrate to a cluster.

gdevenyi commented 4 years ago

what are the clue in the log correlated to a space memory issue

This should never happen, mincblur is a very simple program with no known failure modes expect "can't write to the file"

/home/nistmni/src/libminc/libsrc/netcdf_convenience.c:573 (from miopen): Unable to open file '/tmp/bestlinreg_maget-ynSG4ZBf/s_brain1_t1_resampled.mnc'
/home/nistmni/src/libminc/volume_io/Prog_utils/print.c:226 (from miopen): volume_io error: Error: opening MINC file "/tmp/bestlinreg_maget-ynSG4ZBf/s_brain1_t1_resampled.mnc".

That, along with the fact that I see this:

Reading Volume: 15% done. (153/1000) Time: 20.1 sec out of approx 2.2 min
Reading Volume: DONE in 107 sec
Dilation: 2% done. (8/489) Time: 20 sec out of approx 21 min
Dilation: DONE in 11.2 min
Dilation: 3% done. (16/489) Time: 20.1 sec out of approx 10.3 min
Dilation: DONE in 9.1 min
Dilation: 3% done. (15/489) Time: 20.1 sec out of approx 11 min
Dilation: DONE in 8.9 min
Dilation: 4% done. (19/489) Time: 20.7 sec out of approx 8.9 min
Dilation: DONE in 10.2 min
Dilation: 2% done. (10/489) Time: 21.2 sec out of approx 17.8 min
Dilation: 100% done. (489/489) Time: 9.7 min out of approx 9.7 min
Dilation: DONE in 9.8 min
Dilation: 3% done. (13/489) Time: 20 sec out of approx 12.7 min
Dilation: 100% done. (489/489) Time: 10.8 min out of approx 10.8 min
Dilation: DONE in 11 min
Dilation: 0% done. (2/489) Time: 21.3 sec out of approx 86.7 min
Dilation: DONE in 12.3 min
Dilation: 0% done. (2/489) Time: 33 sec out of approx 2.2 hrs
Dilation: DONE in 10.8 min
Dilation: DONE in 9.7 min
Dilation: 3% done. (13/489) Time: 20.6 sec out of approx 13.1 min
Dilation: DONE in 7.9 min

That mincmorph command is quite fast, I see extreme time estimates, indicating the machine is either very underpowered, or very overloaded.

maximedieudonne commented 4 years ago

thank you for your reply, i solved my problem by changing laptop with more capabilities