search

aces / Loris-MRI

The set of scripts that preprocess and insert MRI data into the database.
10 stars 50 forks source link

[run_defacing_script.pl] Images not defacing #971

Closed zaliqarosli closed 1 year ago

zaliqarosli commented 1 year ago 
run_defacing_script.pl -profile prod -sessionIDs 144

Constant subroutine NeuroDB::objectBroker::ConfigOB::CREATE_CANDIDATES redefined at /usr/lib/x86_64-linux-gnu/perl-base/constant.pm line 171.
PSCID is: MTL144
 CandID id: 300144
 visit_label is: V1
Argument "8Exiting now\n\n" isn't numeric in right bitshift (>>) at /opt/Loris-MRI/bin/mri/tools/run_defacing_script.pl line 682.
Argument "\nAn error occurred when running register_processed_data..." isn't numeric in right bitshift (>>) at /opt/Loris-MRI/bin/mri/tools/run_defacing_script.pl line 682.

Images not appearing in assembly/$candID/$timepoint/$processed and not in DB

zaliqarosli commented 1 year ago

Full log:

....
Processing:.........................................Done
Processing:.........................................Done
Processing:.........................................Done
Processing:.........................................Done
Processing:.........................................Done
Processing:.........................................Done
Processing:.........................................Done
Processing:.........................................Done
Number of iterations: 100
CV of field change: 0.000502756
Processing:................................................................................................................................................................Done
Processing:.........................................Done
Processing:.........................................Done

....

Processing:.........................................Done
Number of iterations: 100
CV of field change: 0.000415
Processing:................................................................................................................................................................Done
Warning: For optimal results, you should use masking.
Usage: bestlinreg_claude.pl [options] source.mnc target.mnc output.xfm [output.mnc]
       bestlinreg_claude.pl -help to list options

-+-------------------------[0]-------------------------
 | steps:          4 4 4
 | blur_fwhm:      8
 | simplex:        16
 | source:         /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_8_blur.mnc
 | target:         /data/tmp/bestlinreg_claude.pl-07WqEJid/Tmni_icbm152_t1_tal_nlin_sym_09c_8_blur.mnc
 | xfm:            /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_0.xfm
-+-----------------------------------------------------

mincblur -clobber -no_apodize -fwhm 8 /data/tmp/deface_minipipe.pl-D82g_82Y/clp_t1w.mnc /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_8
Making byte volume...
mincblur -clobber -no_apodize -fwhm 8 /opt/minc/1.9.18/../share/icbm152_model_09c/mni_icbm152_t1_tal_nlin_sym_09c.mnc /data/tmp/bestlinreg_claude.pl-07WqEJid/Tmni_icbm152_t1_tal_nlin_sym_09c_8
Making byte volume...
minctracc -clobber -xcorr -lsq6 -step 4 4 4 -simplex 16 -tol 0.0001 -transformation /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_init.xfm /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_8_blur.mnc /data/tmp/bestlinreg_claude.pl-07WqEJid/Tmni_icbm152_t1_tal_nlin_sym_09c_8_blur.mnc /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_0.xfm
Initial objective function val = 0.11361250
Final objective function value = 0.06532330
-+-------------------------[1]-------------------------
 | steps:          4 4 4
 | blur_fwhm:      8
 | simplex:        16
 | source:         /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_8_blur.mnc
 | target:         /data/tmp/bestlinreg_claude.pl-07WqEJid/Tmni_icbm152_t1_tal_nlin_sym_09c_8_blur.mnc
 | xfm:            /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_1.xfm
-+-----------------------------------------------------
minctracc -clobber -xcorr -lsq7 -step 4 4 4 -simplex 16 -tol 0.0001 -transformation /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_0.xfm /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_8_blur.mnc /data/tmp/bestlinreg_claude.pl-07WqEJid/Tmni_icbm152_t1_tal_nlin_sym_09c_8_blur.mnc /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_1.xfm
Initial objective function val = 0.06532320
Final objective function value = 0.02490571
-+-------------------------[2]-------------------------
 | steps:          4 4 4
 | blur_fwhm:      4
 | simplex:        8
 | source:         /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_4_blur.mnc
 | target:         /data/tmp/bestlinreg_claude.pl-07WqEJid/Tmni_icbm152_t1_tal_nlin_sym_09c_4_blur.mnc
 | xfm:            /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_2.xfm
-+-----------------------------------------------------

mincblur -clobber -no_apodize -fwhm 4 /data/tmp/deface_minipipe.pl-D82g_82Y/clp_t1w.mnc /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_4
Making byte volume...
mincblur -clobber -no_apodize -fwhm 4 /opt/minc/1.9.18/../share/icbm152_model_09c/mni_icbm152_t1_tal_nlin_sym_09c.mnc /data/tmp/bestlinreg_claude.pl-07WqEJid/Tmni_icbm152_t1_tal_nlin_sym_09c_4
Making byte volume...
minctracc -clobber -xcorr -lsq9 -step 4 4 4 -simplex 8 -tol 0.0001 -transformation /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_1.xfm /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_4_blur.mnc /data/tmp/bestlinreg_claude.pl-07WqEJid/Tmni_icbm152_t1_tal_nlin_sym_09c_4_blur.mnc /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_2.xfm
Initial objective function val = 0.04231220
Final objective function value = 0.03013920
-+-------------------------[3]-------------------------
 | steps:          2 2 2
 | blur_fwhm:      2
 | simplex:        4
 | source:         /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_2_blur.mnc
 | target:         /data/tmp/bestlinreg_claude.pl-07WqEJid/Tmni_icbm152_t1_tal_nlin_sym_09c_2_blur.mnc
 | xfm:            /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_3.xfm
-+-----------------------------------------------------
mincblur -clobber -no_apodize -fwhm 2 /data/tmp/deface_minipipe.pl-D82g_82Y/clp_t1w.mnc /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_2
Making byte volume...
mincblur -clobber -no_apodize -fwhm 2 /opt/minc/1.9.18/../share/icbm152_model_09c/mni_icbm152_t1_tal_nlin_sym_09c.mnc /data/tmp/bestlinreg_claude.pl-07WqEJid/Tmni_icbm152_t1_tal_nlin_sym_09c_2
Making byte volume...
minctracc -clobber -xcorr -lsq9 -step 2 2 2 -simplex 4 -tol 0.0005 -transformation /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_2.xfm /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_2_blur.mnc /data/tmp/bestlinreg_claude.pl-07WqEJid/Tmni_icbm152_t1_tal_nlin_sym_09c_2_blur.mnc /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_3.xfm
Initial objective function val = 0.04554820
Final objective function value = 0.04542599
mv -f /data/tmp/bestlinreg_claude.pl-07WqEJid/Sclp_t1w_3.xfm /data/tmp/deface_minipipe.pl-D82g_82Y/t1w_stx.xfm
[mritoself] [lorisadmin@zrosli-test-25:/home/lorisadmin] [2023-05-04 11:21:48] /opt/minc/1.9.18/bin/autocrop /data/tmp/deface_minipipe.pl-D82g_82Y/clp_demo_300144_V1_t2_001.mnc /data/tmp/mritoself_9777//clp_demo_300144_V1_t2_001_cropS.mnc -step -0.99999999999999944489 -1 -1 -extend 0,0 0,0 -66,0 -byte
[autocrop] [lorisadmin@zrosli-test-25:/home/lorisadmin] [2023-05-04 11:21:48] /opt/minc/1.9.18/bin/mincreshape -byte /data/tmp/deface_minipipe.pl-D82g_82Y/clp_demo_300144_V1_t2_001.mnc /data/tmp/mritoself_9777//clp_demo_300144_V1_t2_001_cropS.mnc  -dimrange xspace=-1,161 -dimrange zspace=0,190 -dimrange yspace=0,256
Copying chunks:.................................................................................................................................................................Done.
[mritoself] [lorisadmin@zrosli-test-25:/home/lorisadmin] [2023-05-04 11:21:48] /opt/minc/1.9.18/bin/autocrop /data/tmp/deface_minipipe.pl-D82g_82Y/clp_t1w.mnc /data/tmp/mritoself_9777//clp_t1w_cropT.mnc -step -0.99999999999999622524 -1 -1 -extend 0,0 0,0 -66,0 -byte
[autocrop] [lorisadmin@zrosli-test-25:/home/lorisadmin] [2023-05-04 11:21:48] /opt/minc/1.9.18/bin/mincreshape -byte /data/tmp/deface_minipipe.pl-D82g_82Y/clp_t1w.mnc /data/tmp/mritoself_9777//clp_t1w_cropT.mnc  -dimrange xspace=-1,161 -dimrange zspace=0,190 -dimrange yspace=-1,225
Copying chunks:.................................................................................................................................................................Done.
[mritoself] [lorisadmin@zrosli-test-25:/home/lorisadmin] [2023-05-04 11:21:49] /opt/minc/1.9.18/bin/minctracc /data/tmp/mritoself_9777//clp_demo_300144_V1_t2_001_cropS.mnc /data/tmp/mritoself_9777//clp_t1w_cropT.mnc /data/tmp/mritoself_9777//demo_300144_V1_t2_001_t1_tmp1.xfm -identity -est_center -lsq6 -mi -groups 256 -step 7.3 7.3 7.3 -simplex 3
WARNING: source volume not UNSIGNED BYTE, will do conversion now.
WARNING: target volume not UNSIGNED BYTE, will do conversion now.
Initial objective function val = -0.65683931
Final objective function value = -0.74529642
[mritoself] [lorisadmin@zrosli-test-25:/home/lorisadmin] [2023-05-04 11:21:51] /opt/minc/1.9.18/bin/minctracc /data/tmp/mritoself_9777//clp_demo_300144_V1_t2_001_cropS.mnc /data/tmp/mritoself_9777//clp_t1w_cropT.mnc /data/tmp/deface_minipipe.pl-D82g_82Y/demo_300144_V1_t2_001_t1.xfm -est_center -transformation /data/tmp/mritoself_9777//demo_300144_V1_t2_001_t1_tmp1.xfm -lsq6 -mi -groups 256 -step 4.3 4.3 4.3 -simplex 1.5
WARNING: source volume not UNSIGNED BYTE, will do conversion now.
WARNING: target volume not UNSIGNED BYTE, will do conversion now.
Initial objective function val = -0.72715789
Final objective function value = -0.72715789

mincbeast --            an implementation of BEaST (Brain Extraction
                        using non-local Segmentation Technique) version 1.90.00

Using OpenMP, max number of threads=1
initialscale:2 targetscale:1
4 1 2 0.20 0.25 0.95 20
2 1 4 0.50 0.25 0.95 20
2: 4 1 2 0.20 0.25 0.95 20
1: 2 1 4 0.50 0.25 0.95 20
0: 4 1 2 0.50 0.25 0.95 20
Performing pre-selection (0 MB/subject)....................done
Patch size: 1
Search area: 2
Beta: 0.250000
Threshold: 0.950000
Selection: 20
Dimensions: 48 57 48
Computing first moment image...done (0 sec)
Computing second moment image...done (0 sec)
Segment 48 /  48 (redoing 1848 voxels) t=0.902500, min=95  (4 sec)
Segment 48 /  48 (redoing 121 voxels) t=0.857375, min=90  (1 sec)
Segment 48 /  48 (redoing 22 voxels) t=0.814506, min=85  (0 sec)
Segment 48 /  48 (redoing 2 voxels) t=0.773781, min=80  (0 sec)
Segment 48 /  48 done (0 sec, t=0.773781, min=80)
Performing pre-selection (4 MB/subject)....................done
Patch size: 1
Search area: 4
Beta: 0.250000
Threshold: 0.950000
Selection: 20
Dimensions: 96 114 96
Computing first moment image...done (0 sec)
Computing second moment image...done (0 sec)
Segment 96 /  96 (redoing 276 voxels) t=0.902500, min=95  (33 sec)
Segment 96 /  96 (redoing 47 voxels) t=0.857375, min=90  (0 sec)
Segment 96 /  96 (redoing 16 voxels) t=0.814506, min=85  (0 sec)
Segment 96 /  96 (redoing 9 voxels) t=0.773781, min=80  (0 sec)
Segment 96 /  96 (redoing 2 voxels) t=0.735092, min=76  (0 sec)
Segment 96 /  96 (redoing 1 voxels) t=0.698337, min=72  (0 sec)
Segment 96 /  96 done (0 sec, t=0.698337, min=72)
Writing a binary volume...
Transforming slices:..............................................Done
Transforming slices:..............................................Done
Processing:..............................................Done
Transforming slices:..............................................Done
Processing:..............................................Done
Transforming slices:..............................................Done
Processing:..............................................Done
Processing:..........................................................................................................................................Done
Dilate 2
Dilate 1
Transforming slices:..............................................Done
Processing:..............................................Done
Transforming slices:..............................................Done
Processing:..........................................................................................................................................Done
Processing:..........................................................................................................................................Done
Transforming slices:......................................................................................Done
Copying chunks:.Done.
Transforming slices:................................................................................................................................................................Done
Transforming slices:..................................................................................Done
Copying chunks:.Done.
Transforming slices:................................................................................................................................................................Done
Copying chunks:................................................................................................................................................................Done.
Copying chunks:................................................................................................................................................................Done.
Constant subroutine NeuroDB::objectBroker::ConfigOB::CREATE_CANDIDATES redefined at /usr/lib/x86_64-linux-gnu/perl-base/constant.pm line 171.
PSCID is: MTL144
 CandID id: 300144
 visit_label is: V1
Argument "8Exiting now\n\n" isn't numeric in right bitshift (>>) at /opt/Loris-MRI/bin/mri/tools/run_defacing_script.pl line 682.
Argument "\nAn error occurred when running register_processed_data..." isn't numeric in right bitshift (>>) at /opt/Loris-MRI/bin/mri/tools/run_defacing_script.pl line 682.
zaliqarosli commented 1 year ago

Tried running deface_minipipe.pl directly and it worked well:

deface_minipipe.pl /data/Loris-MRI/data/assembly/676061/V1/mri/native/demo_676
061_V1_t1_001.mnc /data/Loris-MRI/data/assembly/676061/V1/mri/native/demo_676061_V1_t2_001.mnc /data/tmp/defaced_676061 --kee
p-real-range --beastlib /opt/minc/1.9.18/../share/beast-library-1.1 --model mni_icbm152_t1_tal_nlin_sym_09c --model-dir /opt/m
inc/1.9.18/../share/icbm152_model_09c

I then printed out the deface_minipipe.pl command running in run_defacing_script.pl and ran that directly, and found errors. I think there is an error with the $output_basename provided to deface_minipipe

(loris-mri-python) lorisadmin@zrosli-test-25:/opt/Loris-MRI/bin/mri$ run_defacing_script.pl -profile prod -sessionIDs 2145

DEFACE MINIPIPE COMMAND: deface_minipipe.pl /data/Loris-MRI/data/assembly/676061/V1/mri/native/demo_676061_V1_t1_001.mnc  /data/Loris-MRI/data/assembly/676061/V1/mri/native/demo_676061_V1_t2_001.mnc /data/Loris-MRI/data/assembly/676061/V1/mri/native/demo_676061_V1_t2_001.mnc /data/Loris-MRI/data/assembly/676061/V1/mri/native/demo_676061_V1_t2_001.mnc /data/tmp/deface-tmVg7Wve/676061/V1/demo_676061_V1 --keep-real-range --beastlib /opt/minc/1.9.18/../share/beast-library-1.1  --model mni_icbm152_t1_tal_nlin_sym_09c  --model-dir /opt/minc/1.9.18/../share/icbm152_model_09c

(loris-mri-python) lorisadmin@zrosli-test-25:/opt/Loris-MRI/bin/mri$ deface_minipipe.pl /data/Loris-MRI/data/assembly/676061/V1/mri/native/demo_676061_V1_t1_001.mnc  /data/Loris-MRI/data/assembly/676061/V1/mri/native/demo_676061_V1_t2_001.mnc /data/Loris-MRI/data/assembly/676061/V1/mri/native/demo_676061_V1_t2_001.mnc /data/Loris-MRI/data/assembly/676061/V1/mri/native/demo_676061_V1_t2_001.mnc /data/tmp/deface-tmVg7Wve/676061/V1/demo_676061_V1 --keep-real-range --beastlib /opt/minc/1.9.18/../share/beast-library-1.1  --model mni_icbm152_t1_tal_nlin_sym_09c  --model-dir /opt/minc/1.9.18/../share/icbm152_model_09c

Could not create directory '/data/tmp/deface-tmVg7Wve/676061/V1'. Error is: 'No such file or directory'
Processing:.........................................Done
Processing:.........................................Done
Processing:.........................................Done
Processing:.........................................Done
Processing:.........................................Done
Processing:.........................................Done
Processing:.........................................Done
Processing:.........................................Done

but my /data/tmp permissions are the following:

(loris-mri-python) lorisadmin@zrosli-test-25:/data$ ls -la
total 16
drwxr-xr-x  4 root       root     4096 May  4 10:51 .
drwxr-xr-x 21 root       root     4096 Apr  5 08:45 ..
drwxrws---  3 lorisadmin www-data 4096 Apr 28 14:28 Loris-MRI
drwxrwxrwx  2 lorisadmin www-data 4096 May  4 12:48 tmp

The fix is probably to do mkdir -p at https://github.com/aces/Loris-MRI/blob/7022972f1b228c338f82350929507e71fa34e8ef/uploadNeuroDB/bin/deface_minipipe.pl#L91

cmadjar commented 1 year ago

@zaliqarosli could you test if https://github.com/aces/Loris-MRI/pull/978 resolves the issue?