dorkylever commented 5 years ago

Hi everyone,

I am a PhD candidate within the Arkell laboratory at the Australian National University. I am trying to build a 3D average model for ethanol-exposed embryos at 12.5 dpc (n=29) for future MBM analysis.

I'm currently submitting the pipeline on qsub as the super-computer I'm using lacks PBS. I am using the following python script, bash script and config file. run_MBM_py.txt

launch_MBM_sh.txt

wt_eth_cfg.txt

The pipeline fails at the nu-estimate stage as it can't find MNI-startup.pm, but only for some of the .mnc files: eth_12_5_fail

Should I be specifying a different protocol? I've checked my images and they appear relatively similar. I'm happy to provide more details if asked.

Kind Regards, Kyle Drover

gdevenyi commented 5 years ago

You do not appear to have a proper functioning install of minc-toolkit-v2

dorkylever commented 5 years ago

Hi Gabriel, Is it necessary to specify minc-toolkit-v2 1.9.17 as described in https://github.com/BIC-MNI/minc-toolkit-v2/issues/85 and direct the installation of pyezminc, minc-stuffs and generate_deformation_fields to /opt/minc/1.9.17 as you did in https://github.com/CobraLab/MINC-VM/blob/master/provision.sh ?

bcdarwin commented 5 years ago

I would probably install 1.9.17. Ultimately the installation locations don't really matter as long as you set the appropriate {PATH,LD_LIBRARY_PATH,CPATH,ANTSPATH,PYTHONPATH,PERL5LIB,MANPATH,...} most of which you can do most easily using the included minc-toolkit-config.sh file. As per the Pydpiper INSTALL file, the only dependencies you need to install yourself are minc-toolkit-v2, minc-stuffs, and minc2-simple. You could certainly install these programs in that same location if you like.

dorkylever commented 5 years ago

Hi Ben,

I'm still having issues installing Pydpiper:

We have tried specifying the install location in the minc-toolkit-config.sh and minc-toolkit-config.csh files and when running ccmake (i.e. ../ -DCMAKE_INSTALL_PREFIX:PATH=/usr/bin). The build, make and install will run without errors but the test_MBM_and_MAGeT.py shuts down all but two executors at stage 20 of 1858. Additionally, using which in bash to find mni-perllib, mni_autoreg and N3 will not find any such libraries. We assumed it was something to do with MBM.py trying to find minc-toolkit in /opt/.

As such, we have been trying to run Pydpiper through the MINC-VM (https://github.com/CobraLab/MINC-VM). Briefly, this has involved installing VirtualBox on the head node of our cluster and using the VirtualBox GUI in a xterminal to launch the test_MBM_and_MAGeT.py code. This still results in the test-code getting stuck at the rotational minctracc stage (running htop in the MINC-VM shows that the rotational minctracc stages are put to sleep but take up all of the CPU). We are hoping installing VirtualBox on a specific node on the cluster will fix this.

Is there anyway to specifically diagnose what is going wrong in our installation?

Kind Regards, Kyle Drover

dorkylever commented 5 years ago

Hi Ben and Gabriel,

I've noticed that bic-pipelines/pipeline_face.pl and bic-pipelines/pipeline_deface.pl are missing as well as MNI-startup when a different installation path is specified. Does perl have to be configured in a certain way to ensure the make installs these scripts?

minc-stuffs is installed in /usr/local/share/ and minc2-simple is installed in /usr/local/src/

Kind Regards, Kyle Drover

gdevenyi commented 5 years ago

@Dorky-Lever to help any further I think you need to fully describe the installation process you went through for minc-toolkit-v2 because it looks like the installation is broken in various ways.

Perhaps you can install from one of the packages available here (RPMs and DEBs available) https://packages.bic.mni.mcgill.ca/minc-toolkit/

dorkylever commented 5 years ago

Hi Gabriel and Ben,

Sorry I've taken a while to reply but I've realised why the install seemed broken on our server.

I compared the {PATH,LD_LIBRARY_PATH,CPATH,ANTSPATH,PYTHONPATH,PERL5LIB,MANPATH and MNI_DATAPATH} to what was set on the MINC-VM and realised that our install was missing directories (e.g. /usr/local/pipelines was missing). Potentially changing the default install path of minc-toolkit-v2 in the config file (i.e. {MINC_TOOLKIT} from /opt/minc/1.9.17 to /usr/local causes the make to miss adding these directories. I manually added the missing paths into my ~/.bash_profile. bash_profile.txt

Additionally, the equivalent MNI_DATAPATH folder (opt/minc/share) was missing. As the folder's contents were either masks or images, I transferred the folder to /usr/local/mnishare/ and specified {MNI_DATAPATH} to this location.

I'll raise these issues directly on the minc-toolkit-v2 page.

These are the current paths now specified: all_paths.txt

I've launched the MBM test code both using the shell script: launch_test_sh.txt

To rule out any path-related issues caused by qsub, I ran both locally (ssh'ing on to the compute-5-2 subnode; outputs_node_pipeline.log) and globally (qsub; outputs_pipeline.log ). I also tried to reduce the required run-time for the test-code by only using the img_10* test-images in the test-data directory (is there a way to unlink the number of exceutors with the number of images inputted?).

The test-code can now find mni-startup.pm and gets to approximately a third of the way through MBM.py, however, clients begin to shut-down at the second mincblur -clobber stage with no obvious error except that executors have exceeded idle time (see log files).

This was our make for minc-toolkit-v2. We ran the build ( my_build_sh.txt) , make and make install separately to tee 2&>1 their output into log files: mybuild.log myinstall.log mymakefin.log

These are the minc-toolkit-v2 config files: minc-toolkit-config_csh.txt minc-toolkit-config-sh.txt We can't find any obvious error that occurred during the make.

We ran default install for minc2-simple and minc-stuffs i.e:

mkdir build cd build cmake .. make make install

but we did these installs before I fixed the paths for minc-toolkit-v2. Should we run the make for these two libraries again now that minc-toolkit-v2 should be functional?

Is there any other files that I could provide that may be helpful?

Kind Regards, Kyle Drover Arkell Group - PhD candidate

dorkylever commented 5 years ago

Hi Everyone,

I was able to get the test_MBM_and_MAGeT.py test code to fully work (see below - I had to run the RMINC script locally within MINC-VM as our supercomputer's R version was outdated).

The following commands, however, results in rotational_minctracc not finding MNI-startup.pm during a bootstrap run:

Getting config-file and lsq12 protocol from from test-data

cd ~ wget http://repo.mouseimaging.ca/repo/Pydpiper_test_files/test-data.tar.gz tar xzvf test-data.tar.gz cd test-data && nano sample.cfg [removed all lines but sample]

Running MBM.py locally on supercomputer subnode:

ssh compute-5-2

launch_MBM.sh | tee bootstrap_fail.log

run_MBM_py.txt launch_MBM_sh.txt

bootstrap_fail.log

/home/kyle_arkell/eth_build_out/build_eth_wt_model_processed/18361e2/transforms/../log/rotational_minctracc.py/18361e2_lsq6.log no_startup_pm.txt

dorkylever commented 5 years ago

Performing the same same steps but specifying the test-image directory [i.e. files = glob.glob(("/home/kyle_arkell/test-data/test-images/*.mnc").format(**vars()))] fails as well.

dorkylever commented 5 years ago

Is there other executable test code for Pydpiper?

gdevenyi commented 5 years ago

If you're having trouble with MNI-startup.pm then you're minc-toolkit install still isn't properly configured or the cluster jobs are not inheriting the configuration. This is not a pydpiper bug. The other test run worked because rotational_minctracc probably isn't called in the other test.

Can you successfully run "autocrop" in the terminal before attempting job submission? If so, the issue is in the job submission/cluster configuration.

dorkylever commented 5 years ago

Hi Gabriel,

I can run autocrop in terminal. I'm confused, however, as rotational_minctracc.py is called within the first 40 stages of the test_MBM_and_MAGeT.py pipeline. --bootstrap called rotational_minctracc.py with default settings after the mincblur (i.e default_rotational_minctracc_conf in registration.py). It seems unlikely that a configuration issue would cause a non-default run to pass but a default run to fail.

is --resolution within MBM.py the voxel size in mm? I've tried to configure this by adding --resolution=0.02 (20 um as we use a FOV of 10 during our imaging protocol - see image below) into both MBM.py and the config file but it is getting ignored.

microCT_acquisition

Kind Regards, Kyle Drover

gdevenyi commented 5 years ago

Can you please complete complete logs rather than cropped logs where you think the error is. Also please don't rename the files on upload, its difficult to track the naming.

A few things:

Why are you launching the pipeline from within another python call? You can just do it directly from the command line, the extra complexity might be the origin of the problems. I'm not sure of the relationship of a subprocess call's environment vs the shell environment around it.
It looks like you're on SGE, I know that some SGE installs are misconfigured re: default shell which could explain the missing autocrop command, please try adding #$ -S /bin/bash to job submission
Yes, MINC internal resolution is in mm.

The logfiles indicate that your runtime minc-toolkit environment still isn't configured properly:

     13 Can't locate MNI/Startup.pm in @INC (@INC contains: /usr/local/perl5/lib/perl5/x86_64-linux-thread-multi /usr/local/perl5/
     14 lib/perl5 /usr/local/perl5/perl5lib /usr/local/src/maker/perl/lib /usr/local/src/maker/lib /usr/local/src/vcftools_0.1.12/
     15 lib/perl5/site_perl /opt/perl/lib/site_perl/5.14.2/x86_64-linux-thread-multi /opt/perl/lib/site_perl/5.14.2 /usr/local/src
     16 /bioperl-live//usr/local/perl5/lib/perl5 /usr/local/perl5/perl5lib /usr/local/src/maker/perl/lib /usr/local/src/maker/lib
     17 /usr/local/src/vcftools_0.1.12/lib/perl5/site_perl /opt/perl/lib/site_perl/5.14.2/x86_64-linux-thread-multi /opt/perl/lib/
     18 site_perl/5.14.2 /usr/local/src/bioperl-live/ /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /
     19 usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/local/bin/autocrop line 29.
     20 BEGIN failed--compilation aborted at /usr/local/bin/autocrop line 29.

Compare my environment, which has a minc path in in PERL5LIB: PERL5LIB=/opt/quarantine/minc-toolkit/1.9.17/install/pipeline:/opt/quarantine/minc-toolkit/1.9.17/install/perl If PERL5LIB exists in your terminal but not in the job, this is a job submission issue. Origins are possibly the extra python wrapping, or the SGE shell misconfig.

dorkylever commented 5 years ago

Hi Gabriel,

Thanks for the help. So far I think the error above was due to not specifying the right parameters for a bootstrap run (i.e. I didn't have --resolution=0.02, ---no-nuc and --no-common-space-registration within my MBM call). Specifying these parameters got the call past the first 40 stages.

I've modified my launch_MBM.sh as shown below (I've stopped launching the pipeline with a python call). launch_MBM.sh.txt

These are the steps that I am now using:

ssh compute-5-2 #this is a particular sub-node on our cluster that is constantly empty. ./launch_MBM.sh

Output

It's a bit hard to give the specific output, however, as I noticed that some stages are hanging when the pipeline is initially called. Keyboard interrupting (Ctrl-C) and relaunching the pipeline (./launch_MBM.sh) causes these stages to pass - I'm assuming these stages are run out of order??? This is the output from my latest restart - the pipeline is currently stuck on a mincresample step (shown in within the top screenshot below) - I'll notify you if it finishes automatically.

Warning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 [/opt/python3/lib/python3.5/importlib/_bootstrap.py:222]

Warning: Not masking your images from atlas masks after LSQ6 alignment ... probably not what you want (this can have negative effects on your registration and statistics) [/opt/python3/lib/python3.5/site-packages/pydpiper-2.0.13-py3.5.egg/EGG-INFO/scripts/MBM.py:315]

Total number of stages in the pipeline: 2682 Number of stages already processed: 58 Daemon is running at: 10.1.255.244:39474 The pipeline's uri is: PYRO:obj_c1107ca3d87c4750923af52d784f7a93@10.1.255.244:39474 . Submitted 1 executors (clients) to the queue. Waiting for them to register with the server...

Client registered (Eh!): PYRO:obj_e52d49bb658c452d9d822696b9c66038@10.1.255.244:48232.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^[[D.............................................................................................................................................................................................................................................................................................^[[B......................................................................................................^[[B................................................................................................................................................................................................................................................................................

This is output from the uri_file:

cd pydpiper-master/pydpiper/execution/ check_pipeline_status.py /home/kyle_arkell/test_bootstrap_res_new_uri

Total number of stages in the pipeline: 2682 Number of stages already processed: 58

Number of active clients: 1 Number of clients waiting in the queue: 0

Currently running stages (1): 0 autocrop -clobber -isostep 0.02 -extend 0,0 0,0 0,0 /home/kyle_arkell/zic2_eth_hannah_study/wt_Eth/18285e2_wt.mnc /home/kyle_arkell/outputs_hannah_test_res/test_bootstrap_res_new_reg_target_resampled/18285e2_wt_20micron_resampled.mnc

Number of runnable stages: 0

Number of failed stages: 0 Number of failed/lost/dead executors: 0

Memory requirement of runnable stages: [] Memory available in registered clients: [2.3156]

Which directory is your autocrop located in? mine is located in /usr/local/bin i.e.:

which autocrop /usr/local/bin/autocrop

This is my perl5lib environment (both pipeline and perl folders are there):

echo $PERL5LIB /usr/local/perl:/usr/local/pipeline:/usr/local/perl5/lib/perl5:/usr/local/perl5/perl5lib:/usr/local/src/maker/perl/lib:/usr/local/src/maker/lib:/usr/local/src/vcftools_0.1.12/lib/perl5/site_perl:/opt/perl/lib/site_perl/5.14.2:/usr/local/src/bioperl-live/

Response to a few things:

I originally had trouble with using --files /home/kyle_arkell/zic2_eth_hannah_study/wt_Eth/.mnc which would not recognise .mnc within the launch_MBM.sh shell. I'm not sure why it wouldn't originally work but the modified launch_MBM.sh will now successfully launch the job.
I haven't been using SGE lately and have been running jobs locally on empty subnodes instead i.e.

ssh compute-5-0 ./launch_MBM.sh
the --resolution parameter is working as it should now (I was calling the wrong run_MBM.py script in my run)

gdevenyi commented 5 years ago

--resolution=0.02 is 20um. The pipeline hasn't hung. Its running mincresample and taking forever because of the inappropriate resolution.

dorkylever commented 5 years ago

Hi Gabriel,

I removed --resolution=0.02 from my run but still get an error. i.e.:

MBM.py --pipeline-name test_bootstrap_no_res --num-executors 40 --no-nuc --no-common-space-registration --verbose --bootstrap --config-file /home/kyle_arkell/arkell.cfg --lsq6-large-rotations --no-run-maget --maget-no-mask --lsq12-protocol /home/kyle_arkell/test-data/default_linear_MAGeT_prot.csv --output-dir /home/kyle_arkell/outputs_hannah_test_node/ --files /home/kyle_arkell/zic2_eth_hannah_study/wt_Eth/*.mnc

Total number of stages in the pipeline: 2681 Number of stages already processed: 55 Daemon is running at: 10.1.255.242:49930 The pipeline's uri is: PYRO:obj_47e504429afe46298218d66b4fd6ae7c@10.1.255.242:49930 . Submitted 29 executors (clients) to the queue. Waiting for them to register with the server...

Client registered (Eh!): PYRO:obj_c2df321219634d5f8cf73f074782d4ed@10.1.255.242:60843 Client registered (Eh!): PYRO:obj_e153f3e00911486fa9d088830a412854@10.1.255.242:55143 Client registered (Eh!): PYRO:obj_b231166611ab45c58b416e2e4c701ce8@10.1.255.242:36682 Client registered (Eh!): PYRO:obj_f5d605011e17435982271c7e1041dbf9@10.1.255.242:60560 Client registered (Eh!): PYRO:obj_f02772175328499e9646121b2215c5ee@10.1.255.242:55910 Client registered (Eh!): PYRO:obj_f9d4a648948b4234899b6aced2d2b4c8@10.1.255.242:57166 Client registered (Eh!): PYRO:obj_56a26b176ba1494e8062699ab27501bf@10.1.255.242:50252 Client registered (Eh!): PYRO:obj_569ef5852e3441aca3be2793002aaf2d@10.1.255.242:54236 Client registered (Eh!): PYRO:obj_aa9d431df32f4b38931cbdc00d8635b8@10.1.255.242:52780 Client registered (Eh!): PYRO:obj_f94bbb81cce947579c15b132678e5bf8@10.1.255.242:54490 Client registered (Eh!): PYRO:obj_f3fd475c6e7e4de79bb4dfe04dd78010@10.1.255.242:54164 Client registered (Eh!): PYRO:obj_92b013c0382e48e685c16d83844ab51d@10.1.255.242:55412 Client registered (Eh!): PYRO:obj_0e702515a6db48eab081665a20ef6b5c@10.1.255.242:37254 Client registered (Eh!): PYRO:obj_b4998d57c78d4cf89eabf2e30964a923@10.1.255.242:54287 Client registered (Eh!): PYRO:obj_78813121c0084dafbc0148481d8f3374@10.1.255.242:49486 Client registered (Eh!): PYRO:obj_f691fc3b285d4d1bad03e0649b306d98@10.1.255.242:55425 Client registered (Eh!): PYRO:obj_5b8e83d3844b4af29b5b6389748bdf52@10.1.255.242:57346 Client registered (Eh!): PYRO:obj_defb3d99fcce4721931b82adb733eed6@10.1.255.242:46757 Client registered (Eh!): PYRO:obj_43f28b0baaf140d69bb770b39cd68296@10.1.255.242:50626 Client registered (Eh!): PYRO:obj_7849994b36744a4e816b1c661af6f953@10.1.255.242:44078 Client registered (Eh!): PYRO:obj_7a5c2a5fef7d403f91037eb16165ba7c@10.1.255.242:39402 Client registered (Eh!): PYRO:obj_7cb4e55c80224651aa7fae6844352566@10.1.255.242:57406 Client registered (Eh!): PYRO:obj_c92e1595af2c46ae917275691555d6db@10.1.255.242:37206 Client registered (Eh!): PYRO:obj_8a54b3ab7c65413a93ef14606eb83f86@10.1.255.242:41336 Client registered (Eh!): PYRO:obj_9e7e76ee33384a599ec61edc7e665522@10.1.255.242:38637 Client registered (Eh!): PYRO:obj_db6c808843054ffbae51f5a0f9df2bca@10.1.255.242:50469 Client registered (Eh!): PYRO:obj_8047eea83278461b8aa6aa670aedc675@10.1.255.242:33871 Client registered (Eh!): PYRO:obj_29c492221d0d4cf29b069c7eec44b776@10.1.255.242:41436 Client registered (Eh!): PYRO:obj_7bfc53349a9f4994b86e99f300e6c98d@10.1.255.242:49981. ERROR in Stage 40: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18341e5_wt/tmp/18341e5_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18341e5_wt/transforms/18341e5_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18341e5_wt/transforms/../log/rotational_minctracc.py/18341e5_wt_lsq6.log

ERROR in Stage 26: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18338e5_wt/tmp/18338e5_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18338e5_wt/transforms/18338e5_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18338e5_wt/transforms/../log/rotational_minctracc.py/18338e5_wt_lsq6.log

ERROR in Stage 2: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18285e2_wt/transforms/18285e2_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18285e2_wt/transforms/../log/rotational_minctracc.py/18285e2_wt_lsq6.log

ERROR in Stage 10: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18287e10_wt/tmp/18287e10_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18287e10_wt/transforms/18287e10_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18287e10_wt/transforms/../log/rotational_minctracc.py/18287e10_wt_lsq6.log

ERROR in Stage 20: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18334e5_wt/tmp/18334e5_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18334e5_wt/transforms/18334e5_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18334e5_wt/transforms/../log/rotational_minctracc.py/18334e5_wt_lsq6.log

ERROR in Stage 12: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18287e9_wt/tmp/18287e9_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18287e9_wt/transforms/18287e9_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18287e9_wt/transforms/../log/rotational_minctracc.py/18287e9_wt_lsq6.log

ERROR in Stage 48: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18361e2_wt/tmp/18361e2_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18361e2_wt/transforms/18361e2_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18361e2_wt/transforms/../log/rotational_minctracc.py/18361e2_wt_lsq6.log

ERROR in Stage 28: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18338e6_wt/tmp/18338e6_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18338e6_wt/transforms/18338e6_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18338e6_wt/transforms/../log/rotational_minctracc.py/18338e6_wt_lsq6.log

ERROR in Stage 34: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18339e7_wt/tmp/18339e7_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18339e7_wt/transforms/18339e7_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18339e7_wt/transforms/../log/rotational_minctracc.py/18339e7_wt_lsq6.log

ERROR in Stage 58: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18361e9_wt/tmp/18361e9_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18361e9_wt/transforms/18361e9_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18361e9_wt/transforms/../log/rotational_minctracc.py/18361e9_wt_lsq6.log

ERROR in Stage 52: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18361e4_wt/tmp/18361e4_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18361e4_wt/transforms/18361e4_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18361e4_wt/transforms/../log/rotational_minctracc.py/18361e4_wt_lsq6.log

ERROR in Stage 6: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18285e7_wt/tmp/18285e7_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18285e7_wt/transforms/18285e7_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18285e7_wt/transforms/../log/rotational_minctracc.py/18285e7_wt_lsq6.log ERROR in Stage 46: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyl e_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18361e1_wt/tmp/18361e1_wt_fwhm5.0_blur.mnc /home/kyle_ar kell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_a rkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18361e1_wt/transforms/18361e1_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18361e1_wt/transforms/../log/rotational_minctra cc.py/18361e1_wt_lsq6.log

ERROR in Stage 44: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyl e_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18341e7_wt/tmp/18341e7_wt_fwhm5.0_blur.mnc /home/kyle_ar kell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_a rkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18341e7_wt/transforms/18341e7_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18341e7_wt/transforms/../log/rotational_minctra cc.py/18341e7_wt_lsq6.log

ERROR in Stage 8: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyle _arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18285e8_wt/tmp/18285e8_wt_fwhm5.0_blur.mnc /home/kyle_ark ell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_ar kell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18285e8_wt/transforms/18285e8_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18285e8_wt/transforms/../log/rotational_minctra cc.py/18285e8_wt_lsq6.log

ERROR in Stage 54: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18361e5_wt/tmp/18361e5_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18361e5_wt/transforms/18361e5_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18361e5_wt/transforms/../log/rotational_minctracc.py/18361e5_wt_lsq6.log

ERROR in Stage 42: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18341e6_wt/tmp/18341e6_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18341e6_wt/transforms/18341e6_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18341e6_wt/transforms/../log/rotational_minctracc.py/18341e6_wt_lsq6.log

ERROR in Stage 4: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18285e5_wt/tmp/18285e5_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18285e5_wt/transforms/18285e5_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18285e5_wt/transforms/../log/rotational_minctracc.py/18285e5_wt_lsq6.log

ERROR in Stage 50: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyl e_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18361e3_wt/tmp/18361e3_wt_fwhm5.0_blur.mnc /home/kyle_ar kell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_a rkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18361e3_wt/transforms/18361e3_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18361e3_wt/transforms/../log/rotational_minctra cc.py/18361e3_wt_lsq6.log

............ Client un-registered (Cheers!): PYRO:obj_7a5c2a5fef7d403f91037eb16165ba7c@10.1.255.242:39402 Client un-registered (Cheers!): PYRO:obj_7cb4e55c80224651aa7fae6844352566@10.1.255.242:57406 Client un-registered (Cheers!): PYRO:obj_c92e1595af2c46ae917275691555d6db@10.1.255.242:37206 Client un-registered (Cheers!): PYRO:obj_f02772175328499e9646121b2215c5ee@10.1.255.242:55910 Client un-registered (Cheers!): PYRO:obj_f3fd475c6e7e4de79bb4dfe04dd78010@10.1.255.242:54164 Client un-registered (Cheers!): PYRO:obj_f9d4a648948b4234899b6aced2d2b4c8@10.1.255.242:57166 Client un-registered (Cheers!): PYRO:obj_f691fc3b285d4d1bad03e0649b306d98@10.1.255.242:55425 Client un-registered (Cheers!): PYRO:obj_0e702515a6db48eab081665a20ef6b5c@10.1.255.242:37254 Client un-registered (Cheers!): PYRO:obj_b4998d57c78d4cf89eabf2e30964a923@10.1.255.242:54287 Client un-registered (Cheers!): PYRO:obj_8a54b3ab7c65413a93ef14606eb83f86@10.1.255.242:41336 Client un-registered (Cheers!): PYRO:obj_9e7e76ee33384a599ec61edc7e665522@10.1.255.242:38637 Client un-registered (Cheers!): PYRO:obj_7849994b36744a4e816b1c661af6f953@10.1.255.242:44078 Client un-registered (Cheers!): PYRO:obj_c2df321219634d5f8cf73f074782d4ed@10.1.255.242:60843 Client un-registered (Cheers!): PYRO:obj_db6c808843054ffbae51f5a0f9df2bca@10.1.255.242:50469 Client un-registered (Cheers!): PYRO:obj_b231166611ab45c58b416e2e4c701ce8@10.1.255.242:36682 Client un-registered (Cheers!): PYRO:obj_7bfc53349a9f4994b86e99f300e6c98d@10.1.255.242:49981 Client un-registered (Cheers!): PYRO:obj_29c492221d0d4cf29b069c7eec44b776@10.1.255.242:41436 Client un-registered (Cheers!): PYRO:obj_e153f3e00911486fa9d088830a412854@10.1.255.242:55143 Client un-registered (Cheers!): PYRO:obj_f5d605011e17435982271c7e1041dbf9@10.1.255.242:60560.............................. ......... ERROR in Stage 30: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18338e9_wt/tmp/18338e9_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18338e9_wt/transforms/18338e9_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18338e9_wt/transforms/../log/rotational_minctracc.py/18338e9_wt_lsq6.log

...... ERROR in Stage 56: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18361e6_wt/tmp/18361e6_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18361e6_wt/transforms/18361e6_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18361e6_wt/transforms/../log/rotational_minctracc.py/18361e6_wt_lsq6.log

...... Client un-registered (Cheers!): PYRO:obj_78813121c0084dafbc0148481d8f3374@10.1.255.242:49486.... ERROR in Stage 32: rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18339e1_wt/tmp/18339e1_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18339e1_wt/transforms/18339e1_wt_lsq6.xfm /dev/null Logfile for (potentially) more information: /home/kyle_arkell/outputs_hannah_test_node/test_bootstrap_no_res_processed/18339e1_wt/transforms/../log/rotational_minctracc.py/18339e1_wt_lsq6.log

.. Client un-registered (Cheers!): PYRO:obj_8047eea83278461b8aa6aa670aedc675@10.1.255.242:33871.......... Client un-registered (Cheers!): PYRO:obj_5b8e83d3844b4af29b5b6389748bdf52@10.1.255.242:57346...............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Running one of the rotational_minctracc stages separately (Stage 40):

rotational_minctracc.py -t /dev/shm/ -w 8.0,8.0,8.0 -s 4.0 -g 10.0 -r 50 -i 10 --simplex 20.0 /home/kyle_arkell/outputs_hannah_test_no_res/test_bootstrap_no_res_processed/18285e5_wt/tmp/18285e5_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_no_res/test_bootstrap_no_res_bootstrap_file/18285e2_wt/tmp/18285e2_wt_fwhm5.0_blur.mnc /home/kyle_arkell/outputs_hannah_test_no_res/test_bootstrap_no_res_processed/18285e5_wt/transforms/18285e5_wt_lsq6.xfm /dev/null

perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LANGUAGE = (unset), LC_ALL = (unset), LANG = "C.UTF-8" are supported and installed on your system. perl: warning: Falling back to the standard locale ("C"). /usr/local/src/minc-toolkit-v2/libminc/libsrc/netcdf_convenience.c:573 (from miopen): Unable to open file '/home/kyle_arkell/outputs_hannah_test_no_res/test_bootstrap_no_res_processed/18285e5_wt/tmp/18285e5_wt_fwhm5.0_blur.mnc' mincinfo: Error opening file "/home/kyle_arkell/outputs_hannah_test_no_res/test_bootstrap_no_res_processed/18285e5_wt/tmp/18285e5_wt_fwhm5.0_blur.mnc" autocrop: crashed while running mincinfo (termination status=256) TMP: /dev/shm/ RANGE: 50 INTERVAL: 10 Traceback (most recent call last): File "/opt/python3/bin/rotational_minctracc.py", line 4, in import('pkg_resources').run_script('python-stuffs==0.1.24', 'rotational_minctracc.py') File "/opt/python3/lib/python3.5/site-packages/pkg_resources/init.py", line 666, in run_script self.require(requires)[0].run_script(script_name, ns) File "/opt/python3/lib/python3.5/site-packages/pkg_resources/init.py", line 1460, in run_script exec(script_code, namespace, namespace) File "/opt/python3/lib/python3.5/site-packages/python_stuffs-0.1.24-py3.5.egg/EGG-INFO/scripts/rotational_minctracc.py", line 422, in File "/opt/python3/lib/python3.5/site-packages/python_stuffs-0.1.24-py3.5.egg/EGG-INFO/scripts/rotational_minctracc.py", line 397, in main File "/opt/python3/lib/python3.5/site-packages/python_stuffs-0.1.24-py3.5.egg/EGG-INFO/scripts/rotational_minctracc.py", line 317, in downsample File "/opt/python3/lib/python3.5/subprocess.py", line 584, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['autocrop', '-isostep', '4.0', '/home/kyle_arkell/outputs_hannah_test_no_res/test_bootstrap_no_res_processed/18285e5_wt/tmp/18285e5_wt_fwhm5.0_blur.mnc', '//dev/shm//rot_4335/rot_0..mnc']' returned non-zero exit status 1

Using Display will not open the .mnc file i.e.:

Display /home/kyle_arkell/outputs_hannah_test_no_res/test_bootstrap_no_res_processed/18285e5_wt/tmp/18285e5_wt_fwhm5.0_blur.mnc

Input /home/kyle_arkell/outputs_hannah_test_no_res/test_bootstrap_no_res_processed/18285e5_wt/tmp/18285e5_wt_fwhm5.0_blur.mnc /usr/local/src/minc-toolkit-v2/libminc/libsrc/netcdf_convenience.c:573 (from miopen): Unable to open file '/home/kyle_arkell/outputs_hannah_test_no_res/test_bootstrap_no_res_processed/18285e5_wt/tmp/18285e5_wt_fwhm5.0_blur.mnc' Error: opening MINC file "/home/kyle_arkell/outputs_hannah_test_no_res/test_bootstrap_no_res_processed/18285e5_wt/tmp/18285e5_wt_fwhm5.0_blur.mnc". Error loading /home/kyle_arkell/outputs_hannah_test_no_res/test_bootstrap_no_res_processed/18285e5_wt/tmp/18285e5_wt_fwhm5.0_blur.mnc

I presume 18285e5_wt_fwhm5.0_blur.mnc is somehow getting corrupted during the run??? Is it because MBM.py is incorrectly determining the resolution within my run??

The image files I'm using are raw uCT scans that have been converted from DICOM (i.e. a DICOM stack) --> Drishti NetCDF volumes (pvl.nc) --> .tiff --> .mnc files. Should I additionally downsample these volumes to the 28um resolution originally in Wong et al 2012 (https://dev.biologists.org/content/139/17/3248?ijkey=9f354748ba37ab1e0b928a18c9409c43fd170a22&keytype2=tf_ipsecsha)?

If so, I'm not sure how to do this....

gdevenyi commented 5 years ago

Okay, so here we have some trickle truth coming out. This is your first mention you're not using MRI data.

First:

 DICOM (i.e. a DICOM stack) --> Drishti NetCDF volumes (pvl.nc) --> .tiff --> .mnc

dcm2mnc is a program in minc-toolkit. Please use that.

--num-executors 40

If you truely have 20 um whole-brain data, 40 jobs running on a single machine at that resolution is massively overloaded. If you were to inspect the system logs I suspect you would see that jobs are being OOM killed. Set this to 1. If you want to properly use the cluster with job submission and memory requesting, then that number can be increased.

dorkylever commented 5 years ago

For clarification, they are microCT scans of embryos from a Quantum FX micro CT.

dorkylever commented 5 years ago

Hi Gabriel,

1) There's a complication to directly using dcm2mnc.

My raw DICOM images contain the stage/sample bed from the Quantum FX uCT scanner (see image below - it is a slice of the raw scan viewed within Drishti Import). This would interfere with the volume analysis, especially as each embryo was positioned differently on the sample bed during scanning. sample_bed_labbeled

I've been using the convoluted image conversion process:

DICOM (i.e. a DICOM stack) --> Drishti NetCDF volumes (pvl.nc) --> .tiff --> .mnc

as I've been using using Drishti (https://github.com/nci/drishti) to remove this sample bed from the image but this software can only save the images as either (png, jpeg, tiff, or bmp).

Is there another software available to manually remove undesired volume and save the processed volume as a DICOM directory? If not, do you think binning my resolution by 2 at each axis (x,y,z) will sufficiently reduce the computation demand?

I'm currently trying to contact the creator of Drishti (Ajay Limaye) who is located at ANU to see whether Drishti can or can be modified to save volumes as DICOM directories. (https://github.com/nci/drishti/issues/35).

2) I'm confused by why you want to set the number of executors to 1. The mincresample stage that was taking forever was only using a singe core. I am running the job on a subnode that has 1000G of RAM and 100 cores (I'm unsure about what GHz / cache each core has).

Kind Regards, Kyle Drover

dorkylever commented 5 years ago

May I please ask if you have used microCT scans for Pydpiper before and what was your original file format for your raw microCT files?

gdevenyi commented 5 years ago

I have never done any microCT processing with pydpiper only MRI.

I understand you may have FOV issues. If you're willing to share an example file I can probably hack together an automatic extraction of the embryo data from the MINC file and share the script.

From what I've learned here, you're operating far outside the original design features of pydpiper. It can handle this data, but you're going to need to learn the intricacies and make some choices, rather than relying on the defaults. My first step would be to process this data are much lower resolution (100um+) and get produce a low-res atlas to use as a starting point for a high res run.

mincresample will take forever with your data. It was designed in a world when 2GB was a lot of ram, it can handle arbitrarily sized data (like yours) because it does it piece by piece. The tradeoff is it seems "slow" now, because it needs to do I/O continuously.

dorkylever commented 5 years ago

Hi Gabriel,

I would greatly appreciate your help and I'm happy to help with editing/writing the script etc... I'll do the straight DICOM --> MINC conversion and then release the images on a new Github page. I'm quite willing to learn all the intricacies and have support if this starts to get extremely complex. My supervisor Ruth Arkell and I would be happy to have you as a collaborator and receive authorship for the appropriate publications.

My understanding was that MBM.py and as such Pydpiper would be based around uCT data as the original MBM.py (Wong et al 2014 - https://dev.biologists.org/content/141/12/2533.long) used uCT data (they ran their analysis on images with 1000 500 500 voxel units at 28um resolution).

Can Pydpiper executors be split across multiple subnodes to improve the read/write speed if this is the rate limiting step?

gdevenyi commented 5 years ago

This tool and design concept is a lot older than 2014, its based on mice-build-model.pl, written earlier than 2011 (https://wiki.mouseimaging.ca/display/MICePub/MiceBuildModelUsage).

It was designed and originally validated using MRI data. Its since been used for uCT and other light sheet microscopy among other things, but they require effort. I suggest you contact the first author of that paper for advice on custom configuration for uCT data.

gdevenyi commented 5 years ago

Of course I'm happy to help in any way that I can, but it will require a lot of experimentation compared to getting help from the authors.

dorkylever commented 5 years ago

Hi Gabriel,

I've created a new Github repository and added you as a contributor: https://github.com/Dorky-Lever/Ark_Pydpiper

I'm trying to upload the files but I'm having trouble compressing the MNC files from ~200MB to Github's 25MB limit (I've used mincconvert, gzip and tar on max compression so far). Is there a different method you know / another way to send you the images?

gdevenyi commented 5 years ago

google drive, dropbox, wetransfer are all options for large files.

Github is not a place for binary files.

dorkylever commented 5 years ago

Thanks Gabriel,

I've just shared an One-Drive folder (Arkell_Pydpiper) through to both gabriel.devenyi@mcgill.ca and contact@cobralab.ca.

Kind Regards, Kyle Drover

dorkylever commented 5 years ago

Hi Gabriel, I've wrote a quick script that does the job using mincreshape and volrot (https://github.com/Dorky-Lever/Ark_Pydpiper/blob/master/Image_Processing/12_5_stage_crop.sh).

My supervisor is trying to organise a meeting with representatives of the first author mentioned above to discuss custom settings.

Thank you for everything that you done.

Kind Regards, Kyle Drover

Mouse-Imaging-Centre / pydpiper

v2.013 MBM can't find Startup.pm specifically when --bootstrap is specified #425

Getting config-file and lsq12 protocol from from test-data

Running MBM.py locally on supercomputer subnode:

These are the steps that I am now using:

Output

Response to a few things: