search

ReproNim / sfn2018-training

Materials for SfN 2018 training event
http://repronim.org/sfn2018-training
2 stars 5 forks source link

VM containers must be identical to the one! on shub:// which must be working correctly #24

Open yarikoptic opened 5 years ago

yarikoptic commented 5 years ago

Me and @kyleam wasted hours trying to figure WTF is going on and why the feat call is failing for us. In my case it was a combination of issues:

  1. Cause: obnoxious old override of LD_LIBRARY_PATH in my .bashrc. Summary: We must not use non-sanitized call to singularity run. Relevant PR/discussion: https://github.com/datalad/datalad-container/pull/50

  2. Cause: Multiple flavors of the same container -- one was used during the course (cached under ~/.images) prepared by @mih, but urls pointed to shub:// instances with suffix 'n' (neurodocker version, generated by @satra), of which "fsln" was not entirely correct - was missing file needed package. @mih worked around for that in his Singularity file. I now added/pushed to git within fsl packaging (https://salsa.debian.org/neurodebian-team/fsl) - we build/release now Sumary (for the next round):

    • we need to triple check that cached in VM images are identical to the ones on shub://
    • we should not have name and namen flavors of the same image - there must be only one!
    • location of Singularity recipes should be a single place (not different subdirs across different branches)
    • neurodocker invocations should be scripted, at least until https://github.com/kaczmarj/neurodocker/issues/213 is resolved, so it would be feasible to quickly adjust and regenerate target Singularity files

Additional oddity which I have no clue yet about -- .fsl and .fsln files are different at https://github.com/ReproNim/ohbm2018-training/tree/master/section23/environments BUT on singularity hub shows identical Singularityfile for both and the fsl one also lacks file package. Apparently at some point both were having the same Singularity recipe which @satra fixed quickly in https://github.com/ReproNim/ohbm2018-training/commit/ee3e3caa104c527e963318e75ac8edec263695c1#diff-33c1d6f073cc57d2b44a74f5f9716a34 replacing with Michael's version. Not sure if new image was not regenerated because it was configured to have build manually triggered or a bug in singularity hub somehow.

My course of action now will be

any feedback etc is welcome.

yarikoptic commented 5 years ago

actually now I am even more confused since the fsln.simg within VM is neurodocker generated but has file installed... ok, apparently there is more Singularity files produced while preparing VM by @mjtravers 

(git-annex)hopa:~/proj/repronim/training/sfn2018-training[master]vm
$> git grep Singularity.fsl
files/neurodocker.sh:  --add-to-entrypoint "source /etc/fsl/5.0/fsl.sh" > Singularity.fsl
scripts/section3.sh:  --run-bash "ln -fs /etc/fsl/5.0/fsl.sh /.singularity.d/env/99-fsl.sh" > Singularity.fsl
scripts/section3.sh:sudo singularity build fsl.simg Singularity.fsl

and one of those also fixed up to install file:

$> git grep file scripts/section3.sh files/neurodocker.sh 
files/neurodocker.sh:  --add-to-entrypoint "source /etc/fsl/5.0/fsl.sh" > Dockerfile.fsl
scripts/section3.sh:  --install fsl-5.0-core fsl-mni152-templates file \
scripts/section3.sh:  --install fsl-5.0-core fsl-mni152-templates file \

I think that VM preparation shouldn't duplicate (although now at least I have the neurodocker invocations to use) creation of those Singularity files. We should have them generated/stored in a single location, build images from them, and cache them within VM from the shub.

I would keep my vote on obliterating custom environments within VM per each section. Ideally generation of a VM should be just a list of apt-get install commands, and a few of "singularity pull" ones to cache images. This would eliminate this wide and confusing discrepancy between images/environments.

satra commented 5 years ago

@yarikoptic - i am fine with a single environment, but somebody will have to take the responsibility of coordinating all the sections to a uniform environment.

for ohbm, fair needed specific python repos (master versions), datalad needed specific master versions of neuroimaging and containers-run, and then jupyter/R were needed for the other pieces.

i think creating environments per project/course is actually more of a practical teaching point than a monolithic one. monolithic environments are very fragile for research.

yarikoptic commented 5 years ago

monolithic environments are very fragile for research.

I could argue so well on this one, but it is Xmas ;)

satra commented 5 years ago

merry xmas to you too! happy to "discuss" anytime after xmas :)