Open yarikoptic opened 5 years ago
actually now I am even more confused since the fsln.simg within VM is neurodocker generated but has file installed... ok, apparently there is more Singularity files produced while preparing VM by @mjtravers
(git-annex)hopa:~/proj/repronim/training/sfn2018-training[master]vm
$> git grep Singularity.fsl
files/neurodocker.sh: --add-to-entrypoint "source /etc/fsl/5.0/fsl.sh" > Singularity.fsl
scripts/section3.sh: --run-bash "ln -fs /etc/fsl/5.0/fsl.sh /.singularity.d/env/99-fsl.sh" > Singularity.fsl
scripts/section3.sh:sudo singularity build fsl.simg Singularity.fsl
and one of those also fixed up to install file
:
$> git grep file scripts/section3.sh files/neurodocker.sh
files/neurodocker.sh: --add-to-entrypoint "source /etc/fsl/5.0/fsl.sh" > Dockerfile.fsl
scripts/section3.sh: --install fsl-5.0-core fsl-mni152-templates file \
scripts/section3.sh: --install fsl-5.0-core fsl-mni152-templates file \
I think that VM preparation shouldn't duplicate (although now at least I have the neurodocker invocations to use) creation of those Singularity files. We should have them generated/stored in a single location, build images from them, and cache them within VM from the shub.
I would keep my vote on obliterating custom environments within VM per each section. Ideally generation of a VM should be just a list of apt-get install commands, and a few of "singularity pull" ones to cache images. This would eliminate this wide and confusing discrepancy between images/environments.
@yarikoptic - i am fine with a single environment, but somebody will have to take the responsibility of coordinating all the sections to a uniform environment.
for ohbm, fair needed specific python repos (master versions), datalad needed specific master versions of neuroimaging and containers-run, and then jupyter/R were needed for the other pieces.
i think creating environments per project/course is actually more of a practical teaching point than a monolithic one. monolithic environments are very fragile for research.
monolithic environments are very fragile for research.
I could argue so well on this one, but it is Xmas ;)
merry xmas to you too! happy to "discuss" anytime after xmas :)
Me and @kyleam wasted hours trying to figure WTF is going on and why the
feat
call is failing for us. In my case it was a combination of issues:Cause: obnoxious old override of LD_LIBRARY_PATH in my .bashrc. Summary: We must not use non-sanitized call to singularity run. Relevant PR/discussion: https://github.com/datalad/datalad-container/pull/50
Cause: Multiple flavors of the same container -- one was used during the course (cached under ~/.images) prepared by @mih, but urls pointed to shub:// instances with suffix 'n' (neurodocker version, generated by @satra), of which "fsln" was not entirely correct - was missing
file
needed package. @mih worked around for that in his Singularity file. I now added/pushed to git within fsl packaging (https://salsa.debian.org/neurodebian-team/fsl) - we build/release now Sumary (for the next round):name
andnamen
flavors of the same image - there must be only one!Additional oddity which I have no clue yet about -- .fsl and .fsln files are different at https://github.com/ReproNim/ohbm2018-training/tree/master/section23/environments BUT on singularity hub shows identical Singularityfile for both and the
fsl
one also lacksfile
package. Apparently at some point both were having the same Singularity recipe which @satra fixed quickly in https://github.com/ReproNim/ohbm2018-training/commit/ee3e3caa104c527e963318e75ac8edec263695c1#diff-33c1d6f073cc57d2b44a74f5f9716a34 replacing with Michael's version. Not sure if new image was not regenerated because it was configured to have build manually triggered or a bug in singularity hub somehow.My course of action now will be
datalad run
to register how those files were generated. And adjust text accordingly (if not fsln)any feedback etc is welcome.