Neurodocker 0.9.2 and 0.9.3 produce non-buildable Dockerfiles (does not happen with 0.9.1)

[JohannesWiesner]

We are using this code to produce a Dockerfile (Note that Dockerfile needs example-env.yml to successfully build the image as found in our repo). Creating the Dockerfile works fine but the building process stops with the following error when using repronim/neurodocker:0.9.2 or repronim/neurodocker:0.9.3 (with repronim/neurodocker:0.9.1 everything works fine):

(csp_wiesner_johannes) johannes.wiesner@pf41jell:~/work/repos/csp_neurodocker$ docker build -t csp_neurodocker:test .
Sending build context to Docker daemon  284.2kB
Step 1/22 : FROM neurodebian:stretch-non-free
 ---> a74190ba9142
Step 2/22 : ARG DEBIAN_FRONTEND=noninteractive
 ---> Using cache
 ---> 9792cd04cf1e
Step 3/22 : RUN apt-get update -qq            && apt-get install -y --quiet                   afni                   ants                   fsl                   g++                   gcc                   mricron                   octave            && rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 527b6d9c31ff
Step 4/22 : ENV FORCE_SPMMCR="1"     SPM_HTML_BROWSER="0"     SPMMCRCMD="/opt/spm12-r7771/run_spm12.sh /opt/matlab-compiler-runtime-2010a/v713 script"     LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu:/opt/matlab-compiler-runtime-2010a/v713/runtime/glnxa64:/opt/matlab-compiler-runtime-2010a/v713/bin/glnxa64:/opt/matlab-compiler-runtime-2010a/v713/sys/os/glnxa64:/opt/matlab-compiler-runtime-2010a/v713/extern/bin/glnxa64"     MATLABCMD="/opt/matlab-compiler-runtime-2010a/v713/toolbox/matlab"
 ---> Using cache
 ---> 996e805174a5
Step 5/22 : RUN export TMPDIR="$(mktemp -d)"     && apt-get update -qq     && apt-get install -y -q --no-install-recommends            bc            ca-certificates            curl            libncurses5            libxext6            libxmu6            libxpm-dev            libxt6            multiarch-support            unzip     && rm -rf /var/lib/apt/lists/*     && _reproenv_tmppath="$(mktemp -t tmp.XXXXXXXXXX.deb)"     && curl -fsSL --retry 5 -o "${_reproenv_tmppath}" http://mirrors.kernel.org/debian/pool/main/libx/libxp/libxp6_1.0.2-2_amd64.deb     && apt-get install --yes -q "${_reproenv_tmppath}"     && rm "${_reproenv_tmppath}"     && apt-get update -qq     && apt-get install --yes --quiet --fix-missing     && rm -rf /var/lib/apt/lists/*     && echo "Downloading MATLAB Compiler Runtime ..."     && curl -fL -o "$TMPDIR/MCRInstaller.bin" https://dl.dropbox.com/s/zz6me0c3v4yq5fd/MCR_R2010a_glnxa64_installer.bin     && chmod +x "$TMPDIR/MCRInstaller.bin"     && "$TMPDIR/MCRInstaller.bin" -silent -P installLocation="/opt/matlab-compiler-runtime-2010a"     && rm -rf "$TMPDIR"     && unset TMPDIR     && echo "Downloading standalone SPM12 ..."     && curl -fL -o /tmp/spm12.zip https://www.fil.ion.ucl.ac.uk/spm/download/restricted/utopia/previous/spm12_r7771_R2010a.zip     && unzip -q /tmp/spm12.zip -d /tmp     && mkdir -p /opt/spm12-r7771     && mv /tmp/spm12/* /opt/spm12-r7771/     && chmod -R 777 /opt/spm12-r7771     && rm -rf /tmp/spm*     && /opt/spm12-r7771/run_spm12.sh /opt/matlab-compiler-runtime-2010a/v713 quit
 ---> Using cache
 ---> a90fb3f44909
Step 6/22 : ENV OS="Linux"     PATH="/opt/freesurfer-7.1.1/bin:/opt/freesurfer-7.1.1/fsfast/bin:/opt/freesurfer-7.1.1/tktools:/opt/freesurfer-7.1.1/mni/bin:$PATH"     FREESURFER_HOME="/opt/freesurfer-7.1.1"     FREESURFER="/opt/freesurfer-7.1.1"     SUBJECTS_DIR="/opt/freesurfer-7.1.1/subjects"     LOCAL_DIR="/opt/freesurfer-7.1.1/local"     FSFAST_HOME="/opt/freesurfer-7.1.1/fsfast"     FMRI_ANALYSIS_DIR="/opt/freesurfer-7.1.1/fsfast"     FUNCTIONALS_DIR="/opt/freesurfer-7.1.1/sessions"     FS_OVERRIDE="0"     FIX_VERTEX_AREA=""     FSF_OUTPUT_FORMAT="nii.gz# mni env requirements"     MINC_BIN_DIR="/opt/freesurfer-7.1.1/mni/bin"     MINC_LIB_DIR="/opt/freesurfer-7.1.1/mni/lib"     MNI_DIR="/opt/freesurfer-7.1.1/mni"     MNI_DATAPATH="/opt/freesurfer-7.1.1/mni/data"     MNI_PERL5LIB="/opt/freesurfer-7.1.1/mni/share/perl5"     PERL5LIB="/opt/freesurfer-7.1.1/mni/share/perl5"
 ---> Using cache
 ---> 17d6408470e2
Step 7/22 : RUN apt-get update -qq     && apt-get install -y -q --no-install-recommends            bc            ca-certificates            curl            libgomp1            libxmu6            libxt6            perl            tcsh     && rm -rf /var/lib/apt/lists/*     && echo "Downloading FreeSurfer ..."     && mkdir -p /opt/freesurfer-7.1.1     && curl -fL https://surfer.nmr.mgh.harvard.edu/pub/dist/freesurfer/7.1.1/freesurfer-linux-centos6_x86_64-7.1.1.tar.gz     | tar -xz -C /opt/freesurfer-7.1.1 --owner root --group root --no-same-owner --strip-components 1          --exclude='average/mult-comp-cor'          --exclude='lib/cuda'          --exclude='lib/qt'          --exclude='subjects/V1_average'          --exclude='subjects/bert'          --exclude='subjects/cvs_avg35'          --exclude='subjects/cvs_avg35_inMNI152'          --exclude='subjects/fsaverage3'          --exclude='subjects/fsaverage4'          --exclude='subjects/fsaverage5'          --exclude='subjects/fsaverage6'          --exclude='subjects/fsaverage_sym'          --exclude='trctrain'
 ---> Using cache
 ---> 4fc67b51945a
Step 8/22 : ENV CONDA_DIR="/opt/miniconda-latest"     PATH="/opt/miniconda-latest/bin:$PATH"
 ---> Using cache
 ---> 46195bc158fb
Step 9/22 : RUN apt-get update -qq     && apt-get install -y -q --no-install-recommends            bzip2            ca-certificates            curl     && rm -rf /var/lib/apt/lists/*     && export PATH="/opt/miniconda-latest/bin:$PATH"     && echo "Downloading Miniconda installer ..."     && conda_installer="/tmp/miniconda.sh"     && curl -fsSL -o "$conda_installer" https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh     && bash "$conda_installer" -b -p /opt/miniconda-latest     && rm -f "$conda_installer"     && conda update -yq -nbase conda     && conda config --system --prepend channels conda-forge     && conda config --set channel_priority strict     && conda config --system --set auto_update_conda false     && conda config --system --set show_channel_urls true     && conda init bash     && conda install -y  --name base            "jupyter"     && sync && conda clean --all --yes && sync     && rm -rf ~/.cache/pip/*
 ---> Using cache
 ---> 75ebb00c1a9b
Step 10/22 : COPY ["(",       "'",       "e",       "n",       "v",       "i",       "r",       "o",       "n",       "m",       "e",       "n",       "t",       "-",       "s",       "h",       "o",       "r",       "t",       ".",       "y",       "m",       "l",       "'",       ",",       " ",       "'",       "/",       "t",       "m",       "p",       "/",       "'",       ")"]
failed to process "'": unexpected end of statement while looking for matching single-quote

[Remi-Gau]

FYI I confirm I can reproduce this

[kaczmarj]

the problem is in the COPY instruction. the path is being split into a tuple of strings. i am guessing the cause of this is using tuple("environment.yml /tmp/"). could be related to https://github.com/ReproNim/neurodocker/pull/492 (this commit https://github.com/ReproNim/neurodocker/pull/492/commits/714c5de00a7a98fce9ae7114610fa87b8609cb8d).

indeed the --copy command line is an OptionEatAll subclass, so using tuple in that implementation is likely causing this behavior. when this is fixed, we should also add a test of this use case. i could have sworn we had a test that copies files but perhaps not.

Step 10/22 : COPY ["(",       "'",       "e",       "n",       "v",       "i",       "r",       "o",       "n",       "m",       "e",       "n",       "t",       "-",       "s",       "h",       "o",       "r",       "t",       ".",       "y",       "m",       "l",       "'",       ",",       " ",       "'",       "/",       "t",       "m",       "p",       "/",       "'",       ")"]

[Remi-Gau]

was adding a regression test in a branch. want to open a PR to keep track of this?

[satra]

indeed it would be good to add a regression test. i had hoped it would have been caught by the current tests.

perhaps its truly time to refactor the optionseatall class a bit to be click 8 compliant.

[Remi-Gau]

added a very very simple test in #499

[kaczmarj]

perhaps its truly time to refactor the optionseatall class a bit to be click 8 compliant.

i can ask a stack overflow question to get the attention of click gurus :) one in particular has been especially helpful in the past.

[kaczmarj]

i am debugging this now and the error appears to be coming from https://github.com/ReproNim/neurodocker/blob/4f1cd1493005dc07b19d7b78de62191d5ced24cc/neurodocker/cli/generate.py#L326

i am working on fixing this

[kaczmarj]

it seems like value is a string at this point...

[kaczmarj]

@JohannesWiesner - can you please try with neurodocker 0.9.4? the issue should be fixed now. please reopen if it is not.

[JohannesWiesner]

@kaczmarj - Works like a charm now, much appreciated!