Singularity Issues with CMP v3.0.0-RC1

[smeisler]

Hello,

I built a singularity image of CMP3 from the latest Docker file. When I try to run the file I am met with the following error: COMMAND singularity run -e -B /om4 /om2/user/smeisler/connectomemapper.img --bids_dir /om/project/PARC/BIDS/data --output_dir /om/project/PARC/BIDS/derivatives --participant_label sub-ABCD1702 --anat_pipeline_config /om/project/PARC/BIDS/data/code/ref_anatomical_config.ini --func_pipeline_config /om/project/PARC/BIDS/data/code/ref_fMRI_config.ini --dwi_pipeline /om/project/PARC/BIDS/data/code/ref_diffusion_config.ini --fs_license /om4/group/gablab/dti_proc/license.txt

ERROR id: '\\': no such user id: '\\': no such user /.singularity.d/actions/run: 29: export: /om/project/PARC/BIDS/data: bad variable name

However, when I open the /.singularity.d/actions/run for the CMP container, I only see 23 lines, so I can't seem to find which script is being executed or how to fix it.

Do you know how I may fix this problem?

Thanks, Steven

[smeisler]

I realized I tried building the image with Singularity directly. I am now trying the recommended docker2singularity method and will close the issue if this solves the problem.

[smeisler]

With this method, I now have an error when trying to enter a singularity shell, let alone using singularity exec or run.

/.singularity.d/actions/exec: 28: export: 0: bad variable name

Any idea of how I may fix this?

[sebastientourbier]

Hi Steven,

By looking at the command you used to execute the singularity container, I can see you are missing mounting/binding your local directory (bids_dir and output_dir) into the container i.e.

singularity run --bind /om/project/PARC/BIDS/data:/bids_dir --bind /om/project/PARC/BIDS/derivatives:/output_dir \
/om2/user/smeisler/connectomemapper.img \
/bids_dir /output_dir participant --participant_label sub-ABCD1702 \
--anat_pipeline_config /bids_dir/code/ref_anatomical_config.ini \
--func_pipeline_config /bids_dir/code/ref_fMRI_config.ini \
--dwi_pipeline /bids_dir/code/ref_diffusion_config.ini \
--fs_license /bids_dir/code/license.txt

Note that all paths are then relative to the folder binded inside the container. Also I would recommend you to put the freesurfer license.txt into /om/project/PARC/BIDS/data/code folder.

Let me know if it works.

Best Sebastien

[smeisler]

Thanks for the reply Sebastien. Unfortunately, this does not seem to work, as I still get a similar error.

id: \\\\\smeisler\\\\: no such user
id: \\\\\smeisler\\\: no such user
/.singularity.d/actions/run: 29: export: /output_dir: bad variable name

I am using Singularity version 3.5.1-rc.1+103-g1a8ec23 as it is the most recent version on the HPC I use. Do you think a newer version wouldn't produce this error?

[sebastientourbier]

Singularity 3.5.1 should not be a problem as the cluster on which I am running CMP3 has the same version installed. It seems something related more the way the singularity image is built. Here is the way I exactly built my image:

#!/bin/sh
mkdir -p /home/localadmin/Softwares/singularity/images
docker run -v /var/run/docker.sock:/var/run/docker.sock -v /home/localadmin/Softwares/singularity/images/:/output \
--privileged -t --rm singularityware/docker2singularity \
--name cmp-v3.0.0-RC1.simg sebastientourbier/connectomemapper-bidsapp:v3.0.0-RC1

[sebastientourbier]

Wanted to add: It seems someone had a similar error here when building with Singularity directly that was not occuring when using docker2singularity

[smeisler]

I will try rebuilding the image now and will update when I know more, thanks!

[smeisler]

Unfortunately, I still have an issue. I followed your code, besides changing the output directory of the container.

TEST singularity shell ~/Downloads/test/cmp-v3.0.0-RC1.simg ERROR /.singularity.d/actions/shell: 28: export: 0: bad variable name

I'll get the same error regardless of the singularity command (shell, exec, run, etc), as long as I reference that container. This was true both on my local machine where I generated the container and the HPC where our MR data are stored.

[smeisler]

I built the singularity image as a sandbox with the singularity build --sandbox option. Here I was able to go into the singularity.d folder and try to debug there.

The problem appears to be in the file /singularity.d/env/10-docker2singularity.sh (around lines 28-29)

export content=${content:-"#! /bin/bash\\\\necho User: \\$(id -un \\$USER) && echo Group: \\$(id -gn \\$USER) && . \\\"/opt/fsl/etc/fslconf/fsl.sh\\\" && . activate \\\"py37cmp-core\\\" && xvfb-run -s \\\"-screen 0 900x900x24 -ac +extension GLX -noreset\\\" -a python /app/connectomemapper3/run.py \\$@"}
export content_cov=${content_cov:-"#! /bin/bash\\\\necho User: \\$(id -un \\$USER) && echo Group: \\$(id -gn \\$USER) && . \\\"/opt/fsl/etc/fslconf/fsl.sh\\\" && . activate \\\"py37cmp-core\\\" && xvfb-run -s \\\"-screen 0 900x900x24 -ac +extension GLX -noreset\\\" -a coverage run --source=cmp,cmtklib --omit=*/bidsappmanager/*,*/viz/* /app/connectomemapper3/run.py \\$@ |& tee /bids_dir/code/log.txt && coverage html -d /bids_dir/code/coverage_html && coverage xml -o /bids_dir/code/coverage.xml && coverage json -o /bids_dir/code/coverage.json"}

Following the response in the similar issue you linked, I changed those lines to the following:

content=${content:-"#! /bin/bash\\\\necho User: \\$(id -un \\$USER) && echo Group: \\$(id -gn \\$USER) && . \\\"/opt/fsl/etc/fslconf/fsl.sh\\\" && . activate \\\"py37cmp-core\\\" && xvfb-run -s \\\"-screen 0 900x900x24 -ac +extension GLX -noreset\\\" -a python /app/connectomemapper3/run.py \\$@"}
export content
content_cov=${content_cov:-"#! /bin/bash\\\\necho User: \\$(id -un \\$USER) && echo Group: \\$(id -gn \\$USER) && . \\\"/opt/fsl/etc/fslconf/fsl.sh\\\" && . activate \\\"py37cmp-core\\\" && xvfb-run -s \\\"-screen 0 900x900x24 -ac +extension GLX -noreset\\\" -a coverage run --source=cmp,cmtklib --omit=*/bidsappmanager/*,*/viz/* /app/connectomemapper3/run.py \\$@ |& tee /bids_dir/code/log.txt && coverage html -d /bids_dir/code/coverage_html && coverage xml -o /bids_dir/code/coverage.xml && coverage json -o /bids_dir/code/coverage.json"}
export 

That is, I made the declaration and export statements in separate lines. To be safe, I did with all of the statements in this file, and made sure there were no other statements like this in the /singularity.d/env/ folder. I also removed the shebang (#!/bin/sh) from the top.

Now it looks like it is beginning to run. I can update again when the pipeline finishes (hopefully) or crashes. Interestingly, when I run it in terminal, I get a nipype log detailing what is happening, but when I submit a job with slurm/sbatch, the slurm-out file does not update.

If this works, maybe I can send you the copy of the .sh file or the whole singularity image, in case it may help others!

[smeisler]

I don't know enough about the production of containers, let alone the transition with docker2singularity, but hopefully this is something that can be fixed on your end (that is, it is not a problem with docker2singularity)

[sebastientourbier]

Thanks @smeisler for having investigating this and great to hear you managed to make it run! Yes it sounds like to fix it on my end, I should review the export statements for the creation of the entrypoint scripts in the Dockerfile used to build the Docker i.e. https://github.com/connectomicslab/connectomemapper3/blob/69d50b82532af25ee544a43cdcf3c125ea9cd91f/Dockerfile#L57-L86 This should be then reflected in the Singularity image that will be built from. I will investigate a way to fix this at this level that I expect to include in the upcoming next release candidate. Once I find a correct way to create the content of these files, I will also update the Docker image for RC1 on Dockerhub.

[smeisler]

Great, and as an update from my end, it looked like everything ran to completion without a problem!