Singularity version error/check in process_sample.local.sh

[ashishjain1988]

Hi @williamrowell ,

I have the latest singularity version (apptainer version 1.1.3-1.el7) but when I run the process_sample.local.sh I got the singularity version error. "WorkflowError: Minimum singularity version is 2.4.1."

Can you please help me with this?

Regards, Ashish Jain

[williamrowell]

Can you get the output of:

singularity --version

Also, this must be installed by root in order to run correctly.

[ashishjain1988]

The output is apptainer version 1.1.3-1.el7

[williamrowell]

Hi @ashishjain1988,

So, here's what's going on (thanks to @juniper-lake for digging into this):

• The Singularity project renamed itself apptainer. They create a symlink from singularity to apptainer for backwards compatibility, but when you run singularity --version, it reports itself as apptainer. They also reset the version number to 1.0 with this transition.
• Snakemake checks the output of snakemake version and expects to see a version number higher than 2.4.1. This has been patched in Snakemake 7.8.3, but, as you know, this repo is pinned to Snakemake 6.15.3.
• This repo is pinned to an older version of Snakemake because several versions back, Snakemake started enforcing strict channel priority on conda environments, which breaks a lot of our conda environments. These are solvable problems (#140), but it requires time and resources to test. Rather than playing dependency whack-a-mole, we're trying to devote those resources to replacing the snakemake version of this workflow with a WDL version using Docker/Singularity images to containerize all dependencies.
• The safest thing for us to do here is to pin the singularity version dependency to the last version before the transition to apptainer, which appears to be 3.8.7. Given that we plan on moving away from this version of the workflow soon, I don't think we have the bandwidth to support newer versions of dependencies. I know that's not a particularly helpful answer for you.
• One potential solution is to try to fix the channel priorities (#140) in rules/envs/*.yaml. Once these have been fixed to work with conda config --set channel_priority true, then we should be able to use newer versions of Snakemake. It may be as easy as moving conda-forge to the top of the list in every yaml where it appears. It may be more complicated than that. Also, fixing this problem and unpinning snakemake may uncover new issues that we'd then have to solve.

[McClatcheyM]

Hi @williamrowell @juniper-lake I've been having a similar, related issue with this and was wondering if you could offer any suggestions FYI, this is for the 3rd of 3 trios that I've recently run using a Slurm HPC. The previous two trios have run fairly successfully. I've been running singularity 3.8.5 as this was a stable and available version and I had wanted to replicate the methodology used for the other families.

Unfortunately the singularity version issue has arisen before I could run this family through end-to-end. It managed to pass through process_smrtcell ok but is causing major problems with process_sample on the first sample of three in the trio.

I've been running singularity 3.8.5 as this was a stable and available version and I had wanted to replicate the methodology used for the other families. Other versions available to me as a University HPC user are: singularity-ce/3.10.2 singularity/apptainer/1.1.2

The main issue appears to arise with the ### deepvariant_make_examples jobs, where these will run across the available nodes in a small batch (1-6) but the workflow will terminate upon completion of the batch without continuing with the raminaing jobs (see screenshot of a failed single job cluster log). Unfortunately this is defeating the point of the workflow currently.

Based on the previous post, I've tried re-setting my conda config .condarc channel priority to strict and have de-bugged the relevant conda environments in rules/envs/*.yaml based on error outputs from running process_sample. The output is still the same - i.e. version control issues with singularity.

Is there another way that I could amend the workflow to account for this version control issue as a short-term fix to complete my priority run?! Any suggestions would be much appreciated. Kind regards, Martin

[juniper-lake]

This error happens when snakemake runs the command singularity --version because singularity cannot identify the user. This is therefore an issue that you should discuss with your HPC support. It might be a version issue, but might also be a user permissions problem. Snakemake might be trying to use a singularity installation that wasn't installed by root. Or maybe it's a node-specific issue.

The issue arises only with deepvariant because that's the first step that uses singularity. Conda shouldn't have an impact on it. The singularity version will not affect the workflow in terms of reproducibility, so just use a version that works and make sure it's first place in your PATH.

[williamrowell]

My response above is kind of confusing, so I'll try to clarify a little.

• The problem that this thread describes is an issue with how singularity --version is reported for versions of singularity after it was renamed apptainer. Snakemake calls singularity version internally to make sure that features it expects will be supported.
• This specific problem is addressed in the newest versions of snakemake (~7.8.3)
• The workflows in this repo are currently pinned to an older version of snakemake because of a separate issue related to how snakemake handles conda environments. Because of pinning to the older snakemake version, this workflow is not compatible with the newest versions of singularity.
• I don't think that the error that @McClatcheyM posted has anything to do with either of these. Like @juniper-lake suggests, I think this has to do with singularity permissions. Did you install singularity 3.8.5 yourself, or was this installed by the system administrator? It needs to be installed by a root user to have the correct permissions. What happens if you make sure that singularity-ce/3.10.2 is first in your path?