Example runs with `--use-singularity` fail on R-related tasks

[jpintar]

When I tried running the examples with --use-singularity, the runs would fail with Error in rule mtxs_to_sce_txs and Error in rule mtxs_to_sce_genes. The R log reveals:

Error in library(HDF5Array) : there is no package called ‘HDF5Array’
Execution halted

I'm also having some trouble finding a version of snakemake where runs with --use-conda don't get stuck on solving the kallisto-bustools.yaml environment because of strict channell priority. I know the note says >=7.8.4, <8.0 should work with a warning, but that doesn't seem to be the case. I'm still trying out more versions...

[mfansler]

Thanks for reporting this!

Indeed, the Docker image hasn't been rebuilt in quite some time. I will try to get it updated this week.

[mfansler]

On the channel priority: it must be set to flexible. The note points out that v7.8.0-7.8.3 are incompatible since they raise an Error on flexible. v7.8.4+ will work, but with a Warning that you can safely ignore.

If you think that text could be clearer, feel free to suggest an alternative phrasing. Looking over it, it does point out that a Warning is to be expected.

Personally, I think the flexible/strict thing is the wrong solution to the problem. If Snakemake wants to improve reproducibility, it should 1) support conda-lock lockfiles and 2) use an internal copy of Micromamba (or maybe pixi).

[jpintar]

No, I think the note is perfectly clear, and I totally agree with both your points about Snakemake. It's just that I when I made a fresh environment with snakemake>=7.8.4,<8.0, which resolved to v7.32.4, I still got a LibMambaUnsatisfiableError instead of a warning, so maybe there was another regression in Snakemake after 7.8.4 (or maybe the problem is with the latest mamba). I need to try with some other version(s). Which version do you test with?

[mfansler]

No, I think the note is perfectly clear, and I totally agree with both your points about Snakemake. It's just that I when I made a fresh environment with snakemake>=7.8.4,<8.0, which resolved to v7.32.4, I still got a LibMambaUnsatisfiableError instead of a warning, so maybe there was another regression in Snakemake after 7.8.4 (or maybe the problem is with the latest mamba). I need to try with some other version(s). Which version do you test with?

I just tested my Linux GitHub workflow and it didn't have any issues. I'm trying the macOS one now. What platform are you on?

You can check details of the runs on the scUTRquant-demo repository. On Linux I see:

base environment

• Conda v24.3.0
• Mamba v1.5.8
• Python v3.10.14

snakemake environment

• Python v3.11.9
• Snakemake v7.32.4

I'll see what the macOS gives.

Note that I also don't see an issue with certificates for example downloading (i.e., #76).

[mfansler]

My macOS runs were also fine after switching the workflow to amd64 runners¹. They yielded similar versions as the Linux runs.

Not sure I can address the Mamba solving issue further without more details.

---

_{[1]: Technically, the pipeline can run on Apple Silicon in emulation mode (e.g., CONDA_SUBDIR=osx-64), but the GitHub runners don't support it. However, end-users should be fine.}

[jpintar]

I just re-ran the examples with --use-conda with a freshly cloned scUTRquant and fresh Snakemake v7.32.4 environment (this is all on CentOS 7), and didn't get stuck on the env-creation this time! I'm sorry to have wasted your time - not sure what was misconfigured on my end previously...

[mfansler]

Okay, thanks for checking again. Yeah, CentOS 7 should be smooth sailing.

I'll let you know when I get to the Docker image.

[mfansler]

I pushed a new Docker image. You can either pull the dev branch or manually change the tag in Snakemake file to 0.4.0, i.e.,

container: "docker://mfansler/scutr-quant:0.4.0"

I don't have extensive testing for this component, but I successfully ran a couple examples without issue on our HPC. Please let me know if there is any issue using it.

[jpintar]

Just tested it - works like a charm! Thank you for fixing this so quickly!