search

snakemake / snakemake-executor-plugin-slurm

A Snakemake executor plugin for submitting jobs to a SLURM cluster
MIT License
18 stars 19 forks source link

Installing with mamba tries to downgrades to snakemake 7 #96

Closed conchoecia closed 4 months ago

conchoecia commented 6 months ago

Hey there! I'm trying to migrate my SLURM workflow to snakemake 8 and I think this is the new standard way.

There are only pip instructions on the instruction page, but it also mentions using mamba. So, I tried to install with mamba.

The problem

When I install with conda, it tries to downgrade my snakemake to v7. Isn't it supposed to stay 8?

conda install snakemake-executor-plugin-slurm
Retrieving notices: ...working... done
Channels:
 - defaults
 - conda-forge
 - bioconda
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /lisc/user/schultz/miniconda3/envs/snakemake8

  added / updated specs:
    - snakemake-executor-plugin-slurm

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    snakemake-executor-plugin-slurm-0.5.1|     pyhdfd78af_0          16 KB  bioconda
    snakemake-executor-plugin-slurm-jobstep-0.2.1|     pyhdfd78af_0          12 KB  bioconda
    ------------------------------------------------------------
                                           Total:          28 KB

The following NEW packages will be INSTALLED:

  snakemake-executo~ bioconda/noarch::snakemake-executor-plugin-slurm-0.5.1-pyhdfd78af_0 
  snakemake-executo~ bioconda/noarch::snakemake-executor-plugin-slurm-jobstep-0.2.1-pyhdfd78af_0 

The following packages will be DOWNGRADED:

  snakemake                               8.11.3-hdfd78af_0 --> 7.32.4-hdfd78af_1 
  snakemake-minimal                     8.11.3-pyhdfd78af_0 --> 7.32.4-pyhdfd78af_1 

Proceed ([y]/n)?

When I try to install with mamba, it says that the package does not exist.

mamba install snakemake-executor-plugin-slurm

Looking for: ['snakemake-executor-plugin-slurm']

warning  libmamba Cache file "/lisc/user/schultz/miniconda3/pkgs/cache/47929eba.json" was modified by another program
warning  libmamba Could not parse mod/etag header
pkgs/main/linux-64 (check zst)                     Checked  0.0s
warning  libmamba Cache file "/lisc/user/schultz/miniconda3/pkgs/cache/3e39a7aa.json" was modified by another program
warning  libmamba Could not parse mod/etag header
pkgs/main/noarch (check zst)                       Checked  0.0s
warning  libmamba Cache file "/lisc/user/schultz/miniconda3/pkgs/cache/2ce54b42.json" was modified by another program
warning  libmamba Could not parse mod/etag header
pkgs/r/linux-64 (check zst)                        Checked  0.0s
warning  libmamba Cache file "/lisc/user/schultz/miniconda3/pkgs/cache/4ea078d6.json" was modified by another program
warning  libmamba Could not parse mod/etag header
pkgs/r/noarch (check zst)                           Checked  0.0s
pkgs/r/noarch                                        2.1MB @  17.6MB/s  0.1s
pkgs/main/noarch                                   714.0kB @   2.9MB/s  0.3s
pkgs/r/linux-64                                      1.6MB @   6.4MB/s  0.3s
pkgs/main/linux-64                                   6.1MB @  19.8MB/s  0.4s

Pinned packages:
  - python 3.11.*

Could not solve for environment specs
The following package could not be installed
└─ snakemake-executor-plugin-slurm does not exist (perhaps a typo or a missing channel).

Ideal solution

Ideally there would be install instructions for mamba or conda in addition to those with pip. Also, if it is supposed to work with conda install, ideally the version requirement would be bumped to snakemake v8 (since I believe that plugins are only supported on v8?).

Thank you!

cmeesters commented 6 months ago

Hi,

that's really weird. Perhaps your environment is somehow broken? I suspect this, because the dependency management is part of conda, specified in this repo, and up to now did not cause any such issue. I am not sure how to debug your specific situation.

You are right, the pip instructions are somewhat misleading: Snakemake itself has too many dependencies to be installed with pip alone. Hence, the note in the main docs. Yet, installing plugins into a conda environment via pip ought to work, too. A downgrade to v7 of Snakemake should not happen either way.

Ideally there would be install instructions for mamba or conda in addition to those with pip.

Ah, that is something which maybe should be phrased better in the docs. Actually, the pip install instruction does not stem from this repo. It is added automatically for every plugin similarly.

conchoecia commented 5 months ago

Thanks for your thoughts! Not sure what I could do to help debug except paste all my packages. What I did was to follow the mamba instructions on the snakemake page, and I have an environment just for snakemake 8. Then, I had to install some additional packages required for my analyses, but it was only a few aside from the standards (standards being: numpy, matplotlib, pandas, et cetera).

Do you think spinning up a fresh environment, installing snakemake 8, and trying to install the slurm executor again might help to diagnose what is going on?

cmeesters commented 5 months ago

Might be worth a trial.

Installing Snakemake will also bring numpy and pandas, yet not the full scipy stack and no matplotlib. You could run Snakemake with --software-deployment-method conda --conda-frontend mamba to deploy the dependencies of your workflow on the fly (--software-deployment-method has a shorthand --sdm), to better separate environments. However, the stack you mentioned should not give any issues at all. Hence, it is weird.