search

maxplanck-ie / snakepipes

Customizable workflows based on snakemake and python for the analysis of NGS data
http://snakepipes.readthedocs.io
383 stars 85 forks source link

conda-env create is using /tmp #1049

Open katsikora opened 1 month ago

katsikora commented 1 month ago

snakePipes 3.0.0 - snakemake is calling conda create env and pointing it to /tmp rather than to condaEnvDir defined in the defaults.yaml

katsikora commented 1 month ago

snakePipes config function needs to be modified such that it modifies conda-prefix in the slurm profile file rather than in defaults.yaml

WardDeb commented 1 month ago

As of 3.0.0 condaEnvDir should not be set in defaults.yaml anymore, but should be set in the appropriate profile (wherever that lives). defaults.yaml points to the local profile by default, and the conda-prefix there is set to /tmp. This is needed for the gh actions to work, and is a reasonable default (I think).

As for the snakePipes config function, this would work, but do we still need a function that makes changes to these profiles to begin with ?

katsikora commented 1 month ago

Hi Ward,

from my point of view, the user should be able to edit the config shipped with snakePipes. This is what the snakePipes config function is for. I'd just update it so that all the necessary variables can be edited in the snakemake profile without the user having to look for it in some custom location. Or do you have another solution for this?

The documentation would also have to be updated, e.g. in the quick start section.

Best, Katarzyna

katsikora commented 1 month ago

Btw, snakePipes config has a setting that works for gha. To be honest, I don't like /tmp as default, I'd prefer at least /scratch/local .

WardDeb commented 1 month ago

from my point of view, the user should be able to edit the config shipped with snakePipes. This is what the snakePipes config function is for. I'd just update it so that all the necessary variables can be edited in the snakemake profile without the user having to look for it in some custom location. Or do you have another solution for this?

Agreed for defaults.yaml, though note that profiles are allowed to live outside of the snakePipes directory (in a specific directory, or in the default locations that snakemake expects them to be, /etc/xdg/snakemake/ & ~/.config/snakemake/), and I think it's not so nice to change these files. If you insist of keeping that perhaps a comment line (# changed by snakePipes config) or something similar would be appropriate..

katsikora commented 1 month ago

Ok to comment line after changing configuration. I'd only change the conda-prefix in the snakemake profile file, if it was provided by the user via snakePipes config --condaEnvDir. I'm assuming that they are using the config that is shipped with snakePipes and has /tmp as conda-prefix. If condaEnvDir is not provided by the user, because they have everything they need in their private snakemake profile, then I wouldn't change it at all. Would you agree?

WardDeb commented 1 month ago

Makes sense yea, thanks ! 👍🏼