johnne
commented
3 years ago Hi @lisalenorelowe and sorry for the late reply.
You're right that the environment.yml file does not contain all the packages required to run all steps of the workflow. That's why the first steps taken by the workflow is to install additional packages. The rationale is that a user should need to install as few packages as possible, basically only the ones necessary to get the results of interest. Also, the explicit versions set for packages often cause dependency conflicts with each other just as you say. That's another reason we try to keep the environments small and specific.
If I understand you correctly, users on your system cannot install conda nor create environments with conda? Does that also apply to users home folder? On the HPC that we typically use users can install conda to their home folder (on the login node), then use conda to install environments in project locations they have access to. That's sufficient to use nbis-meta, but maybe your system has more restrictions.
If you as an admin can run conda then you are right that you should be able to break up the install and run steps. You could try to run with a config file where everything is set to True, then run:
snakemake --use-conda --conda-create-envs-only --conda-prefix <path>where you exchange <path> with some location on disk where conda environments may be stored. Once all conda packages have been created, users should be able to run the workflow by also including --use-conda --conda-prefix <path> in their snakemake call.
Let me know if that helps and just ask if you have more questions.
lisalenorelowe
Dear John Sundh and Co.,
I am trying to get nbis-meta up and running on our HPC, initially from a request from Ryan P., but I would like to set it up so anyone can use it.
I created a module (it is a conda environment, but it is set so people can do 'module load') by doing: conda env create --prefix (/path/to/env) -f environment.yml I also 'conda installed' some other packages to that environment, namely Jupyter and others to try the ipyb that were included in the git repo.
When I tried to run things (and I am not bioinformatics and generally have no idea what this does...also I have never used snakemake), I realized that during execution, it downloads a bunch more Conda packages. Our users cannot do that from compute nodes. Can I install a comprehensive set of Conda packages so that a user of nbis-meta would never have to do that? Or do you need to install custom environments each time, because the environments break because of the different versions?
If a user would have to install different stuff all the time, then should I direct them to git clone, do the initial 'conda env create' from environment.yml, etc., themselves, or is it still worth trying to make that available for people to module load (works the same as conda activate), then only conda add the extra stuff to their run directories?
Also, I understand you can break up the steps such that a user can: Step 1) Do all the 'conda env create' and anything else that has to access the internet, including wget or ftp Step 2) Run it. Can you send or point to some instructions for that?
Thanks for your help, Lisa