search

dib-lab / elvers

(formerly eelpond) an automated RNA-Seq workflow system
https://dib-lab.github.io/elvers/
Other
28 stars 3 forks source link

stuck downloading packages when testing #160

Open AntonioBaeza opened 3 years ago

AntonioBaeza commented 3 years ago

The installation went well. But, no error message. Just stuck when testing it.

(base) [ant@hillary ~]$ git clone https://github.com/dib-lab/elvers.git Initialized empty Git repository in /home/ant/elvers/.git/ remote: Enumerating objects: 734, done. remote: Counting objects: 100% (734/734), done. remote: Compressing objects: 100% (352/352), done. remote: Total 7127 (delta 413), reused 526 (delta 254), pack-reused 6393 Receiving objects: 100% (7127/7127), 107.21 MiB | 9.78 MiB/s, done. Resolving deltas: 100% (4591/4591), done. (base) [ant@hillary ~]$ cd elvers (base) [ant@hillary elvers]$ conda env create --file environment.yml -n elvers-env Collecting package metadata (repodata.json): done Solving environment: done

==> WARNING: A newer version of conda exists. <== current version: 4.8.4 latest version: 4.8.5

Please update conda by running

$ conda update -n base -c defaults conda

Preparing transaction: done Verifying transaction: done Executing transaction: done #

To activate this environment, use

#

$ conda activate elvers-env

#

To deactivate an active environment, use

#

$ conda deactivate

(base) [ant@hillary elvers]$ conda activate elvers-env (elvers-env) [ant@hillary elvers]$ pip install -e '.' Obtaining file:///home/ant/elvers Requirement already satisfied: pyyaml in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from elvers==0.1) (5.3.1) Requirement already satisfied: snakemake in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from elvers==0.1) (5.14.0) Requirement already satisfied: pytest in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from elvers==0.1) (6.0.2) Requirement already satisfied: numpy in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from elvers==0.1) (1.19.1) Requirement already satisfied: pandas in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from elvers==0.1) (1.1.1) Collecting graphviz Using cached graphviz-0.14.1-py2.py3-none-any.whl (18 kB) Requirement already satisfied: networkx in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from elvers==0.1) (2.5) Requirement already satisfied: psutil in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from elvers==0.1) (5.7.2) Requirement already satisfied: ftputil in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from elvers==0.1) (3.2) Requirement already satisfied: appdirs in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from snakemake->elvers==0.1) (1.4.4) Requirement already satisfied: requests in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from snakemake->elvers==0.1) (2.24.0) Requirement already satisfied: nbformat in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from snakemake->elvers==0.1) (5.0.7) Requirement already satisfied: configargparse in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from snakemake->elvers==0.1) (1.2.3) Requirement already satisfied: gitpython in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from snakemake->elvers==0.1) (3.1.8) Requirement already satisfied: docutils in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from snakemake->elvers==0.1) (0.15.2) Requirement already satisfied: ratelimiter in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from snakemake->elvers==0.1) (1.2.0) Requirement already satisfied: wrapt in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from snakemake->elvers==0.1) (1.12.1) Requirement already satisfied: jsonschema in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from snakemake->elvers==0.1) (3.2.0) Requirement already satisfied: toposort in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from snakemake->elvers==0.1) (1.5) Requirement already satisfied: datrie in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from snakemake->elvers==0.1) (0.8.2) Requirement already satisfied: attrs>=17.4.0 in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from pytest->elvers==0.1) (20.2.0) Requirement already satisfied: iniconfig in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from pytest->elvers==0.1) (0.0.0) Requirement already satisfied: more-itertools>=4.0.0 in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from pytest->elvers==0.1) (8.5.0) Requirement already satisfied: packaging in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from pytest->elvers==0.1) (20.4) Requirement already satisfied: pluggy<1.0,>=0.12 in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from pytest->elvers==0.1) (0.13.1) Requirement already satisfied: py>=1.8.2 in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from pytest->elvers==0.1) (1.9.0) Requirement already satisfied: toml in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from pytest->elvers==0.1) (0.10.1) Requirement already satisfied: importlib-metadata>=0.12 in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from pytest->elvers==0.1) (1.7.0) Requirement already satisfied: python-dateutil>=2.7.3 in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from pandas->elvers==0.1) (2.8.1) Requirement already satisfied: pytz>=2017.2 in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from pandas->elvers==0.1) (2020.1) Requirement already satisfied: decorator>=4.3.0 in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from networkx->elvers==0.1) (4.4.2) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from requests->snakemake->elvers==0.1) (1.25.10) Requirement already satisfied: chardet<4,>=3.0.2 in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from requests->snakemake->elvers==0.1) (3.0.4) Requirement already satisfied: certifi>=2017.4.17 in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from requests->snakemake->elvers==0.1) (2020.6.20) Requirement already satisfied: idna<3,>=2.5 in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from requests->snakemake->elvers==0.1) (2.10) Requirement already satisfied: traitlets>=4.1 in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from nbformat->snakemake->elvers==0.1) (4.3.3) Requirement already satisfied: ipython-genutils in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from nbformat->snakemake->elvers==0.1) (0.2.0) Requirement already satisfied: jupyter-core in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from nbformat->snakemake->elvers==0.1) (4.6.3) Requirement already satisfied: gitdb<5,>=4.0.1 in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from gitpython->snakemake->elvers==0.1) (4.0.5) Requirement already satisfied: setuptools in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from jsonschema->snakemake->elvers==0.1) (49.6.0.post20200814) Requirement already satisfied: pyrsistent>=0.14.0 in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from jsonschema->snakemake->elvers==0.1) (0.17.3) Requirement already satisfied: six>=1.11.0 in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from jsonschema->snakemake->elvers==0.1) (1.15.0) Requirement already satisfied: pyparsing>=2.0.2 in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from packaging->pytest->elvers==0.1) (2.4.7) Requirement already satisfied: zipp>=0.5 in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from importlib-metadata>=0.12->pytest->elvers==0.1) (3.1.0) Requirement already satisfied: smmap<4,>=3.0.1 in /home/ant/anaconda3/envs/elvers-env/lib/python3.6/site-packages (from gitdb<5,>=4.0.1->gitpython->snakemake->elvers==0.1) (3.0.4) Installing collected packages: graphviz, elvers Running setup.py develop for elvers Successfully installed elvers graphviz-0.14.1 (elvers-env) [ant@hillary elvers]$ elvers examples/nema.yaml default

checking for required files:

    Found configfile at /home/ant/elvers/examples/nema.yaml
    Added default parameters from rule-specific params files.
    Writing full params to /home/ant/elvers/examples/.ep_nema.yaml

details! snakefile: /home/ant/elvers/elvers/Snakefile config: /home/ant/elvers/examples/nema.yaml params: /home/ant/elvers/examples/.ep_nema.yaml targets: ['default', 'get_data'] output: '/home/ant/elvers/nema_out' report: '/home/ant/elvers/nema_out/logs/report.html'

Building DAG of jobs... Creating conda environment elvers/rules/dammit/environment.yml... Downloading and installing remote packages.

it has been at this stage for >24h

bluegenes commented 3 years ago

Hi @AntonioBaeza! The dammit installation seems to be broken. I'm working on a fix and will let you know here when it's in place.

AntonioBaeza commented 3 years ago

Thanks bluegenes! Looking forward to start using it!

AntonioBaeza commented 3 years ago

Any news on fixing this issue. Just anxious/looking forward to use it!

bluegenes commented 3 years ago

Hi @AntonioBaeza! Sorry for the delay- we are very close to releasing a revamped version of dammit that will circumvent the long conda install times. In the mean time, you can run elvers with a workflow that does not include dammit. For example, you can run just the assembly workflow (trimming + Trinity assembly) like so:

elvers examples/nema.yaml assemble

or the assembly + salmon quantification workflows together:

elvers examples/nema.yaml assemble quantify

When you run your own data, you'll want to set a higher max_memory parameter for Trinity in your yaml config file. The amount will depend on what memory you have available, of course.

You can set the trinity max_memory in the yaml config by including the following:

trinity:
  max_memory: 100G

For a large amount of RNAseq data, I'll often run trinity with 20 threads and 100G of memory, but you can run it with much less, of course - check out the trinity documentation. You can run elvers with additional threads use -t, like so: elvers examples/nema.yaml assemble quantify -t 20

more info on the docs: https://dib-lab.github.io/elvers/assemble/

AntonioBaeza commented 3 years ago

Thanks Tessa!

On Wed, Oct 14, 2020 at 12:56 PM Tessa Pierce notifications@github.com wrote:

Hi @AntonioBaeza https://github.com/AntonioBaeza! Sorry for the delay- we are very close to releasing a revamped version of dammit that will circumvent the long conda install times. In the mean time, you can run elvers with a workflow that does not include dammit. For example, you can run just the assembly workflow (trimming + Trinity assembly) like so:

elvers examples/nema.yaml assemble

or the assembly + salmon quantification workflows together:

elvers examples/nema.yaml assemble quantify

When you run your own data, you'll want to set a higher max_memory parameter for Trinity in your yaml config file. The amount will depend on what memory you have available, of course.

You can set the trinity max_memory in the yaml config by including the following:

trinity: max_memory: 100G

For a large amount of RNAseq data, I'll often run trinity with 20 threads and 100G of memory, but you can run it with much less, of course - check out the trinity documentation. You can run elvers with additional threads use -t, like so: elvers examples/nema.yaml assemble quantify -t 20

more info on the docs: https://dib-lab.github.io/elvers/assemble/

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dib-lab/elvers/issues/160#issuecomment-708531068, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALIJRA2FMHFN3FHV4QK2ADTSKXJ43ANCNFSM4R37KARA .

-- J. Antonio Baeza

Associate Professor Department of Biological Sciences, Clemson University South Carolina, USA & Research Associate Smithsonian Marine Station at Fort Pierce, Florida, USA & Adjunct Faculty Universidad Catolica del Norte, Coquimbo, Chile

Email: baeza.antonio@google.com & jbaezam@clemson.edu Website: http://baezaantonio.wix.com/baezalabclemson Website (CI-team): https://baezaantonio.wixsite.com/clemsonmitogenomics Website ResearchGate: https://www.researchgate.net/profile/J_Baeza/ Website SemanticScholar: https://www.semanticscholar.org/author/Juan-Antonio-Baeza/144723920