search

dib-lab / elvers

(formerly eelpond) an automated RNA-Seq workflow system
https://dib-lab.github.io/elvers/
Other
28 stars 3 forks source link

Trinity failing, empty log file #85

Open mandyfrazier opened 5 years ago

mandyfrazier commented 5 years ago

Hi @bluegenes! I tried running eelpond with four samples (~142 million reads total) and it is failing somewhere in Trinity, but I can't actually see the error. I'm thinking that I have a memory issue, but do you have any other suggestions apart from increasing the instance size? Right now the Trinity log file is empty: cat burbot_assemble_out__experiment1/logs/trinity/trinity.log

This is the error I have access to because I was running it in screen: 

ellyfish.kmers.fa --run_inchworm -K 25 -L 25 --monitor 1   --DS  --num_threads 1  --PARALLEL_IWORM  > /home/ubuntu/data/burbot_assemble_out__experiment1/assembly/trinity_out_dir/inchworm.K25.L25.DS.fa.tmp 2>tmp.15615.1550094136.stderr died with ret 34304  at /home/ubuntu/eelpond/.snakemake/conda/0d7b2c94/opt/trinity-2.8.4/PerlLib/Pipeliner.pm line 186.
        Pipeliner::run(Pipeliner=HASH(0x5629871c63f8)) called at /home/ubuntu/eelpond/.snakemake/conda/0d7b2c94/bin/Trinity line 2517
        eval {...} called at /home/ubuntu/eelpond/.snakemake/conda/0d7b2c94/bin/Trinity line 2512
        main::run_inchworm("/home/ubuntu/data/burbot_assemble_out__experiment1/assembly/t"..., "/home/ubuntu/data/burbot_assemble_out__experiment1/assembly/t"..., undef, "") called at /home/ubuntu/eelpond/.snakemake/conda/0d7b2c94/bin/Trinity line 1650
        main::run_Trinity() called at /home/ubuntu/eelpond/.snakemake/conda/0d7b2c94/bin/Trinity line 1345
        eval {...} called at /home/ubuntu/eelpond/.snakemake/conda/0d7b2c94/bin/Trinity line 1344

If it indicates bad_alloc(), then Inchworm ran out of memory.  You'll need to either reduce the size of your data set or run Trinity on a server with more memory available.

** The inchworm process failed.Traceback (most recent call last):
  File "/home/ubuntu/eelpond/.snakemake/scripts/tmpah23sgvr.trinity-wrapper.py", line 49, in <module>
    shell("Trinity {input_cmd} --CPU {snakemake.threads} --max_memory {max_memory} --seqType {seqtype} --output {outdir} {snakemake.params.extra} {log}")
  File "/home/ubuntu/miniconda3/envs/eelpond/lib/python3.6/site-packages/snakemake/shell.py", line 142, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command ' set -euo pipefail;  Trinity  --left /home/ubuntu/data/burbot_assemble_out__experiment1/preprocess/Ma07P_L_1.khmer.fq.gz,/home/ubuntu/data/burbot_assemble_out__experiment1/preprocess/Ma10P_L_1.khmer.fq.gz,/home/ubuntu/data/burbot_assemble_out__experiment1/preprocess/Ma11P_L_1.khmer.fq.gz,/home/ubuntu/data/burbot_assemble_out__experiment1/preprocess/Ma12P_L_1.khmer.fq.gz  --right /home/ubuntu/data/burbot_assemble_out__experiment1/preprocess/Ma07P_L_2.khmer.fq.gz,/home/ubuntu/data/burbot_assemble_out__experiment1/preprocess/Ma10P_L_2.khmer.fq.gz,/home/ubuntu/data/burbot_assemble_out__experiment1/preprocess/Ma11P_L_2.khmer.fq.gz,/home/ubuntu/data/burbot_assemble_out__experiment1/preprocess/Ma12P_L_2.khmer.fq.gz --CPU 1 --max_memory 10G --seqType fq --output /home/ubuntu/data/burbot_assemble_out__experiment1/assembly/trinity_out_dir   2> burbot_assemble_out__experiment1/logs/trinity/trinity.log ' returned non-zero exit status 1.
[Wed Feb 13 22:36:30 2019]
Error in rule trinity:
    jobid: 4
    output: /home/ubuntu/data/burbot_assemble_out__experiment1/assembly/trinity_out_dir/Trinity.fasta, /home/ubuntu/data/burbot_assemble_out__experiment1/assembly/trinity_out_dir/Trinity.fasta.gene_trans_map
    log: burbot_assemble_out__experiment1/logs/trinity/trinity.log
    conda-env: /home/ubuntu/eelpond/.snakemake/conda/0d7b2c94

RuleException:
CalledProcessError in line 54 of /home/ubuntu/eelpond/rules/trinity/trinity.rule:
Command 'source activate /home/ubuntu/eelpond/.snakemake/conda/0d7b2c94; set -euo pipefail;  python /home/ubuntu/eelpond/.snakemake/scripts/tmpah23sgvr.trinity-wrapper.py ' returned non-zero exit status 1.
  File "/home/ubuntu/eelpond/rules/trinity/trinity.rule", line 54, in __rule_trinity
  File "/home/ubuntu/miniconda3/envs/eelpond/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /home/ubuntu/eelpond/.snakemake/log/2019-02-13T214058.248119.snakemake.log
bluegenes commented 5 years ago

Hi @mandyfrazier!

Ah, we need to increase the memory we allow Trinity to use by specifying some parameters in your config.yaml file.

steps:

  1. run ./run_eelpond config.yaml trinity --print_params to print trinity parameters to the terminal. Note that config.yaml is a placeholder here, and nothing happens to it.
  2. copy the printed output into your config.yaml file, and modify the max_memory parameter to suit your needs. Trinity recommends ~1G per million reads. You'll probably need less because we do some digital normalization prior to the trinity step.
  3. To speed things up, when you run the workflow, add --threads and specify the number of cpu you want to use (you might be doing this already).

This doc explains in a bit more detail: https://dib-lab.github.io/eelpond/trinity , but also let me know if you have further questions

mandyfrazier commented 5 years ago

Thanks @bluegenes for the suggestions, assembly was successful after increasing CPU and GB in the yaml file!