search

sr320 / LabDocs

Roberts Lab Documents
http://sr320.github.io/LabDocs/
9 stars 17 forks source link

Trinity error on Mox #673

Closed sr320 closed 6 years ago

sr320 commented 6 years ago

Hit bump with Trinity on Mox... seems to be something to do with normalization?

TLDR the error.

Thread 8 terminated abnormally: Error, not all specified records have been retrieved (missing 907086) from /gscratch/srlab/sr320/analyses/trinity_out_dir/NR021_R2_comb.fastq.PwU.qtrim.fq, see file: /gscratch/srlab/sr320/analyses/trinity_out_dir/insilico_read_normalization/NR021_R2_comb.fastq.PwU.qtrim.fq.normalized_K25_C50_pctSD10000.fq.missing_accs for list of missing entries at /gscratch/srlab/programs/trinity/util/insilico_read_normalization.pl line 548.
Thread 7 terminated abnormally: Error, not all specified records have been retrieved (missing 907086) from /gscratch/srlab/sr320/analyses/trinity_out_dir/NR021_R1_comb.fastq.PwU.qtrim.fq, see file: /gscratch/srlab/sr320/analyses/trinity_out_dir/insilico_read_normalization/NR021_R1_comb.fastq.PwU.qtrim.fq.normalized_K25_C50_pctSD10000.fq.missing_accs for list of missing entries at /gscratch/srlab/programs/trinity/util/insilico_read_normalization.pl line 548.
Error encountered with thread.
Error encountered with thread.
Error, at least one thread died at /gscratch/srlab/programs/trinity/util/insilico_read_normalization.pl line 419.
Error, cmd: /gscratch/srlab/programs/trinity/util/insilico_read_normalization.pl --seqType fq --JM 500G  --max_cov 50 --CPU 50 --output /gscratch/srlab/sr320/analyses/trinity_out_dir/insilico_read_normalization   --max_pct_stdev 10000  --left NR021_R1_comb.fastq.PwU.qtrim.fq --right NR021_R2_comb.fastq.PwU.qtrim.fq --pairs_together --PARALLEL_STATS   died with ret 6400 at /gscratch/srlab/programs/trinity/Trinity line 2462.

Full SLURM

Left read files: $VAR1 = [
          '/gscratch/srlab/sr320/data/NR021_R1_comb.fastq'
        ];
Right read files: $VAR1 = [
          '/gscratch/srlab/sr320/data/NR021_R2_comb.fastq'
        ];
Trinity version: Trinity-v2.4.0
-ERROR: couldn't run the network check to confirm latest Trinity software version.

Friday, August 4, 2017: 10:48:33    CMD: java -Xmx64m -XX:ParallelGCThreads=2 -jar /gscratch/srlab/programs/trinity/util/support_scripts/ExitTester.jar 0
Friday, August 4, 2017: 10:48:33    CMD: java -Xmx64m -XX:ParallelGCThreads=2 -jar /gscratch/srlab/programs/trinity/util/support_scripts/ExitTester.jar 1

----------------------------------------------------------------------------------
-------------- Trinity Phase 1: Clustering of RNA-Seq Reads  ---------------------
----------------------------------------------------------------------------------

---------------------------------------------------------------
------ Quality Trimming Via Trimmomatic  ---------------------
<< ILLUMINACLIP:/gscratch/srlab/programs/trinity/trinity-plugins/Trimmomatic/adapters/TruSeq3-PE.fa:2:30:10 SLIDINGWINDOW:4:5 LEADING:5 TRAILING:5 MINLEN:25 >>
---------------------------------------------------------------

## Running Trimmomatic on read files: /gscratch/srlab/sr320/data/NR021_R1_comb.fastq, /gscratch/srlab/sr320/data/NR021_R2_comb.fastq
###############################################################################
#### Trimmomatic  process was previously completed. Skipping it and using existing qual-trimmed files: NR021_R1_comb.fastq.PwU.qtrim.fq, NR021_R2_comb.fastq.PwU.qtrim.fq
###############################################################################
---------------------------------------------------------------
------------ In silico Read Normalization ---------------------
-- (Removing Excess Reads Beyond 50 Coverage --
---------------------------------------------------------------

# running normalization on reads: $VAR1 = [
          [
            'NR021_R1_comb.fastq.PwU.qtrim.fq'
          ],
          [
            'NR021_R2_comb.fastq.PwU.qtrim.fq'
          ]
        ];

Friday, August 4, 2017: 10:48:34    CMD: /gscratch/srlab/programs/trinity/util/insilico_read_normalization.pl --seqType fq --JM 500G  --max_cov 50 --CPU 50 --output /gscratch/srlab/sr320/analyses/trinity_out_dir/insilico_read_normalization   --max_pct_stdev 10000  --left NR021_R1_comb.fastq.PwU.qtrim.fq --right NR021_R2_comb.fastq.PwU.qtrim.fq --pairs_together --PARALLEL_STATS  
Converting input files. (both directions in parallel)left file exists, nothing to doright file exists, nothing to do-------------------------------------------
----------- Jellyfish  --------------------
-- (building a k-mer catalog from reads) --
-------------------------------------------

-sorting each stats file by read name.
Done converting input files.Thread 4 terminated abnormally: Error, not all specified records have been retrieved (missing 907086) from /gscratch/srlab/sr320/analyses/trinity_out_dir/NR021_R2_comb.fastq.PwU.qtrim.fq, see file: /gscratch/srlab/sr320/analyses/trinity_out_dir/insilico_read_normalization/NR021_R2_comb.fastq.PwU.qtrim.fq.normalized_K25_C50_pctSD10000.fq.missing_accs for list of missing entries at /gscratch/srlab/programs/trinity/util/insilico_read_normalization.pl line 548.
Thread 3 terminated abnormally: Error, not all specified records have been retrieved (missing 907086) from /gscratch/srlab/sr320/analyses/trinity_out_dir/NR021_R1_comb.fastq.PwU.qtrim.fq, see file: /gscratch/srlab/sr320/analyses/trinity_out_dir/insilico_read_normalization/NR021_R1_comb.fastq.PwU.qtrim.fq.normalized_K25_C50_pctSD10000.fq.missing_accs for list of missing entries at /gscratch/srlab/programs/trinity/util/insilico_read_normalization.pl line 548.
Error encountered with thread.
Error encountered with thread.
Error, at least one thread died at /gscratch/srlab/programs/trinity/util/insilico_read_normalization.pl line 419.
Error, cmd: /gscratch/srlab/programs/trinity/util/insilico_read_normalization.pl --seqType fq --JM 500G  --max_cov 50 --CPU 50 --output /gscratch/srlab/sr320/analyses/trinity_out_dir/insilico_read_normalization   --max_pct_stdev 10000  --left NR021_R1_comb.fastq.PwU.qtrim.fq --right NR021_R2_comb.fastq.PwU.qtrim.fq --pairs_together --PARALLEL_STATS   died with ret 6400 at /gscratch/srlab/programs/trinity/Trinity line 2462.

Script

#!/bin/bash
## Job Name
#SBATCH --job-name=Geoduck-trinity-01
## Allocation Definition 
#SBATCH --account=srlab
#SBATCH --partition=srlab
## Resources
## Nodes (We only get 1, so this is fixed)
#SBATCH --nodes=1   
## Walltime (days-hours:minutes:seconds format)
#SBATCH --time=90:30:00
## Memory per node
#SBATCH --mem=500G
## Specify the working directory for this job
#SBATCH --workdir=/gscratch/srlab/sr320/analyses/

source /gscratch/srlab/programs/scripts/paths.sh

Trinity \
--seqType fq \
--left /gscratch/srlab/sr320/data/NR021_R1_comb.fastq \
--right /gscratch/srlab/sr320/data/NR021_R2_comb.fastq \
--CPU 50 --trimmomatic --max_memory 500G
sr320 commented 6 years ago

Maybe getting further...


 done parsing 116846678 Kmers, 116846678 added, taking 131 seconds.
STATS_GENERATION_TIME: 113 seconds.
STATS_GENERATION_TIME: 113 seconds.
CMD finished (269 seconds)
CMD finished (269 seconds)
CMD: touch left.fa.K25.stats.ok
CMD finished (0 seconds)
CMD: touch right.fa.K25.stats.ok
-sorting each stats file by read name.
CMD finished (0 seconds)
CMD: /bin/sort --parallel=50 -k5,5 -T . -S 250G left.fa.K25.stats > left.fa.K25.stats.sort
CMD: /bin/sort --parallel=50 -k5,5 -T . -S 250G right.fa.K25.stats > right.fa.K25.stats.sort
CMD finished (3 seconds)
CMD finished (4 seconds)
CMD: touch left.fa.K25.stats.sort.ok
CMD finished (0 seconds)
CMD: touch right.fa.K25.stats.sort.ok
CMD finished (0 seconds)
CMD: /gscratch/srlab/programs/trinity/util/..//util/support_scripts//nbkc_merge_left_right_stats.pl --left left.fa.K25.stats.sort --right right.fa.K25.stats.sort --sorted > pairs.K25.stats
-opening left.fa.K25.stats.sort
-opening right.fa.K25.stats.sort
-done opening files.
CMD finished (28 seconds)
CMD: touch pairs.K25.stats.ok
CMD finished (0 seconds)
CMD: /gscratch/srlab/programs/trinity/util/..//util/support_scripts//nbkc_normalize.pl pairs.K25.stats 50 10000 > pairs.K25.stats.C50.pctSD10000.accs
3860667 / 5969058 = 64.68% reads selected during normalization.
0 / 5969058 = 0.00% reads discarded as likely aberrant based on coverage profiles.
0 / 5969058 = 0.00% reads missing kmer coverage (N chars included?).
CMD finished (9 seconds)
CMD: touch pairs.K25.stats.C50.pctSD10000.accs.ok
CMD finished (0 seconds)
CMD: touch /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/insilico_read_normalization/NR021_S8_L001_R1_001.fastq.PwU.qtrim.fq_ext_all_reads.normalized_K25_C50_pctSD10000.fq.ok
CMD finished (0 seconds)
CMD: touch /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/insilico_read_normalization/NR021_S8_L001_R2_001.fastq.PwU.qtrim.fq_ext_all_reads.normalized_K25_C50_pctSD10000.fq.ok
CMD finished (0 seconds)
CMD: ln -sf /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/insilico_read_normalization/NR021_S8_L001_R1_001.fastq.PwU.qtrim.fq_ext_all_reads.normalized_K25_C50_pctSD10000.fq left.norm.fq
CMD finished (0 seconds)
CMD: ln -sf /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/insilico_read_normalization/NR021_S8_L001_R2_001.fastq.PwU.qtrim.fq_ext_all_reads.normalized_K25_C50_pctSD10000.fq right.norm.fq
-removing tmp dir /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/insilico_read_normalization/tmp_normalized_reads
CMD finished (0 seconds)

Normalization complete. See outputs: 
    /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/insilico_read_normalization/NR021_S8_L001_R1_001.fastq.PwU.qtrim.fq_ext_all_reads.normalized_K25_C50_pctSD10000.fq
    /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/insilico_read_normalization/NR021_S8_L001_R2_001.fastq.PwU.qtrim.fq_ext_all_reads.normalized_K25_C50_pctSD10000.fq
Friday, August 4, 2017: 12:11:19    CMD: touch /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/insilico_read_normalization/normalization.ok
Converting input files. (in parallel)Friday, August 4, 2017: 12:11:19   CMD: cat /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/insilico_read_normalization/left.norm.fq | seqtk-trinity seq -A - >> left.fa
Friday, August 4, 2017: 12:11:19    CMD: cat /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/insilico_read_normalization/right.norm.fq | seqtk-trinity seq -A - >> right.fa
Friday, August 4, 2017: 12:11:22    CMD: touch left.fa.ok
Friday, August 4, 2017: 12:11:23    CMD: touch right.fa.ok
Friday, August 4, 2017: 12:11:23    CMD: touch left.fa.ok right.fa.ok
Friday, August 4, 2017: 12:11:23    CMD: cat left.fa right.fa > /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/both.fa
Friday, August 4, 2017: 12:11:24    CMD: touch /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/both.fa.ok
-------------------------------------------
----------- Jellyfish  --------------------
-- (building a k-mer catalog from reads) --
-------------------------------------------

* Running CMD: /gscratch/srlab/programs/trinity/trinity-plugins/jellyfish/bin/jellyfish count -t 50 -m 25 -s 100000000  --canonical  /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/both.fa
* Running CMD: /gscratch/srlab/programs/trinity/trinity-plugins/jellyfish/bin/jellyfish dump -L 1 mer_counts.jf > jellyfish.kmers.fa
* Running CMD: /gscratch/srlab/programs/trinity/trinity-plugins/jellyfish/bin/jellyfish histo -t 50 -o jellyfish.kmers.fa.histo mer_counts.jf
----------------------------------------------
--------------- Inchworm ---------------------
-- (Linear contig construction from k-mers) --
----------------------------------------------

* Running CMD: /gscratch/srlab/programs/trinity/Inchworm/bin//inchworm --kmers jellyfish.kmers.fa --run_inchworm -K 25 -L 25 --monitor 1   --DS  --num_threads 6  --PARALLEL_IWORM  > /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/inchworm.K25.L25.DS.fa.tmp
[sr320@mox1 trinity-geo-02]$ pbcopy 
-bash: pbcopy: command not found
[sr320@mox1 trinity-geo-02]$ clear

[sr320@mox1 trinity-geo-02]$ cat slurm-54837.out 
Left read files: $VAR1 = [
          '/gscratch/srlab/sr320/data/NR021_S8_L001_R1_001.fastq',
          '/gscratch/srlab/sr320/data/NR021_S8_L002_R1_001.fastq'
        ];
Right read files: $VAR1 = [
          '/gscratch/srlab/sr320/data/NR021_S8_L001_R2_001.fastq',
          '/gscratch/srlab/sr320/data/NR021_S8_L002_R2_001.fastq'
        ];
Trinity version: Trinity-v2.4.0
-ERROR: couldn't run the network check to confirm latest Trinity software version.

Friday, August 4, 2017: 12:03:08    CMD: java -Xmx64m -XX:ParallelGCThreads=2 -jar /gscratch/srlab/programs/trinity/util/support_scripts/ExitTester.jar 0
Friday, August 4, 2017: 12:03:08    CMD: java -Xmx64m -XX:ParallelGCThreads=2 -jar /gscratch/srlab/programs/trinity/util/support_scripts/ExitTester.jar 1
Friday, August 4, 2017: 12:03:08    CMD: mkdir -p /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir
Friday, August 4, 2017: 12:03:08    CMD: mkdir -p /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/chrysalis

----------------------------------------------------------------------------------
-------------- Trinity Phase 1: Clustering of RNA-Seq Reads  ---------------------
----------------------------------------------------------------------------------

---------------------------------------------------------------
------ Quality Trimming Via Trimmomatic  ---------------------
<< ILLUMINACLIP:/gscratch/srlab/programs/trinity/trinity-plugins/Trimmomatic/adapters/TruSeq3-PE.fa:2:30:10 SLIDINGWINDOW:4:5 LEADING:5 TRAILING:5 MINLEN:25 >>
---------------------------------------------------------------

## Running Trimmomatic on read files: /gscratch/srlab/sr320/data/NR021_S8_L001_R1_001.fastq, /gscratch/srlab/sr320/data/NR021_S8_L001_R2_001.fastq
Friday, August 4, 2017: 12:03:08    CMD: java -jar /gscratch/srlab/programs/trinity/trinity-plugins/Trimmomatic/trimmomatic.jar PE -threads 50 -phred33  /gscratch/srlab/sr320/data/NR021_S8_L001_R1_001.fastq /gscratch/srlab/sr320/data/NR021_S8_L001_R2_001.fastq  NR021_S8_L001_R1_001.fastq.P.qtrim NR021_S8_L001_R1_001.fastq.U.qtrim  NR021_S8_L001_R2_001.fastq.P.qtrim NR021_S8_L001_R2_001.fastq.U.qtrim  ILLUMINACLIP:/gscratch/srlab/programs/trinity/trinity-plugins/Trimmomatic/adapters/TruSeq3-PE.fa:2:30:10 SLIDINGWINDOW:4:5 LEADING:5 TRAILING:5 MINLEN:25 
TrimmomaticPE: Started with arguments:
 -threads 50 -phred33 /gscratch/srlab/sr320/data/NR021_S8_L001_R1_001.fastq /gscratch/srlab/sr320/data/NR021_S8_L001_R2_001.fastq NR021_S8_L001_R1_001.fastq.P.qtrim NR021_S8_L001_R1_001.fastq.U.qtrim NR021_S8_L001_R2_001.fastq.P.qtrim NR021_S8_L001_R2_001.fastq.U.qtrim ILLUMINACLIP:/gscratch/srlab/programs/trinity/trinity-plugins/Trimmomatic/adapters/TruSeq3-PE.fa:2:30:10 SLIDINGWINDOW:4:5 LEADING:5 TRAILING:5 MINLEN:25
Using PrefixPair: 'TACACTCTTTCCCTACACGACGCTCTTCCGATCT' and 'GTGACTGGAGTTCAGACGTGTGCTCTTCCGATCT'
ILLUMINACLIP: Using 1 prefix pairs, 0 forward/reverse sequences, 0 forward only sequences, 0 reverse only sequences
Input Read Pairs: 4037578 Both Surviving: 3184940 (78.88%) Forward Only Surviving: 850809 (21.07%) Reverse Only Surviving: 0 (0.00%) Dropped: 1829 (0.05%)
TrimmomaticPE: Completed successfully
Friday, August 4, 2017: 12:03:14    CMD: cp NR021_S8_L001_R1_001.fastq.P.qtrim NR021_S8_L001_R1_001.fastq.PwU.qtrim.fq
Friday, August 4, 2017: 12:03:15    CMD: cp NR021_S8_L001_R2_001.fastq.P.qtrim NR021_S8_L001_R2_001.fastq.PwU.qtrim.fq
Friday, August 4, 2017: 12:03:16    CMD: touch trimmomatic.ok
Friday, August 4, 2017: 12:03:16    CMD: gzip NR021_S8_L001_R1_001.fastq.P.qtrim NR021_S8_L001_R1_001.fastq.U.qtrim NR021_S8_L001_R2_001.fastq.P.qtrim NR021_S8_L001_R2_001.fastq.U.qtrim &

## Running Trimmomatic on read files: /gscratch/srlab/sr320/data/NR021_S8_L002_R1_001.fastq, /gscratch/srlab/sr320/data/NR021_S8_L002_R2_001.fastq
Friday, August 4, 2017: 12:03:16    CMD: java -jar /gscratch/srlab/programs/trinity/trinity-plugins/Trimmomatic/trimmomatic.jar PE -threads 50 -phred33  /gscratch/srlab/sr320/data/NR021_S8_L002_R1_001.fastq /gscratch/srlab/sr320/data/NR021_S8_L002_R2_001.fastq  NR021_S8_L002_R1_001.fastq.P.qtrim NR021_S8_L002_R1_001.fastq.U.qtrim  NR021_S8_L002_R2_001.fastq.P.qtrim NR021_S8_L002_R2_001.fastq.U.qtrim  ILLUMINACLIP:/gscratch/srlab/programs/trinity/trinity-plugins/Trimmomatic/adapters/TruSeq3-PE.fa:2:30:10 SLIDINGWINDOW:4:5 LEADING:5 TRAILING:5 MINLEN:25 
TrimmomaticPE: Started with arguments:
 -threads 50 -phred33 /gscratch/srlab/sr320/data/NR021_S8_L002_R1_001.fastq /gscratch/srlab/sr320/data/NR021_S8_L002_R2_001.fastq NR021_S8_L002_R1_001.fastq.P.qtrim NR021_S8_L002_R1_001.fastq.U.qtrim NR021_S8_L002_R2_001.fastq.P.qtrim NR021_S8_L002_R2_001.fastq.U.qtrim ILLUMINACLIP:/gscratch/srlab/programs/trinity/trinity-plugins/Trimmomatic/adapters/TruSeq3-PE.fa:2:30:10 SLIDINGWINDOW:4:5 LEADING:5 TRAILING:5 MINLEN:25
Using PrefixPair: 'TACACTCTTTCCCTACACGACGCTCTTCCGATCT' and 'GTGACTGGAGTTCAGACGTGTGCTCTTCCGATCT'
ILLUMINACLIP: Using 1 prefix pairs, 0 forward/reverse sequences, 0 forward only sequences, 0 reverse only sequences
Input Read Pairs: 3575592 Both Surviving: 2784118 (77.86%) Forward Only Surviving: 788777 (22.06%) Reverse Only Surviving: 0 (0.00%) Dropped: 2697 (0.08%)
TrimmomaticPE: Completed successfully
Friday, August 4, 2017: 12:03:22    CMD: cp NR021_S8_L002_R1_001.fastq.P.qtrim NR021_S8_L002_R1_001.fastq.PwU.qtrim.fq
Friday, August 4, 2017: 12:03:23    CMD: cp NR021_S8_L002_R2_001.fastq.P.qtrim NR021_S8_L002_R2_001.fastq.PwU.qtrim.fq
Friday, August 4, 2017: 12:03:24    CMD: touch trimmomatic.ok
Friday, August 4, 2017: 12:03:24    CMD: gzip NR021_S8_L002_R1_001.fastq.P.qtrim NR021_S8_L002_R1_001.fastq.U.qtrim NR021_S8_L002_R2_001.fastq.P.qtrim NR021_S8_L002_R2_001.fastq.U.qtrim &
---------------------------------------------------------------
------------ In silico Read Normalization ---------------------
-- (Removing Excess Reads Beyond 50 Coverage --
---------------------------------------------------------------

# running normalization on reads: $VAR1 = [
          [
            'NR021_S8_L001_R1_001.fastq.PwU.qtrim.fq',
            'NR021_S8_L002_R1_001.fastq.PwU.qtrim.fq'
          ],
          [
            'NR021_S8_L001_R2_001.fastq.PwU.qtrim.fq',
            'NR021_S8_L002_R2_001.fastq.PwU.qtrim.fq'
          ]
        ];

Friday, August 4, 2017: 12:03:24    CMD: /gscratch/srlab/programs/trinity/util/insilico_read_normalization.pl --seqType fq --JM 500G  --max_cov 50 --CPU 50 --output /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/insilico_read_normalization   --max_pct_stdev 10000  --left NR021_S8_L001_R1_001.fastq.PwU.qtrim.fq,NR021_S8_L002_R1_001.fastq.PwU.qtrim.fq --right NR021_S8_L001_R2_001.fastq.PwU.qtrim.fq,NR021_S8_L002_R2_001.fastq.PwU.qtrim.fq --pairs_together --PARALLEL_STATS  
Converting input files. (both directions in parallel)CMD: seqtk-trinity seq -A /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/NR021_S8_L001_R1_001.fastq.PwU.qtrim.fq >> left.fa
CMD: seqtk-trinity seq -A /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/NR021_S8_L001_R2_001.fastq.PwU.qtrim.fq >> right.fa
CMD finished (2 seconds)
CMD: seqtk-trinity seq -A /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/NR021_S8_L002_R1_001.fastq.PwU.qtrim.fq >> left.fa
CMD finished (2 seconds)
CMD: seqtk-trinity seq -A /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/NR021_S8_L002_R2_001.fastq.PwU.qtrim.fq >> right.fa
CMD finished (3 seconds)
CMD finished (3 seconds)
CMD: touch left.fa.ok
CMD finished (0 seconds)
CMD: touch right.fa.ok
CMD finished (0 seconds)
Done converting input files.CMD: cat left.fa right.fa > both.fa
CMD finished (3 seconds)
CMD: touch both.fa.ok
-------------------------------------------
----------- Jellyfish  --------------------
-- (building a k-mer catalog from reads) --
-------------------------------------------

CMD finished (0 seconds)
CMD: /gscratch/srlab/programs/trinity/util/..//trinity-plugins/jellyfish/bin/jellyfish count -t 50 -m 25 -s 100000000  --canonical  both.fa
CMD finished (30 seconds)
CMD: /gscratch/srlab/programs/trinity/util/..//trinity-plugins/jellyfish/bin/jellyfish histo -t 50 -o jellyfish.K25.min2.kmers.fa.histo mer_counts.jf
CMD finished (14 seconds)
CMD: /gscratch/srlab/programs/trinity/util/..//trinity-plugins/jellyfish/bin/jellyfish dump -L 2 mer_counts.jf > jellyfish.K25.min2.kmers.fa
CMD finished (36 seconds)
CMD: touch jellyfish.K25.min2.kmers.fa.success
CMD finished (0 seconds)
CMD: /gscratch/srlab/programs/trinity/util/..//Inchworm/bin/fastaToKmerCoverageStats --reads left.fa --kmers jellyfish.K25.min2.kmers.fa --kmer_size 25  --num_threads 50  --DS  > left.fa.K25.stats
CMD: /gscratch/srlab/programs/trinity/util/..//Inchworm/bin/fastaToKmerCoverageStats --reads right.fa --kmers jellyfish.K25.min2.kmers.fa --kmer_size 25  --num_threads 50  --DS  > right.fa.K25.stats
-reading Kmer occurrences...
-reading Kmer occurrences...

 done parsing 116846678 Kmers, 116846678 added, taking 130 seconds.

 done parsing 116846678 Kmers, 116846678 added, taking 131 seconds.
STATS_GENERATION_TIME: 113 seconds.
STATS_GENERATION_TIME: 113 seconds.
CMD finished (269 seconds)
CMD finished (269 seconds)
CMD: touch left.fa.K25.stats.ok
CMD finished (0 seconds)
CMD: touch right.fa.K25.stats.ok
-sorting each stats file by read name.
CMD finished (0 seconds)
CMD: /bin/sort --parallel=50 -k5,5 -T . -S 250G left.fa.K25.stats > left.fa.K25.stats.sort
CMD: /bin/sort --parallel=50 -k5,5 -T . -S 250G right.fa.K25.stats > right.fa.K25.stats.sort
CMD finished (3 seconds)
CMD finished (4 seconds)
CMD: touch left.fa.K25.stats.sort.ok
CMD finished (0 seconds)
CMD: touch right.fa.K25.stats.sort.ok
CMD finished (0 seconds)
CMD: /gscratch/srlab/programs/trinity/util/..//util/support_scripts//nbkc_merge_left_right_stats.pl --left left.fa.K25.stats.sort --right right.fa.K25.stats.sort --sorted > pairs.K25.stats
-opening left.fa.K25.stats.sort
-opening right.fa.K25.stats.sort
-done opening files.
CMD finished (28 seconds)
CMD: touch pairs.K25.stats.ok
CMD finished (0 seconds)
CMD: /gscratch/srlab/programs/trinity/util/..//util/support_scripts//nbkc_normalize.pl pairs.K25.stats 50 10000 > pairs.K25.stats.C50.pctSD10000.accs
3860667 / 5969058 = 64.68% reads selected during normalization.
0 / 5969058 = 0.00% reads discarded as likely aberrant based on coverage profiles.
0 / 5969058 = 0.00% reads missing kmer coverage (N chars included?).
CMD finished (9 seconds)
CMD: touch pairs.K25.stats.C50.pctSD10000.accs.ok
CMD finished (0 seconds)
CMD: touch /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/insilico_read_normalization/NR021_S8_L001_R1_001.fastq.PwU.qtrim.fq_ext_all_reads.normalized_K25_C50_pctSD10000.fq.ok
CMD finished (0 seconds)
CMD: touch /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/insilico_read_normalization/NR021_S8_L001_R2_001.fastq.PwU.qtrim.fq_ext_all_reads.normalized_K25_C50_pctSD10000.fq.ok
CMD finished (0 seconds)
CMD: ln -sf /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/insilico_read_normalization/NR021_S8_L001_R1_001.fastq.PwU.qtrim.fq_ext_all_reads.normalized_K25_C50_pctSD10000.fq left.norm.fq
CMD finished (0 seconds)
CMD: ln -sf /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/insilico_read_normalization/NR021_S8_L001_R2_001.fastq.PwU.qtrim.fq_ext_all_reads.normalized_K25_C50_pctSD10000.fq right.norm.fq
-removing tmp dir /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/insilico_read_normalization/tmp_normalized_reads
CMD finished (0 seconds)

Normalization complete. See outputs: 
    /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/insilico_read_normalization/NR021_S8_L001_R1_001.fastq.PwU.qtrim.fq_ext_all_reads.normalized_K25_C50_pctSD10000.fq
    /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/insilico_read_normalization/NR021_S8_L001_R2_001.fastq.PwU.qtrim.fq_ext_all_reads.normalized_K25_C50_pctSD10000.fq
Friday, August 4, 2017: 12:11:19    CMD: touch /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/insilico_read_normalization/normalization.ok
Converting input files. (in parallel)Friday, August 4, 2017: 12:11:19   CMD: cat /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/insilico_read_normalization/left.norm.fq | seqtk-trinity seq -A - >> left.fa
Friday, August 4, 2017: 12:11:19    CMD: cat /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/insilico_read_normalization/right.norm.fq | seqtk-trinity seq -A - >> right.fa
Friday, August 4, 2017: 12:11:22    CMD: touch left.fa.ok
Friday, August 4, 2017: 12:11:23    CMD: touch right.fa.ok
Friday, August 4, 2017: 12:11:23    CMD: touch left.fa.ok right.fa.ok
Friday, August 4, 2017: 12:11:23    CMD: cat left.fa right.fa > /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/both.fa
Friday, August 4, 2017: 12:11:24    CMD: touch /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/both.fa.ok
-------------------------------------------
----------- Jellyfish  --------------------
-- (building a k-mer catalog from reads) --
-------------------------------------------

* Running CMD: /gscratch/srlab/programs/trinity/trinity-plugins/jellyfish/bin/jellyfish count -t 50 -m 25 -s 100000000  --canonical  /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/both.fa
* Running CMD: /gscratch/srlab/programs/trinity/trinity-plugins/jellyfish/bin/jellyfish dump -L 1 mer_counts.jf > jellyfish.kmers.fa
* Running CMD: /gscratch/srlab/programs/trinity/trinity-plugins/jellyfish/bin/jellyfish histo -t 50 -o jellyfish.kmers.fa.histo mer_counts.jf
----------------------------------------------
--------------- Inchworm ---------------------
-- (Linear contig construction from k-mers) --
----------------------------------------------

* Running CMD: /gscratch/srlab/programs/trinity/Inchworm/bin//inchworm --kmers jellyfish.kmers.fa --run_inchworm -K 25 -L 25 --monitor 1   --DS  --num_threads 6  --PARALLEL_IWORM  > /gscratch/srlab/sr320/analyses/trinity-geo-02/trinity_out_dir/inchworm.K25.L25.DS.fa.tmp
[sr320@mox1 trinity-geo-02]$
sr320 commented 6 years ago

Not sure what was different but script is here-

Could be cat of fq caused problem as here separate fastq are add with comma

#!/bin/bash
## Job Name
#SBATCH --job-name=Geoduck-trinity-01
## Allocation Definition 
#SBATCH --account=srlab
#SBATCH --partition=srlab
## Resources
## Nodes (We only get 1, so this is fixed)
#SBATCH --nodes=1   
## Walltime (days-hours:minutes:seconds format)
#SBATCH --time=90:30:00
## Memory per node
#SBATCH --mem=500G
## Specify the working directory for this job
#SBATCH --workdir=/gscratch/srlab/sr320/analyses/02

source /gscratch/srlab/programs/scripts/paths.sh

Trinity \
--seqType fq \
--left /gscratch/srlab/sr320/data/NR021_S8_L001_R1_001.fastq,/gscratch/srlab/sr320/data/NR021_S8_L002_R1_001.fastq \
--right /gscratch/srlab/sr320/data/NR021_S8_L001_R2_001.fastq,/gscratch/srlab/sr320/data/NR021_S8_L002_R2_001.fastq \
--CPU 50 --trimmomatic --max_memory 500G