search

dereneaton / ipyrad

Interactive assembly and analysis of RAD-seq data sets
http://ipyrad.readthedocs.io
GNU General Public License v3.0
72 stars 39 forks source link

Step 3 hanging in unusual way #300

Closed isaacovercast closed 4 years ago

isaacovercast commented 6 years ago 
 I have demultiplexed four lanes of ddRAD data using ipyrad v.0.7.19. Prior to continuing the run, I wanted to explore parameter behaviour on a small subset (12 samples). I am using a HPCC system with 80 nodes and 10GB per cpu. The first test run reached 83% completion of clustering in Step 3 in about 6 hours; however the run has not progressed over the next 2 and a half days. I noticed that ipyrad has not chunked two of the samples (in the tmpalign folder). Has the program stalled?
Step 1: Loading sorted fastq data to Samples
[####################] 100% loading reads | 0:00:11
12 fastq files loaded to 12 Samples.
Step 2: Filtering reads
[####################] 100% processing reads | 0:03:54
Step 3: Clustering/Mapping reads
[####################] 100% dereplicating | 0:00:31
[################ ] 83% clustering | 1 day, 0:17:04
here are the names of the 12 files that I am using for parameter exploration: 15220_DR1.fastq 15424_HR1.fastq 15426_JR1.fastq 15430_CR1.fastq 15500_CR1.fastq 15501_IR1.fastq
15421_IR1.fastq 15424_JR1.fastq 15429_MR1.fastq 15442_AR1.fastq 15501_ER1.fastq 15537_GR1.fastq
All of the names are structurally the same, a collection number, letter for the individual and R1.
/opuntia_TEST80_clust_0.8$ ls -ltr
total 1117696
-rw-r--r-- 1 mporter columbuslab 8353924 Jun 9 10:45 15424_H.htemp
-rw-r--r-- 1 mporter columbuslab 22616994 Jun 9 10:45 15424_H.utemp
-rw-r--r-- 1 mporter columbuslab 22616994 Jun 9 10:45 15424_H.utemp.sort
-rw-r--r-- 1 mporter columbuslab 9748118 Jun 9 10:45 15424_H.clust.gz
-rw-r--r-- 1 mporter columbuslab 11377764 Jun 9 10:48 15429_M.htemp
-rw-r--r-- 1 mporter columbuslab 22575371 Jun 9 10:48 15429_M.utemp
-rw-r--r-- 1 mporter columbuslab 22575371 Jun 9 10:48 15429_M.utemp.sort
-rw-r--r-- 1 mporter columbuslab 13088000 Jun 9 10:48 15501_E.htemp
-rw-r--r-- 1 mporter columbuslab 24381436 Jun 9 10:48 15501_E.utemp
-rw-r--r-- 1 mporter columbuslab 24381436 Jun 9 10:48 15501_E.utemp.sort
-rw-r--r-- 1 mporter columbuslab 10980645 Jun 9 10:48 15429_M.clust.gz
-rw-r--r-- 1 mporter columbuslab 12377527 Jun 9 10:48 15501_E.clust.gz
-rw-r--r-- 1 mporter columbuslab 11966537 Jun 9 10:50 15501_I.htemp
-rw-r--r-- 1 mporter columbuslab 23539654 Jun 9 10:50 15501_I.utemp
-rw-r--r-- 1 mporter columbuslab 23539654 Jun 9 10:50 15501_I.utemp.sort
-rw-r--r-- 1 mporter columbuslab 11618296 Jun 9 10:51 15501_I.clust.gz
-rw-r--r-- 1 mporter columbuslab 17976974 Jun 9 10:52 15500_C.htemp
-rw-r--r-- 1 mporter columbuslab 29978573 Jun 9 10:52 15500_C.utemp
-rw-r--r-- 1 mporter columbuslab 29978573 Jun 9 10:52 15500_C.utemp.sort
-rw-r--r-- 1 mporter columbuslab 16113106 Jun 9 10:52 15500_C.clust.gz
-rw-r--r-- 1 mporter columbuslab 10413974 Jun 9 10:57 15537_G.htemp
-rw-r--r-- 1 mporter columbuslab 33478735 Jun 9 10:57 15537_G.utemp
-rw-r--r-- 1 mporter columbuslab 33478735 Jun 9 10:57 15537_G.utemp.sort
-rw-r--r-- 1 mporter columbuslab 13743582 Jun 9 10:57 15537_G.clust.gz
-rw-r--r-- 1 mporter columbuslab 19084859 Jun 9 11:00 15426_J.htemp
-rw-r--r-- 1 mporter columbuslab 42698205 Jun 9 11:00 15426_J.utemp
-rw-r--r-- 1 mporter columbuslab 42698205 Jun 9 11:00 15426_J.utemp.sort
-rw-r--r-- 1 mporter columbuslab 20166847 Jun 9 11:00 15426_J.clust.gz
-rw-r--r-- 1 mporter columbuslab 23508687 Jun 9 11:05 15442_A.htemp
-rw-r--r-- 1 mporter columbuslab 31803005 Jun 9 11:05 15442_A.utemp
-rw-r--r-- 1 mporter columbuslab 31803005 Jun 9 11:05 15442_A.utemp.sort
-rw-r--r-- 1 mporter columbuslab 18853671 Jun 9 11:06 15442_A.clust.gz
-rw-r--r-- 1 mporter columbuslab 26423095 Jun 9 11:09 15220_D.htemp
-rw-r--r-- 1 mporter columbuslab 35397190 Jun 9 11:09 15220_D.utemp
-rw-r--r-- 1 mporter columbuslab 35397190 Jun 9 11:09 15220_D.utemp.sort
-rw-r--r-- 1 mporter columbuslab 21365333 Jun 9 11:10 15220_D.clust.gz
-rw-r--r-- 1 mporter columbuslab 25360367 Jun 9 19:29 15424_J.htemp
-rw-r--r-- 1 mporter columbuslab 38287291 Jun 9 19:29 15424_J.utemp
-rw-r--r-- 1 mporter columbuslab 38287291 Jun 9 19:29 15424_J.utemp.sort
-rw-r--r-- 1 mporter columbuslab 21389779 Jun 9 19:30 15424_J.clust.gz
-rw-r--r-- 1 mporter columbuslab 50331648 Jun 10 14:31 15430_C.utemp
-rw-r--r-- 1 mporter columbuslab 33554432 Jun 11 01:40 15430_C.htemp
-rw-r--r-- 1 mporter columbuslab 33554432 Jun 11 02:54 15421_I.htemp
-rw-r--r-- 1 mporter columbuslab 83886080 Jun 11 06:28 15421_I.utemp
/opuntia_TEST80_edits$ ls -l
total 3585024
-rw-r--r-- 1 mporter columbuslab 173364756 Jun 9 10:41 15220_D.trimmedR1.fastq.gz
-rw-r--r-- 1 mporter columbuslab 76229835 Jun 11 10:15 15220_D_derep.fastq
-rw-r--r-- 1 mporter columbuslab 254519730 Jun 9 10:40 15421_I.trimmedR1.fastq.gz
-rw-r--r-- 1 mporter columbuslab 311438095 Jun 11 10:15 15421_I_derep.fastq
-rw-r--r-- 1 mporter columbuslab 158156641 Jun 9 10:40 15424_H.trimmedR1.fastq.gz
-rw-r--r-- 1 mporter columbuslab 39915256 Jun 11 10:15 15424_H_derep.fastq
-rw-r--r-- 1 mporter columbuslab 179658938 Jun 9 10:41 15424_J.trimmedR1.fastq.gz
-rw-r--r-- 1 mporter columbuslab 79192111 Jun 11 10:15 15424_J_derep.fastq
-rw-r--r-- 1 mporter columbuslab 274067908 Jun 9 10:41 15426_J.trimmedR1.fastq.gz
-rw-r--r-- 1 mporter columbuslab 78824913 Jun 11 10:15 15426_J_derep.fastq
-rw-r--r-- 1 mporter columbuslab 154886088 Jun 9 10:41 15429_M.trimmedR1.fastq.gz
-rw-r--r-- 1 mporter columbuslab 42928090 Jun 11 10:15 15429_M_derep.fastq
-rw-r--r-- 1 mporter columbuslab 326688904 Jun 9 10:42 15430_C.trimmedR1.fastq.gz
-rw-r--r-- 1 mporter columbuslab 438029303 Jun 11 10:15 15430_C_derep.fastq
-rw-r--r-- 1 mporter columbuslab 168733476 Jun 9 10:40 15442_A.trimmedR1.fastq.gz
-rw-r--r-- 1 mporter columbuslab 68207491 Jun 11 10:15 15442_A_derep.fastq
-rw-r--r-- 1 mporter columbuslab 160165476 Jun 9 10:40 15500_C.trimmedR1.fastq.gz
-rw-r--r-- 1 mporter columbuslab 60039888 Jun 11 10:15 15500_C_derep.fastq
-rw-r--r-- 1 mporter columbuslab 154624083 Jun 9 10:41 15501_E.trimmedR1.fastq.gz
-rw-r--r-- 1 mporter columbuslab 47224379 Jun 11 10:15 15501_E_derep.fastq
-rw-r--r-- 1 mporter columbuslab 155009130 Jun 9 10:40 15501_I.trimmedR1.fastq.gz
-rw-r--r-- 1 mporter columbuslab 44911195 Jun 11 10:15 15501_I_derep.fastq
-rw-r--r-- 1 mporter columbuslab 159926330 Jun 9 10:41 15537_G.trimmedR1.fastq.gz
-rw-r--r-- 1 mporter columbuslab 57211078 Jun 11 10:15 15537_G_derep.fastq
-rw-r--r-- 1 mporter columbuslab 1741 Jun 9 10:42 s2_rawedit_stats.txt
------- ipyrad params file (v.0.5.15)-------------------------------------------
opuntia_TEST80 ## [0] [assembly_name]: Assembly name. Used to name output directories for assembly steps
~/bigdata/ddRAD/opuntia_ipyrad/TESTS ## [1] [project_dir]: Project dir (made in curdir if not present)
## [2] [raw_fastq_path]: Location of raw non-demultiplexed fastq files
## [3] [barcodes_path]: Location of barcodes file
~/bigdata/ddRAD/opuntia_ipyrad/newsclero_TEST_fastqs/*.fastq ## [4] [sorted_fastq_path]: Location of demultiplexed/sorted fastq files
denovo ## [5] [assembly_method]: Assembly method (denovo, reference, denovo+reference, denovo-reference)
## [6] [reference_sequence]: Location of reference sequence file
ddrad ## [7] [datatype]: Datatype (see docs): rad, gbs, ddrad, etc.
TA, TGCA ## [8] [restriction_overhang]: Restriction overhang (cut1,) or (cut1, cut2)
1 ## [9] [max_low_qual_bases]: Max low quality base calls (Q<20) in a read
33 ## [10] [phred_Qscore_offset]: phred Q score offset (33 is default and very standard)
10 ## [11] [mindepth_statistical]: Min depth for statistical base calling
10 ## [12] [mindepth_majrule]: Min depth for majority-rule base calling
10000 ## [13] [maxdepth]: Max cluster depth within samples
0.80 ## [14] [clust_threshold]: Clustering threshold for de novo assembly
0 ## [15] [max_barcode_mismatch]: Max number of allowable mismatches in barcodes
0 ## [16] [filter_adapters]: Filter for adapters/primers (1 or 2=stricter)
50 ## [17] [filter_min_trim_len]: Min length of reads after adapter trim
2 ## [18] [max_alleles_consens]: Max alleles per site in consensus sequences
2 ## [19] [max_Ns_consens]: Max N's (uncalled bases) in consensus (R1, R2)
8 ## [20] [max_Hs_consens]: Max Hs (heterozygotes) in consensus (R1, R2)
4 ## [21] [min_samples_locus]: Min # samples per locus for output
20 ## [22] [max_SNPs_locus]: Max # SNPs per locus (R1, R2)
8 ## [23] [max_Indels_locus]: Max # of indels per locus (R1, R2)
0.5 ## [24] [max_shared_Hs_locus]: Max # heterozygous sites per locus (R1, R2)
0, 0, 0, 0 ## [25] [trim_reads]: Trim raw read edges (R1>, <R1, R2>, <R2) (see docs) 0, 0, 0, 0 ## [26] [trim_loci]: Trim locus edges (see docs) (R1>, <R1, R2>, <R2)
G, a, g, k, m, l, n, p, s, u, t, v ## [27] [output_formats]: Output formats (see docs)
## [28] [pop_assign_file]: Path to population assignment file
isaacovercast commented 6 years ago

This is fucking weird:

-rw-r--r-- 1 mporter columbuslab 33554432 Jun 11 01:40 15430_C.htemp
-rw-r--r-- 1 mporter columbuslab 33554432 Jun 11 02:54 15421_I.htemp
isaacovercast commented 6 years ago

Different runs produce the same behavior with different samples?!?!

In looking at the *clust_0.8 folder I spotted two new files with identical size. As you pointed out, this is very unlikely. Note that this is not the same two files as lasttime, though one individual is the same.
-rw-r--r-- 1 mporter columbuslab 16777216 Jun 11 12:09 15421_I.utemp
-rw-r--r-- 1 mporter columbuslab 16777216 Jun 11 12:01 15424_J.utemp
isaacovercast commented 6 years ago 
/bigdata/columbuslab/mporter/ddRAD/opuntia_ipyrad/TESTS$ grep DEBUG ipyrad_log.txt | tail
2018-06-12 18:37:52,555 pid=6571 [cluster_within.py] DEBUG ['/rhome/mporter/miniconda2/lib/python2.7/site-packages/bin/vsearch-linux-x86_64', '-cluster_smallmem', '/bigdata/columbuslab/mporter/ddRAD/opuntia_ipyrad/TESTS/opuntia_TEST80_edits/15501_I_derep.fastq', '-strand', 'plus', '-query_cov', '0.75', '-id', '0.8', '-minsl', '0.5', '-userout', '/bigdata/columbuslab/mporter/ddRAD/opuntia_ipyrad/TESTS/opuntia_TEST80_clust_0.8/15501_I.utemp', '-userfields', 'query+target+id+gaps+qstrand+qcov', '-maxaccepts', '1', '-maxrejects', '0', '-threads', '4', '-notmatched', '/bigdata/columbuslab/mporter/ddRAD/opuntia_ipyrad/TESTS/opuntia_TEST80_clust_0.8/15501_I.htemp', '-fasta_width', '0', '-fastq_qmax', '100', '-fulldp', '-usersort']
2018-06-12 18:37:52,560 pid=6581 [cluster_within.py] DEBUG ['/rhome/mporter/miniconda2/lib/python2.7/site-packages/bin/vsearch-linux-x86_64', '-cluster_smallmem', '/bigdata/columbuslab/mporter/ddRAD/opuntia_ipyrad/TESTS/opuntia_TEST80_edits/15424_J_derep.fastq', '-strand', 'plus', '-query_cov', '0.75', '-id', '0.8', '-minsl', '0.5', '-userout', '/bigdata/columbuslab/mporter/ddRAD/opuntia_ipyrad/TESTS/opuntia_TEST80_clust_0.8/15424_J.utemp', '-userfields', 'query+target+id+gaps+qstrand+qcov', '-maxaccepts', '1', '-maxrejects', '0', '-threads', '4', '-notmatched', '/bigdata/columbuslab/mporter/ddRAD/opuntia_ipyrad/TESTS/opuntia_TEST80_clust_0.8/15424_J.htemp', '-fasta_width', '0', '-fastq_qmax', '100', '-fulldp', '-usersort']
2018-06-12 18:37:52,569 pid=6570 [cluster_within.py] DEBUG ['/rhome/mporter/miniconda2/lib/python2.7/site-packages/bin/vsearch-linux-x86_64', '-cluster_smallmem', '/bigdata/columbuslab/mporter/ddRAD/opuntia_ipyrad/TESTS/opuntia_TEST80_edits/15430_C_derep.fastq', '-strand', 'plus', '-query_cov', '0.75', '-id', '0.8', '-minsl', '0.5', '-userout', '/bigdata/columbuslab/mporter/ddRAD/opuntia_ipyrad/TESTS/opuntia_TEST80_clust_0.8/15430_C.utemp', '-userfields', 'query+target+id+gaps+qstrand+qcov', '-maxaccepts', '1', '-maxrejects', '0', '-threads', '4', '-notmatched', '/bigdata/columbuslab/mporter/ddRAD/opuntia_ipyrad/TESTS/opuntia_TEST80_clust_0.8/15430_C.htemp', '-fasta_width', '0', '-fastq_qmax', '100', '-fulldp', '-usersort']
2018-06-12 18:37:52,556 pid=34702 [cluster_within.py] DEBUG ['/rhome/mporter/miniconda2/lib/python2.7/site-packages/bin/vsearch-linux-x86_64', '-cluster_smallmem', '/bigdata/columbuslab/mporter/ddRAD/opuntia_ipyrad/TESTS/opuntia_TEST80_edits/15537_G_derep.fastq', '-strand', 'plus', '-query_cov', '0.75', '-id', '0.8', '-minsl', '0.5', '-userout', '/bigdata/columbuslab/mporter/ddRAD/opuntia_ipyrad/TESTS/opuntia_TEST80_clust_0.8/15537_G.utemp', '-userfields', 'query+target+id+gaps+qstrand+qcov', '-maxaccepts', '1', '-maxrejects', '0', '-threads', '4', '-notmatched', '/bigdata/columbuslab/mporter/ddRAD/opuntia_ipyrad/TESTS/opuntia_TEST80_clust_0.8/15537_G.htemp', '-fasta_width', '0', '-fastq_qmax', '100', '-fulldp', '-usersort']
2018-06-12 18:37:52,560 pid=2577 [cluster_within.py] DEBUG ['/rhome/mporter/miniconda2/lib/python2.7/site-packages/bin/vsearch-linux-x86_64', '-cluster_smallmem', '/bigdata/columbuslab/mporter/ddRAD/opuntia_ipyrad/TESTS/opuntia_TEST80_edits/15500_C_derep.fastq', '-strand', 'plus', '-query_cov', '0.75', '-id', '0.8', '-minsl', '0.5', '-userout', '/bigdata/columbuslab/mporter/ddRAD/opuntia_ipyrad/TESTS/opuntia_TEST80_clust_0.8/15500_C.utemp', '-userfields', 'query+target+id+gaps+qstrand+qcov', '-maxaccepts', '1', '-maxrejects', '0', '-threads', '4', '-notmatched', '/bigdata/columbuslab/mporter/ddRAD/opuntia_ipyrad/TESTS/opuntia_TEST80_clust_0.8/15500_C.htemp', '-fasta_width', '0', '-fastq_qmax', '100', '-fulldp', '-usersort']
2018-06-12 18:37:52,574 pid=24784 [cluster_within.py] DEBUG ['/rhome/mporter/miniconda2/lib/python2.7/site-packages/bin/vsearch-linux-x86_64', '-cluster_smallmem', '/bigdata/columbuslab/mporter/ddRAD/opuntia_ipyrad/TESTS/opuntia_TEST80_edits/15426_J_derep.fastq', '-strand', 'p
isaacovercast commented 6 years ago 
ipyrad [v.0.7.19]
Interactive assembly and analysis of RAD-seq data
loading Assembly: opuntia_TEST80
from saved path: ~/bigdata/ddRAD/opuntia_ipyrad/TESTS/opuntia_TEST80.json
establishing parallel connection:
host compute node: [10 cores] on c08
host compute node: [10 cores] on c01
host compute node: [10 cores] on c05
host compute node: [10 cores] on c07
Step 3: Clustering/Mapping reads
[####################] 100% dereplicating | 0:00:36
[################ ] 83% clustering | 14:51:37
isaacovercast commented 4 years ago

I am going to assume this is some kind of non-deterministic behavior from the old version which is not replicatable in the new version.