search

superphy / prairiedog

next-gen pangenome graphs for predictive genomics
Other
0 stars 0 forks source link

python3: lib/lemongraph.c:622: _edge_index: Assertion `DB_SUCCESS == r' failed. #70

Closed kevinkle closed 5 years ago

kevinkle commented 5 years ago

Same file as #67 

2019-06-20 09:18:49 panther prairiedog[16565] DEBUG 4800000/4975087, 96%
2019-06-20 09:18:58 panther prairiedog[16565] DEBUG 4900000/4975087, 98%
2019-06-20 09:19:05 panther prairiedog[16565] DEBUG Done graphing SRR3665189.fasta, covering 4975087 kmers in 415.99421286582947 s
Current graph size is 2.240581512451172 GB
rule 'pangenome' on Kmer 3 / 10
2019-06-20 09:19:08 panther prairiedog[16565] INFO Indexing SRR3295596.fasta ...
2019-06-20 09:19:08 panther prairiedog[16565] INFO Done indexing SRR3295596.fasta
2019-06-20 09:19:08 panther prairiedog[16565] DEBUG Starting to graph SRR3295596.fasta in pid 16565
2019-06-20 09:19:16 panther prairiedog[16565] DEBUG 100000/5002124, 1%
2019-06-20 09:19:24 panther prairiedog[16565] DEBUG 200000/5002124, 3%
2019-06-20 09:19:33 panther prairiedog[16565] DEBUG 300000/5002124, 5%
2019-06-20 09:19:42 panther prairiedog[16565] DEBUG 400000/5002124, 7%
2019-06-20 09:19:50 panther prairiedog[16565] DEBUG 500000/5002124, 9%
2019-06-20 09:19:59 panther prairiedog[16565] DEBUG 600000/5002124, 11%
2019-06-20 09:20:08 panther prairiedog[16565] DEBUG 700000/5002124, 13%
2019-06-20 09:20:16 panther prairiedog[16565] DEBUG 800000/5002124, 15%
2019-06-20 09:20:25 panther prairiedog[16565] DEBUG 900000/5002124, 17%
2019-06-20 09:20:33 panther prairiedog[16565] DEBUG 1000000/5002124, 19%
2019-06-20 09:20:41 panther prairiedog[16565] DEBUG 1100000/5002124, 21%
2019-06-20 09:20:50 panther prairiedog[16565] DEBUG 1200000/5002124, 23%
2019-06-20 09:20:58 panther prairiedog[16565] DEBUG 1300000/5002124, 25%
2019-06-20 09:21:07 panther prairiedog[16565] DEBUG 1400000/5002124, 27%
2019-06-20 09:21:15 panther prairiedog[16565] DEBUG 1500000/5002124, 29%
2019-06-20 09:21:24 panther prairiedog[16565] DEBUG 1600000/5002124, 31%
2019-06-20 09:21:33 panther prairiedog[16565] DEBUG 1700000/5002124, 33%
2019-06-20 09:21:41 panther prairiedog[16565] DEBUG 1800000/5002124, 35%
2019-06-20 09:21:50 panther prairiedog[16565] DEBUG 1900000/5002124, 37%
2019-06-20 09:21:59 panther prairiedog[16565] DEBUG 2000000/5002124, 39%
2019-06-20 09:22:08 panther prairiedog[16565] DEBUG 2100000/5002124, 41%
python3: lib/lemongraph.c:622: _edge_index: Assertion `DB_SUCCESS == r' failed.
Aborted
Shutting down, this might take some time.
2019-06-20 09:22:14 panther snakemake.logging[15995] WARNING Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
2019-06-20 09:22:14 panther snakemake.logging[15995] ERROR Exiting because a job execution failed. Look above for error message
Complete log: /home/kevin/prairiedog/.snakemake/log/2019-06-20T090428.769167.snakemake.log
2019-06-20 09:22:14 panther snakemake.logging[15995] WARNING Complete log: /home/kevin/prairiedog/.snakemake/log/2019-06-20T090428.769167.snakemake.log
(venv) kevin@panther:~/prairiedog$
kevinkle commented 5 years ago 
2019-06-20 09:40:57 panther prairiedog[17209] DEBUG 4900000/4975087, 98%
2019-06-20 09:41:04 panther prairiedog[17209] DEBUG Done graphing SRR3665189.fasta, covering 4975087 kmers in 423.2506105899811 s
Current graph size is 2.242229461669922 GB
rule 'pangenome' on Kmer 3 / 9
2019-06-20 09:41:08 panther prairiedog[17209] INFO Indexing SRR2567162.fasta ...
2019-06-20 09:41:08 panther prairiedog[17209] INFO Done indexing SRR2567162.fasta
2019-06-20 09:41:08 panther prairiedog[17209] DEBUG Starting to graph SRR2567162.fasta in pid 17209
2019-06-20 09:41:16 panther prairiedog[17209] DEBUG 100000/4927080, 2%
2019-06-20 09:41:24 panther prairiedog[17209] DEBUG 200000/4927080, 4%
2019-06-20 09:41:32 panther prairiedog[17209] DEBUG 300000/4927080, 6%
2019-06-20 09:41:41 panther prairiedog[17209] DEBUG 400000/4927080, 8%
2019-06-20 09:41:49 panther prairiedog[17209] DEBUG 500000/4927080, 10%
2019-06-20 09:41:57 panther prairiedog[17209] DEBUG 600000/4927080, 12%
2019-06-20 09:42:06 panther prairiedog[17209] DEBUG 700000/4927080, 14%
2019-06-20 09:42:15 panther prairiedog[17209] DEBUG 800000/4927080, 16%
2019-06-20 09:42:24 panther prairiedog[17209] DEBUG 900000/4927080, 18%
2019-06-20 09:42:33 panther prairiedog[17209] DEBUG 1000000/4927080, 20%
2019-06-20 09:42:42 panther prairiedog[17209] DEBUG 1100000/4927080, 22%
2019-06-20 09:42:51 panther prairiedog[17209] DEBUG 1200000/4927080, 24%
2019-06-20 09:43:00 panther prairiedog[17209] DEBUG 1300000/4927080, 26%
2019-06-20 09:43:09 panther prairiedog[17209] DEBUG 1400000/4927080, 28%
2019-06-20 09:43:17 panther prairiedog[17209] DEBUG 1500000/4927080, 30%
2019-06-20 09:43:26 panther prairiedog[17209] DEBUG 1600000/4927080, 32%
2019-06-20 09:43:34 panther prairiedog[17209] DEBUG 1700000/4927080, 34%
2019-06-20 09:43:43 panther prairiedog[17209] DEBUG 1800000/4927080, 36%
2019-06-20 09:43:52 panther prairiedog[17209] DEBUG 1900000/4927080, 38%
2019-06-20 09:44:01 panther prairiedog[17209] DEBUG 2000000/4927080, 40%
2019-06-20 09:44:10 panther prairiedog[17209] DEBUG 2100000/4927080, 42%
2019-06-20 09:44:19 panther prairiedog[17209] DEBUG 2200000/4927080, 44%
python3: lib/lemongraph.c:615: _edge_index: Assertion `DB_SUCCESS == r' failed.
Aborted
Shutting down, this might take some time.
2019-06-20 09:44:23 panther snakemake.logging[16691] WARNING Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
2019-06-20 09:44:23 panther snakemake.logging[16691] ERROR Exiting because a job execution failed. Look above for error message
Complete log: /home/kevin/prairiedog/.snakemake/log/2019-06-20T092610.702269.snakemake.log
2019-06-20 09:44:23 panther snakemake.logging[16691] WARNING Complete log: /home/kevin/prairiedog/.snakemake/log/2019-06-20T092610.702269.snakemake.log
kevinkle commented 5 years ago

https://github.com/superphy/prairiedog/tree/headless-badda51 works fine for more than 3+ genomes and if probably what we benchmarked https://github.com/superphy/prairiedog/issues/53#issuecomment-501756475 against. But this doesnt include the incr encodings yet - will have to figure a way to get this to work in large datasets

kevinkle commented 5 years ago

This is after https://github.com/superphy/prairiedog/commit/d6e73df84b839e401df1553e100ec26c0d958bae where we switched edge_type to an encoded value

2019-06-20 16:01:29 panther prairiedog[21362] DEBUG Done graphing SRR3665189.fasta, covering 4975087 kmers in 450.1172878742218 s
Current graph size is 2.124927520751953 GB
rule 'pangenome' on Kmer 3 / 9
2019-06-20 16:01:32 panther prairiedog[21362] INFO Indexing SRR2567162.fasta ...
2019-06-20 16:01:32 panther prairiedog[21362] INFO Done indexing SRR2567162.fasta
2019-06-20 16:01:32 panther prairiedog[21362] DEBUG Starting to graph SRR2567162.fasta in pid 21362
2019-06-20 16:01:41 panther prairiedog[21362] DEBUG 100000/4927080, 2%
2019-06-20 16:01:50 panther prairiedog[21362] DEBUG 200000/4927080, 4%
2019-06-20 16:01:59 panther prairiedog[21362] DEBUG 300000/4927080, 6%
2019-06-20 16:02:07 panther prairiedog[21362] DEBUG 400000/4927080, 8%
2019-06-20 16:02:16 panther prairiedog[21362] DEBUG 500000/4927080, 10%
2019-06-20 16:02:25 panther prairiedog[21362] DEBUG 600000/4927080, 12%
2019-06-20 16:02:35 panther prairiedog[21362] DEBUG 700000/4927080, 14%
2019-06-20 16:02:44 panther prairiedog[21362] DEBUG 800000/4927080, 16%
2019-06-20 16:02:54 panther prairiedog[21362] DEBUG 900000/4927080, 18%
2019-06-20 16:03:03 panther prairiedog[21362] DEBUG 1000000/4927080, 20%
2019-06-20 16:03:13 panther prairiedog[21362] DEBUG 1100000/4927080, 22%
2019-06-20 16:03:22 panther prairiedog[21362] DEBUG 1200000/4927080, 24%
2019-06-20 16:03:31 panther prairiedog[21362] DEBUG 1300000/4927080, 26%
2019-06-20 16:03:40 panther prairiedog[21362] DEBUG 1400000/4927080, 28%
2019-06-20 16:03:49 panther prairiedog[21362] DEBUG 1500000/4927080, 30%
2019-06-20 16:03:58 panther prairiedog[21362] DEBUG 1600000/4927080, 32%
2019-06-20 16:04:07 panther prairiedog[21362] DEBUG 1700000/4927080, 34%
2019-06-20 16:04:16 panther prairiedog[21362] DEBUG 1800000/4927080, 36%
2019-06-20 16:04:26 panther prairiedog[21362] DEBUG 1900000/4927080, 38%
2019-06-20 16:04:36 panther prairiedog[21362] DEBUG 2000000/4927080, 40%
2019-06-20 16:04:46 panther prairiedog[21362] DEBUG 2100000/4927080, 42%
2019-06-20 16:04:55 panther prairiedog[21362] DEBUG 2200000/4927080, 44%
2019-06-20 16:05:04 panther prairiedog[21362] DEBUG 2300000/4927080, 46%
2019-06-20 16:05:14 panther prairiedog[21362] DEBUG 2400000/4927080, 48%
2019-06-20 16:05:23 panther prairiedog[21362] DEBUG 2500000/4927080, 50%
2019-06-20 16:05:33 panther prairiedog[21362] DEBUG 2600000/4927080, 52%
2019-06-20 16:05:43 panther prairiedog[21362] DEBUG 2700000/4927080, 54%
2019-06-20 16:05:52 panther prairiedog[21362] DEBUG 2800000/4927080, 56%
2019-06-20 16:06:02 panther prairiedog[21362] DEBUG 2900000/4927080, 58%
2019-06-20 16:06:11 panther prairiedog[21362] DEBUG 3000000/4927080, 60%
2019-06-20 16:06:21 panther prairiedog[21362] DEBUG 3100000/4927080, 62%
2019-06-20 16:06:31 panther prairiedog[21362] DEBUG 3200000/4927080, 64%
2019-06-20 16:06:40 panther prairiedog[21362] DEBUG 3300000/4927080, 66%
2019-06-20 16:06:50 panther prairiedog[21362] DEBUG 3400000/4927080, 69%
err: MDB_MAP_FULL: Environment mapsize limit reached
python3: lib/lemongraph.c:529: _log_append: Assertion `DB_SUCCESS == r' failed.
Aborted
Shutting down, this might take some time.
2019-06-20 16:06:54 panther snakemake.logging[20843] WARNING Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
2019-06-20 16:06:54 panther snakemake.logging[20843] ERROR Exiting because a job execution failed. Look above for error message
Complete log: /home/kevin/prairiedog/.snakemake/log/2019-06-20T154543.916792.snakemake.log
2019-06-20 16:06:54 panther snakemake.logging[20843] WARNING Complete log: /home/kevin/prairiedog/.snakemake/log/2019-06-20T154543.916792.snakemake.log
kevinkle commented 5 years ago

this is with map-size set to ret = LemonGraph.lib.graph_set_mapsize(self.g._graph, (4 << 30) * 10). will bump more

2019-06-21 20:58:55 panther prairiedog[8558] DEBUG 4400000/4485888, 98%
2019-06-21 20:59:07 panther prairiedog[8558] DEBUG Done graphing SRR3665176.fasta, covering 4485888 kmers in 596.4769449234009 s
Current graph size is 39.28785705566406 GB
rule 'pangenome' on Kmer 23 / 100
2019-06-21 20:59:13 panther prairiedog[8558] INFO Indexing SRR3933051.fasta ...
2019-06-21 20:59:13 panther prairiedog[8558] INFO Done indexing SRR3933051.fasta
2019-06-21 20:59:13 panther prairiedog[8558] DEBUG Starting to graph SRR3933051.fasta in pid 8558
2019-06-21 20:59:24 panther prairiedog[8558] DEBUG 100000/5445017, 1%
2019-06-21 20:59:35 panther prairiedog[8558] DEBUG 200000/5445017, 3%
2019-06-21 20:59:47 panther prairiedog[8558] DEBUG 300000/5445017, 5%
2019-06-21 20:59:59 panther prairiedog[8558] DEBUG 400000/5445017, 7%
2019-06-21 21:00:12 panther prairiedog[8558] DEBUG 500000/5445017, 9%
2019-06-21 21:00:24 panther prairiedog[8558] DEBUG 600000/5445017, 11%
2019-06-21 21:00:37 panther prairiedog[8558] DEBUG 700000/5445017, 12%
2019-06-21 21:00:50 panther prairiedog[8558] DEBUG 800000/5445017, 14%
2019-06-21 21:01:03 panther prairiedog[8558] DEBUG 900000/5445017, 16%
2019-06-21 21:01:15 panther prairiedog[8558] DEBUG 1000000/5445017, 18%
2019-06-21 21:01:28 panther prairiedog[8558] DEBUG 1100000/5445017, 20%
2019-06-21 21:01:41 panther prairiedog[8558] DEBUG 1200000/5445017, 22%
2019-06-21 21:01:54 panther prairiedog[8558] DEBUG 1300000/5445017, 23%
2019-06-21 21:02:07 panther prairiedog[8558] DEBUG 1400000/5445017, 25%
2019-06-21 21:02:20 panther prairiedog[8558] DEBUG 1500000/5445017, 27%
2019-06-21 21:02:33 panther prairiedog[8558] DEBUG 1600000/5445017, 29%
2019-06-21 21:02:46 panther prairiedog[8558] DEBUG 1700000/5445017, 31%
2019-06-21 21:03:00 panther prairiedog[8558] DEBUG 1800000/5445017, 33%
2019-06-21 21:03:13 panther prairiedog[8558] DEBUG 1900000/5445017, 34%
2019-06-21 21:03:26 panther prairiedog[8558] DEBUG 2000000/5445017, 36%
2019-06-21 21:03:39 panther prairiedog[8558] DEBUG 2100000/5445017, 38%
2019-06-21 21:03:53 panther prairiedog[8558] DEBUG 2200000/5445017, 40%
2019-06-21 21:04:07 panther prairiedog[8558] DEBUG 2300000/5445017, 42%
2019-06-21 21:04:21 panther prairiedog[8558] DEBUG 2400000/5445017, 44%
2019-06-21 21:04:35 panther prairiedog[8558] DEBUG 2500000/5445017, 45%
2019-06-21 21:04:48 panther prairiedog[8558] DEBUG 2600000/5445017, 47%
2019-06-21 21:05:01 panther prairiedog[8558] DEBUG 2700000/5445017, 49%
2019-06-21 21:05:16 panther prairiedog[8558] DEBUG 2800000/5445017, 51%
2019-06-21 21:05:30 panther prairiedog[8558] DEBUG 2900000/5445017, 53%
2019-06-21 21:05:44 panther prairiedog[8558] DEBUG 3000000/5445017, 55%
2019-06-21 21:05:57 panther prairiedog[8558] DEBUG 3100000/5445017, 56%
2019-06-21 21:06:11 panther prairiedog[8558] DEBUG 3200000/5445017, 58%
2019-06-21 21:06:25 panther prairiedog[8558] DEBUG 3300000/5445017, 60%
2019-06-21 21:06:39 panther prairiedog[8558] DEBUG 3400000/5445017, 62%
2019-06-21 21:06:52 panther prairiedog[8558] DEBUG 3500000/5445017, 64%
2019-06-21 21:07:06 panther prairiedog[8558] DEBUG 3600000/5445017, 66%
2019-06-21 21:07:20 panther prairiedog[8558] DEBUG 3700000/5445017, 67%
2019-06-21 21:07:34 panther prairiedog[8558] DEBUG 3800000/5445017, 69%
2019-06-21 21:07:49 panther prairiedog[8558] DEBUG 3900000/5445017, 71%
2019-06-21 21:08:03 panther prairiedog[8558] DEBUG 4000000/5445017, 73%
2019-06-21 21:08:17 panther prairiedog[8558] DEBUG 4100000/5445017, 75%
2019-06-21 21:08:31 panther prairiedog[8558] DEBUG 4200000/5445017, 77%
2019-06-21 21:08:46 panther prairiedog[8558] DEBUG 4300000/5445017, 78%
python3: lib/lemongraph.c:622: _edge_index: Assertion `DB_SUCCESS == r' failed.
Aborted
Shutting down, this might take some time.
2019-06-21 21:08:59 panther snakemake.logging[3368] WARNING Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
2019-06-21 21:08:59 panther snakemake.logging[3368] ERROR Exiting because a job execution failed. Look above for error message
Complete log: /home/kevin/prairiedog/.snakemake/log/2019-06-21T155438.199857.snakemake.log
2019-06-21 21:08:59 panther snakemake.logging[3368] WARNING Complete log: /home/kevin/prairiedog/.snakemake/log/2019-06-21T155438.199857.snakemake.log