Closed psur9757 closed 5 years ago
brianwalenz
commented
5 years ago What's in meryl-configure.err?
I'm guessing it will be complaining about failing to load some compiler library, which is usually caused by compiling on a machine that differs from the ones you're trying to run on (different OS version, or compiler not installed on the nodes, etc).
psur9757
commented
5 years ago meryl-configure.err
Found perl:
/usr/bin/perl
Found java:
/usr/bin/java
openjdk version "1.8.0_151"
Found canu:
/usr/local/canu/1.8/bin/canu
Use of implicit split to @_ is deprecated at /usr/local/canu/1.8/bin/../lib/site_perl/canu/Grid_Cloud.pm line 73.
Canu 1.8Well, that wasn't very illuminating. :-(
That's the output from meryl-configure.sh. This has a series of 'meryl' commands that are attempting to decide how much data to process at one time. Can you run any of those commands by hand, on both the head node and a compute node? Best to leave out the file redirection, just to see any errors.
psur9757
commented
5 years ago No errors I ran a couple commands and they all ran well. I removed threads parameter since I was running on the local node.
$ /usr/local/canu/1.8/bin/meryl -C k=16 count segment=1/01 ../../OSR409.seqStore
Counting 7825 million canonical 16-mers from 1 input file:
canu-seqStore: ../../OSR409.seqStore
SIMPLE MODE
-----------
16-mers
-> 4294967296 entries for counts up to 65535.
-> 64 Gbits memory used
8205743089 input bases
-> expected max count of 32822972, needing 10 extra bits.
-> 40 Gbits memory used
13 GB memory needed
COMPLEX MODE
------------
prefix # of struct kmers/ segs/ data total
bits prefix memory prefix prefix memory memory
------ ------- ------- ------- ------- ------- -------
1 2 P 3792 kB 3912 MM 237 kS 29 GB 29 GB
2 4 P 3670 kB 1956 MM 114 kS 28 GB 28 GB
3 8 P 3548 kB 978 MM 55 kS 27 GB 27 GB
4 16 P 3427 kB 489 MM 26 kS 26 GB 26 GB
5 32 P 3306 kB 244 MM 12 kS 25 GB 25 GB
6 64 P 3187 kB 122 MM 6362 S 24 GB 24 GB
7 128 P 3071 kB 61 MM 3059 S 23 GB 23 GB
8 256 P 2962 kB 30 MM 1469 S 22 GB 22 GB
9 512 P 2864 kB 15 MM 704 S 21 GB 21 GB
10 1024 P 2792 kB 7825 kM 337 S 21 GB 21 GB
11 2048 P 2768 kB 3912 kM 161 S 20 GB 20 GB
12 4096 P 2848 kB 1956 kM 77 S 19 GB 19 GB
13 8192 P 3136 kB 978 kM 37 S 18 GB 18 GB
14 16 kP 3840 kB 489 kM 18 S 17 GB 17 GB
15 32 kP 5376 kB 244 kM 9 S 17 GB 17 GB
16 64 kP 8192 kB 122 kM 4 S 15 GB 16 GB Best Value!
17 128 kP 14 MB 61 kM 2 S 15 GB 16 GB
18 256 kP 26 MB 30 kM 1 S 15 GB 16 GB
19 512 kP 52 MB 15 kM 1 S 31 GB 32 GB
20 1024 kP 104 MB 7826 M 1 S 63 GB 64 GB
FINAL CONFIGURATION
-------------------
Configured simple mode for 13.000 GB memory per batch, and up to 1 batch.
Bye.Oh, sorry! I didn't notice this is PBS. There were some flaws with PBS support in the 1.8 release. Can you install the latest non-release version from github? It would be best to remove the current assembly directory before restarting - all it's done so far is copy the input reads to the assembly database.
If you can't (or don't want to) install the new version, this assembly should be small enough to run without grid support, using just one node. Canu options useGrid=0 maxThreads=24 maxMemory=96g will disable grid support, limit it to 24 CPUs and 96 GB memory. Allocating an entire node would be best, in which case you can omit the maxThreads and maxMemory options.
psur9757
commented
5 years ago I updated Canu and restarted the process. This time the error is with submitting overlap jobs.
canu.out
Found perl:
/usr/bin/perl
Found java:
/usr/bin/java
openjdk version "1.8.0_151"
Found canu:
/usr/local/canu/1.8+268/bin/canu
Canu snapshot v1.8 +268 changes (r9478 b00a9c2f8c17b2273f1123eed3013a50252dc5b8)
-- Canu snapshot v1.8 +268 changes (r9478 b00a9c2f8c17b2273f1123eed3013a50252dc5b8)
--
-- CITATIONS
--
...
--
-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '1.8.0_151' (from 'java') with -d64 support.
--
-- WARNING:
-- WARNING: Failed to run gnuplot using command '/usr/local/gnuplot/5.0.5/bin/gnuplot'.
-- WARNING: Plots will be disabled.
-- WARNING:
--
-- Detected 24 CPUs and 126 gigabytes of memory.
-- Detected PBSPro 'PBSPro_13.1.0.160576' with 'pbsnodes' binary in /usr/local/pbs/default/bin/pbsnodes.
-- Detecting PBSPro resources.
--
-- Found 49 hosts with 48 cores and 185 GB memory under PBSPro control.
-- Found 27 hosts with 36 cores and 187 GB memory under PBSPro control.
-- Found 80 hosts with 32 cores and 123 GB memory under PBSPro control.
-- Found 2 hosts with 24 cores and 502 GB memory under PBSPro control.
-- Found 58 hosts with 24 cores and 123 GB memory under PBSPro control.
-- Found 2 hosts with 24 cores and 250 GB memory under PBSPro control.
-- Found 3 hosts with 64 cores and 6057 GB memory under PBSPro control.
--
-- (tag)Threads
-- (tag)Memory |
-- (tag) | | algorithm
-- ------- ------ -------- -----------------------------
-- Grid: meryl 12 GB 4 CPUs (k-mer counting)
-- Grid: hap 8 GB 4 CPUs (read-to-haplotype assignment)
-- Grid: cormhap 13 GB 4 CPUs (overlap detection with mhap)
-- Grid: obtovl 8 GB 4 CPUs (overlap detection)
-- Grid: utgovl 8 GB 4 CPUs (overlap detection)
-- Grid: cor 16 GB 4 CPUs (read correction)
-- Grid: ovb 4 GB 1 CPU (overlap store bucketizer)
-- Grid: ovs 8 GB 1 CPU (overlap store sorting)
-- Grid: red 10 GB 4 CPUs (read error detection)
-- Grid: oea 4 GB 1 CPU (overlap error adjustment)
-- Grid: bat 64 GB 8 CPUs (contig construction with bogart)
-- Grid: cns --- GB 8 CPUs (consensus)
-- Grid: gfa 16 GB 8 CPUs (GFA alignment and processing)
--
-- In 'OSR409.seqStore', found PacBio reads:
-- Raw: 717541
-- Corrected: 452118
-- Trimmed: 0
--
-- Generating assembly 'OSR409' in '/scratch/RDS-FAE-OSR-RW/OSR409/results/assembly/canu'
--
-- Parameters:
--
-- genomeSize 90000000
--
-- Overlap Generation Limits:
-- corOvlErrorRate 0.2400 ( 24.00%)
-- obtOvlErrorRate 0.0450 ( 4.50%)
-- utgOvlErrorRate 0.0450 ( 4.50%)
--
-- Overlap Processing Limits:
-- corErrorRate 0.3000 ( 30.00%)
-- obtErrorRate 0.0450 ( 4.50%)
-- utgErrorRate 0.0450 ( 4.50%)
-- cnsErrorRate 0.0750 ( 7.50%)
--
--
-- BEGIN TRIMMING
--
-- Meryl finished successfully. Kmer frequency histogram:
--
-- WARNING: gnuplot failed.
--
----------------------------------------
--
-- 22-mers Fraction
-- Occurrences NumMers Unique Total
-- 1- 1 0 0.0000 0.0000
-- 2- 2 31711297 ********************************************************************** 0.2369 0.0127
-- 3- 4 17062027 ************************************* 0.3220 0.0195
-- 5- 7 7067797 *************** 0.3893 0.0274
-- 8- 11 3562696 ******* 0.4258 0.0341
-- 12- 16 3250439 ******* 0.4486 0.0404
-- 17- 22 5583516 ************ 0.4737 0.0505
-- 23- 29 8548392 ****************** 0.5184 0.0754
-- 30- 37 7648771 **************** 0.5823 0.1216
-- 38- 46 7072174 *************** 0.6362 0.1709
-- 47- 56 13869009 ****************************** 0.6914 0.2351
-- 57- 67 15777451 ********************************** 0.7994 0.3880
-- 68- 79 6170512 ************* 0.9121 0.5767
-- 80- 92 1676089 *** 0.9526 0.6559
-- 93- 106 889656 * 0.9642 0.6827
-- 107- 121 767600 * 0.9707 0.7000
-- 122- 137 624811 * 0.9764 0.7176
-- 138- 154 405393 0.9809 0.7333
-- 155- 172 306459 0.9839 0.7449
-- 173- 191 244282 0.9861 0.7548
-- 192- 211 208108 0.9879 0.7636
-- 212- 232 165733 0.9895 0.7719
-- 233- 254 129966 0.9907 0.7792
-- 255- 277 109992 0.9916 0.7855
-- 278- 301 89973 0.9925 0.7913
-- 302- 326 84644 0.9931 0.7965
-- 327- 352 74243 0.9938 0.8018
-- 353- 379 63079 0.9943 0.8068
-- 380- 407 55517 0.9948 0.8114
-- 408- 436 47618 0.9952 0.8158
-- 437- 466 42148 0.9955 0.8198
-- 467- 497 37196 0.9958 0.8236
-- 498- 529 33325 0.9961 0.8272
-- 530- 562 29072 0.9964 0.8306
-- 563- 596 25431 0.9966 0.8337
-- 597- 631 23576 0.9968 0.8367
-- 632- 667 20832 0.9970 0.8396
-- 668- 704 20130 0.9971 0.8423
-- 705- 742 19076 0.9973 0.8450
-- 743- 781 17609 0.9974 0.8478
-- 782- 821 16643 0.9975 0.8505
--
-- 0 (max occurrences)
-- 4994801169 (total mers, non-unique)
-- 133866243 (distinct mers, non-unique)
-- 0 (unique mers)
-- Finished stage 'meryl-process', reset canuIteration.
--
-- Removing meryl database 'trimming/0-mercounts/OSR409.ms22'.
--
-- OVERLAPPER (normal) (trimming) erate=0.045
--
----------------------------------------
-- Starting command on Tue Jul 2 01:11:48 2019 with 55786.535 GB free disk space
cd trimming/1-overlapper
/usr/local/canu/1.8+268/bin/overlapInCorePartition \
-S ../../OSR409.seqStore \
-hl 160000000 \
-rl 5000000000 \
-ol 500 \
-o ./OSR409.partition \
> ./OSR409.partition.err 2>&1
-- Finished on Tue Jul 2 01:11:48 2019 (in the blink of an eye) with 55786.535 GB free disk space
----------------------------------------
--
-- Configured 35 overlapInCore jobs.
-- Finished stage 'obt-overlapConfigure', reset canuIteration.
--
-- Running jobs. First attempt out of 2.
--
-- Failed to submit compute jobs. Delay 10 seconds and try again.
CRASH:
CRASH: Canu snapshot v1.8 +268 changes (r9478 b00a9c2f8c17b2273f1123eed3013a50252dc5b8)
CRASH: Please panic, this is abnormal.
ABORT:
CRASH: Failed to submit compute jobs.
CRASH:
CRASH: Failed at /usr/local/canu/1.8+268/bin/../lib/site_perl/canu/Execution.pm line 1241
CRASH: canu::Execution::submitOrRunParallelJob('OSR409', 'obtovl', 'trimming/1-overlapper', 'overlap', 1, 2, 3, 4, 5, ...) called at /usr/local/canu/1.8+268
/bin/../lib/site_perl/canu/OverlapInCore.pm line 414
CRASH: canu::OverlapInCore::overlapCheck('OSR409', 'obt', 'partial') called at /usr/local/canu/1.8+268/bin/canu line 644
CRASH: main::overlap('OSR409', 'obt') called at /usr/local/canu/1.8+268/bin/canu line 895
CRASH:
CRASH: Last 50 lines of the relevant log file (trimming/1-overlapper/overlap.jobSubmit-01.out):
CRASH:
CRASH: qsub: Cannot be used with select or place: nodes
CRASH:trimming/1-overlapper/overlap.jobSubmit-01.out
qsub: Cannot be used with select or place: nodesAssuming you're using the same command as before, remove your nodes/ppn settings from all the gridOptions. Canu will set these itself.
gridOptionsOBTOVL="-l walltime=156:00:00 -l nodes=1:ppn=8"See the docs for option gridEngineResourceOption if you want to modify the format. The default for PBSpro is -l select=1:ncpus=THREADS:mem=MEMORY. What happened here is that both the older node/ppn format and newer select/ncpus options were set, and PBS doesn't like that.
Further, if all you're setting is walltime, you can use the global gridOptions= to set it for all components.
Last, once you've found a set of setting that works, you can put all these grid/system options into ~/.canu or $bin/canu.defaults (one per line) to set them for all future runs.
psur9757
commented
5 years ago Thank you this issue is solved. My assembly ran to completion.
focusjava
commented
5 years ago How is your problem solved? I have this problem, too @psur9757
skoren
commented
5 years ago @focusjava are you using PBS? If so, there were issues with PBS support in the 1.8 release, using the 1.9 branch should fix them. If you're not on PBS then the error is likely due to a different cause and you should open a new issue describing it in detail and posting the same logs as above (e.g. meryl outputs).
Hello Canu Team,
I am trying to run Canu v1.8 and got an error that it
Any help is much appreciated. Thank you.
Best Regards, Priyanka
PBS Script
Error Output
Output directory