skoren
commented
11 months ago What's the full report (or is that it you posted above)? What's the contents of the assembly folder and the canu.out file there?
Closed emilyjunkins closed 11 months ago
skoren
commented
11 months ago What's the full report (or is that it you posted above)? What's the contents of the assembly folder and the canu.out file there?
emilyjunkins
commented
11 months ago That is the full report.
canu.out:
Found perl:
/usr/bin/perl
This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux-thread-multi
Found java:
/usr/bin/java
openjdk version "1.8.0_171"
Found canu:
/home/ejunkins/canu-2.2/bin/canu
canu 2.2
-- canu 2.2
--
-- CITATIONS
--
-- For assemblies of PacBio HiFi reads:
-- Nurk S, Walenz BP, Rhiea A, Vollger MR, Logsdon GA, Grothe R, Miga KH, Eichler EE, Phillippy AM, Koren S.
-- HiCanu: accurate assembly of segmental duplications, satellites, and allelic variants from high-fidelity long reads.
-- biorXiv. 2020.
-- https://doi.org/10.1101/2020.03.14.992248
--
-- Read and contig alignments during correction and consensus use:
-- Šošic M, Šikic M.
-- Edlib: a C/C ++ library for fast, exact sequence alignment using edit distance.
-- Bioinformatics. 2017 May 1;33(9):1394-1395.
-- http://doi.org/10.1093/bioinformatics/btw753
--
-- Overlaps are generated using:
-- Berlin K, et al.
-- Assembling large genomes with single-molecule sequencing and locality-sensitive hashing.
-- Nat Biotechnol. 2015 Jun;33(6):623-30.
-- http://doi.org/10.1038/nbt.3238
--
-- Myers EW, et al.
-- A Whole-Genome Assembly of Drosophila.
-- Science. 2000 Mar 24;287(5461):2196-204.
-- http://doi.org/10.1126/science.287.5461.2196
--
-- Contig consensus sequences are generated using an algorithm derived from pbdagcon:
-- Chin CS, et al.
-- Nonhybrid, finished microbial genome assemblies from long-read SMRT sequencing data.
-- Nat Methods. 2013 Jun;10(6):563-9
-- http://doi.org/10.1038/nmeth.2474
--
-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '1.8.0_171' (from 'java') with -d64 support.
--
-- WARNING:
-- WARNING: Failed to run gnuplot using command 'gnuplot'.
-- WARNING: Plots will be disabled.
-- WARNING:
--
--
-- Detected 1 CPUs and 1410000 gigabytes of memory on the local machine.
--
-- Detected Slurm with 'sinfo' binary in /usr/bin/sinfo.
-- Detected Slurm with task IDs up to 1000 allowed.
--
-- Slurm support detected. Resources available:
-- 11 hosts with 24 cores and 187 GB memory.
-- 4 hosts with 40 cores and 1376 GB memory.
-- 67 hosts with 40 cores and 187 GB memory.
--
-- (tag)Threads
-- (tag)Memory |
-- (tag) | | algorithm
-- ------- ---------- -------- -----------------------------
-- Grid: meryl 12.000 GB 4 CPUs (k-mer counting)
-- Grid: hap 8.000 GB 4 CPUs (read-to-haplotype assignment)
-- Grid: cormhap 6.000 GB 8 CPUs (overlap detection with mhap)
-- Grid: obtovl 4.000 GB 8 CPUs (overlap detection)
-- Grid: utgovl 4.000 GB 8 CPUs (overlap detection)
-- Grid: cor -.--- GB 4 CPUs (read correction)
-- Grid: ovb 4.000 GB 1 CPU (overlap store bucketizer)
-- Grid: ovs 8.000 GB 1 CPU (overlap store sorting)
-- Grid: red 16.000 GB 4 CPUs (read error detection)
-- Grid: oea 8.000 GB 1 CPU (overlap error adjustment)
-- Grid: bat 16.000 GB 4 CPUs (contig construction with bogart)
-- Grid: cns -.--- GB 4 CPUs (consensus)
--
-- Found PacBio HiFi reads in 'LS01_001.seqStore':
-- Libraries:
-- PacBio HiFi: 1
-- Reads:
-- Corrected: 15775775005
--
--
-- Generating assembly 'LS01_001' in '/home/ejunkins/LS01_001_comparemethods/canu_out5':
-- genomeSize:
-- 5000000
--
-- Overlap Generation Limits:
-- corOvlErrorRate 0.0000 ( 0.00%)
-- obtOvlErrorRate 0.0250 ( 2.50%)
-- utgOvlErrorRate 0.0100 ( 1.00%)
--
-- Overlap Processing Limits:
-- corErrorRate 0.0000 ( 0.00%)
-- obtErrorRate 0.0250 ( 2.50%)
-- utgErrorRate 0.0003 ( 0.03%)
-- cnsErrorRate 0.0500 ( 5.00%)
--
-- Stages to run:
-- trim corrected reads.
-- assemble corrected and trimmed reads.
--
--
-- Correction skipped; not enabled.
--
-- BEGIN TRIMMING
-- Meryl finished successfully. Kmer frequency histogram:
--
-- WARNING: gnuplot failed.
--
----------------------------------------
--
-- 22-mers Fraction
-- Occurrences NumMers Unique Total
-- 1- 1 0 0.0000 0.0000
-- 2- 2 386913834 ********************************************************************** 0.4276 0.0771
-- 3- 7 376482552 ******************************************************************** 0.6139 0.1274
-- 8- 16 83264287 *************** 0.8648 0.2459
-- 17- 29 20412775 *** 0.9391 0.3245
-- 30- 46 9941945 * 0.9590 0.3645
-- 47- 67 6078892 * 0.9697 0.4017
-- 68- 92 3992616 0.9763 0.4359
-- 93- 121 3393714 0.9805 0.4653
-- 122- 154 4526761 0.9843 0.5038
-- 155- 191 495318 0.9892 0.5628
-- 192- 232 199281 0.9897 0.5705
-- 233- 277 682960 0.9899 0.5747
-- 278- 326 1902755 0.9907 0.5933
-- 327- 379 1364404 0.9928 0.6504
-- 380- 436 1543101 0.9943 0.6980
-- 437- 497 2825750 0.9960 0.7624
-- 498- 562 365296 0.9991 0.8921
-- 563- 631 60892 0.9995 0.9098
-- 632- 704 38268 0.9996 0.9134
-- 705- 781 37126 0.9996 0.9159
-- 782- 862 49369 0.9996 0.9187
-- 863- 947 48639 0.9997 0.9228
-- 948- 1036 20167 0.9998 0.9271
-- 1037- 1129 13975 0.9998 0.9290
-- 1130- 1226 12953 0.9998 0.9306
-- 1227- 1327 14918 0.9998 0.9321
-- 1328- 1432 10563 0.9998 0.9340
-- 1433- 1541 8414 0.9998 0.9354
-- 1542- 1654 4439 0.9998 0.9367
-- 1655- 1771 7415 0.9998 0.9374
-- 1772- 1892 6160 0.9999 0.9386
-- 1893- 2017 5893 0.9999 0.9397
-- 2018- 2146 7953 0.9999 0.9409
-- 2147- 2279 5025 0.9999 0.9425
-- 2280- 2416 4032 0.9999 0.9437
-- 2417- 2557 4772 0.9999 0.9446
-- 2558- 2702 3419 0.9999 0.9458
-- 2703- 2851 5382 0.9999 0.9467
-- 2852- 3004 4859 0.9999 0.9482
-- 3005- 3161 3349 0.9999 0.9496
--
-- 0 (max occurrences)
-- 10039868242 (total mers, non-unique)
-- 904843411 (distinct mers, non-unique)
-- 0 (unique mers)
-- Finished stage 'meryl-process', reset canuIteration.
--
-- Removing meryl database 'trimming/0-mercounts/LS01_001.ms22'.
--
-- OVERLAPPER (normal) (trimming) erate=0.025
--
----------------------------------------
-- Starting command on Wed Sep 27 13:53:52 2023 with 69471.029 GB free disk space
cd trimming/1-overlapper
/home/ejunkins/canu-2.2/bin/overlapInCorePartition \
-S ../../LS01_001.seqStore \
-hl 80000000 \
-rl 1000000000 \
-ol 500 \
-o ./LS01_001.partition \
> ./LS01_001.partition.err 2>&1
-- Finished on Wed Sep 27 13:53:52 2023 (lickety-split) with 69471.029 GB free disk space
----------------------------------------
--
-- Configured 918 overlapInCore jobs.
-- Finished stage 'obt-overlapConfigure', reset canuIteration.
--
-- Running jobs. First attempt out of 2.
--
-- 'overlap.jobSubmit-01.sh' -> job 3239876 tasks 1-100.
-- 'overlap.jobSubmit-02.sh' -> job 3239877 tasks 101-200.
-- Failed to submit compute jobs. Delay 10 seconds and try again.
CRASH:
CRASH: canu 2.2
CRASH: Please panic, this is abnormal.
CRASH:
CRASH: Failed to submit compute jobs.
CRASH:
CRASH: Failed at /home/ejunkins/canu-2.2/bin/../lib/site_perl/canu/Execution.pm line 1259.
CRASH: canu::Execution::submitOrRunParallelJob('LS01_001', 'obtovl', 'trimming/1-overlapper', 'overlap', 1, 2, 3, 4, 5, ...) called at /home/ejunkins/canu-2.2/bin/../lib/site_perl/canu/OverlapInCore.pm line 394
CRASH: canu::OverlapInCore::overlapCheck('LS01_001', 'obt', 'partial') called at /home/ejunkins/canu-2.2/bin/canu line 1012
CRASH: main::overlap('LS01_001', 'obt') called at /home/ejunkins/canu-2.2/bin/canu line 1103
CRASH:
CRASH: Last 50 lines of the relevant log file (trimming/1-overlapper/overlap.jobSubmit-03.out):
CRASH:
CRASH: sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)
CRASH:Seems like a cluster issue but previous use of the gridEngine flag seemed to work.
This is my output in the assembly folder:
canu-logs canu.out canu-scripts LS01_001.report LS01_001.seqStore LS01_001.seqStore.err LS01_001.seqStore.sh trimming
skoren
commented
11 months ago This looks like an issue with your grid system not allowing the job submission. I'm guessing there's a limit on the number of total jobs you're allowed to submit, see issues #1752 and #1883 for more details on these limits. If this limit cannot be increased, you'll have to follow the suggestion in #1883 and use useGrid=remote and manually run the maximum jobs you're allowed at a time.
skoren
commented
11 months ago Idle, incompatible grid setup.
Hello, I have HiFi metagenomic reads from a low complexity sample. I first tried running canu but very few reads were retained, I ran again following suggestions from this issue and was able to get all reads read in but now I have no contigs.
I am running canu 2.2 on Linux HPC system. I have also tried without the
-untrimmedoption get get the same results.canu -p LS01_001 -d /home/ejunkins/LS01_001_comparemethods/canu_out5 genomeSize=5m gridEngineArrayMaxJobs=100 maxInputCoverage=100000 -untrimmed -pacbio-hifi /home/ejunkins/LS01_001_comparemethods/pbio-2668.26240.bc1021_BAK8B_OA--bc1021_BAK8B_OA.ccs.filter.fastq_copy.gzThis is the report for the above run: