search

marbl / canu

A single molecule sequence assembler for genomes large and small.
http://canu.readthedocs.io/
658 stars 179 forks source link

GFA alignment error #1545

Closed Giacomoggioli4111 closed 4 years ago

Giacomoggioli4111 commented 5 years ago

Hello, I am using Canu 1.8 on PacBio data with 110 coverage with the following settings:

canu \
 -p Oasisia -d /data/scratch/btx604/Oasisia/canu_V8 \
 -pacbio-raw /data/SBCS-MartinDuranLab/03-Giacomo/raw_data/PacBio/01-Oasisia/Oasisia_pb_raw.fastq.gz \
 genomeSize=1g \
 useGrid=true \
 gridEngineMemoryOption="-l h_vmem=MEMORY" \
 gridEngineThreadsOption="-pe smp THREADS" \
 gridOptions="-l h_rt=24:0:0 -j y -l highmem" \
 "corOutCoverage=200" \
 "correctedErrorRate=0.040" \
 "batOptions=-dg 3 -db 3 -dr 1 -ca 500 -cp 50" \
 "gridOptionsGFA=-l h_vmem=300G" \
 "gridOptionsovb=-l h_vmem=120G" \
 "gridOptionsovs=-l h_vmem=120G" \
 "gridOptionscorovl=-l h_vmem=30G" \
 "gridOptionscor=-l h_vmem=30G" \
 "gridOptionscns=l h_vmem=40G" \
 gnuplot=$(which gnuplot) \
 java=$(which java)

I am getting this error from consensus.197.out:

generateTemplateStitch()-- significant size difference, stopping.
utgcns: utgcns/libcns/unitigConsensus.C:557: char* generateTemplateStitch(abAbacus*, tgPosition*, uint32, double, bool): Assertion `(tiglen < 100000) || ((-50.0 <= pd) && (pd <= 50.0))' failed.
                           ----------CONTAINED READS----------  -DOVETAIL  READS-
  tigID    length   reads      used coverage  ignored coverage      used coverage
------- --------- -------  -------- -------- -------- --------  -------- --------
   1137   3074151    5294      4117   34.71x      903    1.00x       274    5.29x
   1138     35945       8         7    5.07x        0    0.00x         1    1.00x
   1139   1022203    1531      1451   30.59x        0    0.00x        80    4.74x
   1140    522500     614       575   25.94x        0    0.00x        39    4.68x
   1141     64539      67        62   17.88x        0    0.00x         5    3.74x
   1142   1882180    3018      2853   34.42x        0    0.00x       165    5.39x
   1143     97949     267        98   36.49x      163   22.39x         6    3.63x
   1144   1054998    1952      1061   34.57x      801    7.93x        90    5.44x
   1145    366913     174       150    9.23x        0    0.00x        24    2.96x
   1146     83535       3         2    0.77x        0    0.00x         1    1.00x
   1147    575006     860       642   35.04x      174    1.52x        44    4.98x
   1148   1269910    2053      1903   34.23x       26    0.03x       124    5.77x
   1149     67764      23        21   10.86x        0    0.00x         2    1.87x

Failed with 'Aborted'; backtrace (libbacktrace):
utility/system-stackTrace.C::89 in _Z17AS_UTL_catchCrashiP7siginfoPv()
(null)::0 in (null)()
(null)::0 in (null)()
(null)::0 in (null)()
(null)::0 in (null)()
(null)::0 in (null)()
utgcns/libcns/unitigConsensus.C::557 in _Z22generateTemplateStitchP8abAbacusP10tgPositionjdb()
utgcns/libcns/unitigConsensus.C::786 in _ZN15unitigConsensus13generatePBDAGEP5tgTigcPSt3mapIjP6sqReadSt4lessIjESaISt4pairIKjS4_EEEPS2_IjP10sqReadDataS6_SaIS7_IS8_SE_EEE()
utgcns/utgcns.C::522 in main()
(null)::0 in (null)()
(null)::0 in (null)()
/opt/sge_spool/8.5.4/sm3/job_scripts/863077: line 100: 44812 Aborted                 $bin/utgcns -S ../Oasisia.${tag}Store/partitionedReads.seqStore -T ../Oasisia.${tag}Store 1 $jobid -O ./${tag}cns/$jobid.cns.WORKING -maxcoverage 40 -e 0.025 -pbdagcon -edlib -threads 2-8

I have already changed -pbdagcon to -norealign and set -e 0.4 at the bottom of consensus.sh, like you suggested in previous issues. Still it's not working. Is there something else I can try to do?

Thank you

skoren commented 5 years ago

Norealign doesn't actually exist in 1.8 so I think you might have a version between 1.8 and 1.9. There is also something strange with the command for canu you posted and the consensus script, it says you are using corrected error rate of 4% but consensus is using 2.5%. Consensus should be set to at least corrected error rate, not lower.

When you said you tried -e 0.4 did you then run the command by hand or re-start Canu again? I'd suggest running it by hand on an interactive instance (sh consensus.sh 197). The log should report that it's using 40% error, I didn't see that in the log you posted so I wanted to make sure it's not still using 2.5%. If that still doesn't work you can try disabling the assert in the code to get through consensus. Another option would be to try the 1.9 release, which has several consensus fixes. I think it should run on your existing assembly but make a backup in case it does not.

Giacomoggioli4111 commented 5 years ago

I just tried to run sh consensus.sh 197 and I got this:

Found perl: /usr/bin/perl This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux-thread-multi

Found java: /share/apps/centos7/java/1.8.0_152/bin/java java version "1.8.0_152"

Found canu: /data/SBCS-MartinDuranLab/03-Giacomo/src/canu-1.8/Linux-amd64/bin/canu Canu 1.8

Running job 197 based on command line options. -- Opening seqStore '../Oasisia.utgStore/partitionedReads.seqStore' partition 50. sqStore()-- failed to open '../Oasisia.utgStore/partitionedReads.seqStore' for read-only access: store doesn't exist.

skoren commented 5 years ago

You should be running in the unitigging/5-consensus folder.

Giacomoggioli4111 commented 5 years ago

I am running it in that folder

Giacomoggioli4111 commented 5 years ago

the Oasisia.utgStore folder is in unitigging folder though

skoren commented 5 years ago

It's not complaining about the utgStore folder but the read portioning within it. Have you cleaned up/removed anything from the run? The error implies the partitioned store has been removed which Canu would not do until the consensus step was complete. It seems like this run advanced past consensus at some point.

What's the log from the last canu.out script you have? What's the contents of Oasisia.utgStore, Oasisia.utgStore/partitionedReads.seqStore, and Oasisia.utgStore/partitionedReads.seqStore/partitions? Without the partitions it's going to be a bit harder to re-run the failed job.

Giacomoggioli4111 commented 5 years ago

I haven't removed anything from the folder. This is canu.49.out:

Found perl:
   /bin/perl
   This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux-thread-multi

Found java:
   /bin/java
   openjdk version "1.8.0_222"

Found canu:
   /data/SBCS-MartinDuranLab/03-Giacomo/src/canu-1.8/Linux-amd64/bin/canu
   Canu 1.8

-- Canu 1.8
--
-- CITATIONS
--
-- Koren S, Walenz BP, Berlin K, Miller JR, Phillippy AM.
-- Canu: scalable and accurate long-read assembly via adaptive k-mer weighting and repeat separation.
-- Genome Res. 2017 May;27(5):722-736.
-- http://doi.org/10.1101/gr.215087.116
-- 
-- Koren S, Rhie A, Walenz BP, Dilthey AT, Bickhart DM, Kingan SB, Hiendleder S, Williams JL, Smith TPL, Phillippy AM.
-- De novo assembly of haplotype-resolved genomes with trio binning.
-- Nat Biotechnol. 2018
-- https//doi.org/10.1038/nbt.4277
-- 
-- Read and contig alignments during correction, consensus and GFA building use:
--   Šošic M, Šikic M.
--   Edlib: a C/C ++ library for fast, exact sequence alignment using edit distance.
--   Bioinformatics. 2017 May 1;33(9):1394-1395.
--   http://doi.org/10.1093/bioinformatics/btw753
-- 
-- Overlaps are generated using:
--   Berlin K, et al.
--   Assembling large genomes with single-molecule sequencing and locality-sensitive hashing.
--   Nat Biotechnol. 2015 Jun;33(6):623-30.
--   http://doi.org/10.1038/nbt.3238
-- 
--   Myers EW, et al.
--   A Whole-Genome Assembly of Drosophila.
--   Science. 2000 Mar 24;287(5461):2196-204.
--   http://doi.org/10.1126/science.287.5461.2196
-- 
-- Corrected read consensus sequences are generated using an algorithm derived from FALCON-sense:
--   Chin CS, et al.
--   Phased diploid genome assembly with single-molecule real-time sequencing.
--   Nat Methods. 2016 Dec;13(12):1050-1054.
--   http://doi.org/10.1038/nmeth.4035
-- 
-- Contig consensus sequences are generated using an algorithm derived from pbdagcon:
--   Chin CS, et al.
--   Nonhybrid, finished microbial genome assemblies from long-read SMRT sequencing data.
--   Nat Methods. 2013 Jun;10(6):563-9
--   http://doi.org/10.1038/nmeth.2474
-- 
-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '1.8.0_222' (from 'java') with -d64 support.
--
-- WARNING:
-- WARNING:  Failed to run gnuplot using command 'gnuplot'.
-- WARNING:  Plots will be disabled.
-- WARNING:
--
-- Detected 12 CPUs and 23 gigabytes of memory.
-- Detected Sun Grid Engine in '/opt/sge/default'.
-- User supplied Grid Engine environment '-pe smp THREADS'.
-- User supplied Grid Engine consumable '-l h_vmem=MEMORY'.
-- 
-- Found 150 hosts with  12 cores and   23 GB memory under Sun Grid Engine control.
-- Found   2 hosts with  36 cores and  755 GB memory under Sun Grid Engine control.
-- Found   1 host  with  40 cores and  755 GB memory under Sun Grid Engine control.
-- Found   2 hosts with  16 cores and   31 GB memory under Sun Grid Engine control.
-- Found  32 hosts with  16 cores and   62 GB memory under Sun Grid Engine control.
-- Found   2 hosts with  32 cores and  377 GB memory under Sun Grid Engine control.
-- Found  63 hosts with  24 cores and  188 GB memory under Sun Grid Engine control.
-- Found  46 hosts with  32 cores and  251 GB memory under Sun Grid Engine control.
-- Found  11 hosts with  48 cores and  503 GB memory under Sun Grid Engine control.
-- Found   1 host  with  32 cores and  503 GB memory under Sun Grid Engine control.
--
--                     (tag)Threads
--            (tag)Memory         |
--        (tag)         |         |  algorithm
--        -------  ------  --------  -----------------------------
-- Grid:  meryl     31 GB    8 CPUs  (k-mer counting)
-- Grid:  hap       16 GB   16 CPUs  (read-to-haplotype assignment)
-- Grid:  cormhap   23 GB   12 CPUs  (overlap detection with mhap)
-- Grid:  obtovl    16 GB   12 CPUs  (overlap detection)
-- Grid:  utgovl    16 GB   12 CPUs  (overlap detection)
-- Grid:  ovb        4 GB    1 CPU   (overlap store bucketizer)
-- Grid:  ovs       32 GB    1 CPU   (overlap store sorting)
-- Grid:  red       11 GB    6 CPUs  (read error detection)
-- Grid:  oea        4 GB    1 CPU   (overlap error adjustment)
-- Grid:  bat      256 GB   16 CPUs  (contig construction with bogart)
-- Grid:  gfa       16 GB   16 CPUs  (GFA alignment and processing)
--
-- In 'Oasisia.seqStore', found PacBio reads:
--   Raw:        5225176
--   Corrected:  3468687
--   Trimmed:    3276877
--
-- Generating assembly 'Oasisia' in '/data/scratch/btx604/Oasisia/canu_V8'
--
-- Parameters:
--
--  genomeSize        1000000000
--
--  Overlap Generation Limits:
--    corOvlErrorRate 0.2400 ( 24.00%)
--    obtOvlErrorRate 0.0450 (  4.50%)
--    utgOvlErrorRate 0.0450 (  4.50%)
--
--  Overlap Processing Limits:
--    corErrorRate    0.3000 ( 30.00%)
--    obtErrorRate    0.0450 (  4.50%)
--    utgErrorRate    0.0450 (  4.50%)
--    cnsErrorRate    0.0750 (  7.50%)
--
--
-- BEGIN ASSEMBLY
--
--
-- Graph alignment jobs failed, tried 2 times, giving up.
--

ABORT:
ABORT: Canu 1.8
ABORT: Don't panic, but a mostly harmless error occurred and Canu stopped.
ABORT: Try restarting.  If that doesn't work, ask for help.
ABORT:

And this is the content of Oasisia.utgStore:

-rw-r--r-- 1 btx604 qmul  119804572 Nov  2 17:08 seqDB.v001.dat
-rw-r--r-- 1 btx604 qmul     238580 Nov  2 17:08 seqDB.v001.tig
-rw-r--r-- 1 btx604 qmul 3541428430 Nov  4 13:58 seqDB.v002.dat
-rw-r--r-- 1 btx604 qmul     238580 Nov  4 13:58 seqDB.v002.tig

So no partitions apparently...

Giacomoggioli4111 commented 5 years ago

Should I just try to run another canu job using the option: -pacbio-corrected Oasisia.trimmedReads.fasta.gz and -assemble?

skoren commented 5 years ago

No partitions but also no consensus errors and you have v002 files which implies consensus exists. This means Canu recovered from the consensus error and generated the consensus sequences which is why you don't see partitions. I expect this was due to the change from 2.5 to 4%, I'd guess you had a run without the corrected error rate increase. There is no reason to re-run since you already have an assembly and consensus finished.

So either way nothing to fix in consensus. The last error, graph alignment jobs, may be a memory issue. There will be a log in 4-unitigging/alignGFA*, post that.

Giacomoggioli4111 commented 5 years ago

Here it is 4-unitigger/alignGFA.1.out:

Found perl:
   /bin/perl
   This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux-thread-multi

Found java:
   /bin/java
   openjdk version "1.8.0_222"

Found canu:
   /data/SBCS-MartinDuranLab/03-Giacomo/src/canu-1.8/Linux-amd64/bin/canu
   Canu 1.8

/opt/sge_spool/8.5.4/nxv40/job_scripts/863081: line 53:  9719 Segmentation fault      $bin/alignGFA -T ../Oasisia.utgStore 2 -i ./Oasisia.unitigs.gfa -o ./Oasisia.unitigs.aligned.gfa -t 16 > ./Oasisia.unitigs.aligned.gfa.err 2>&1
/opt/sge_spool/8.5.4/nxv40/job_scripts/863081: line 76: 10202 Segmentation fault      $bin/alignGFA -bed -T ../Oasisia.utgStore 2 -C ../Oasisia.ctgStore 2 -i ./Oasisia.unitigs.bed -o ./Oasisia.unitigs.aligned.bed -t 16 > ./Oasisia.unitigs.aligned.bed.err 2>&1
GFA alignments failed.
skoren commented 5 years ago

Given that these are errors with the unitig graph and you're on an older version, I think it's safe to just skip this step and continue to get your assembly. Run the following commands in the unitigging/4-unitigger folder:

ln -s asm.unitigs.aligned.gfa asm.unitigs.gfa
ln -s asm.unitigs.aligned.bed asm.unitigs.bed

and re-run the last canu command you used. This will make it think the gfa alignments completed and it will output your assembled contigs.

Giacomoggioli4111 commented 5 years ago

I have created those two links, they are broken though. This is the content of unitigging/4-unitigger folder:


-rw-r--r-- 1 btx604 qmul       3140 Nov  4 14:56 alignGFA.1.out
-rw-r--r-- 1 btx604 qmul        132 Nov  4 17:16 alignGFA.jobSubmit-01.out
-rwxr-xr-x 1 btx604 qmul        215 Nov  4 17:16 alignGFA.jobSubmit-01.sh
-rwxr-xr-x 1 btx604 qmul       1642 Nov  4 13:58 alignGFA.sh
lrwxrwxrwx 1 btx604 qmul         23 Nov  4 17:14 asm.unitigs.bed -> asm.unitigs.aligned.bed
lrwxrwxrwx 1 btx604 qmul         23 Nov  4 17:14 asm.unitigs.gfa -> asm.unitigs.aligned.gfa
-rw-r--r-- 1 btx604 qmul       1209 Nov  2 16:52 Oasisia.001.filterOverlaps.thr000.num000.log
-rw-r--r-- 1 btx604 qmul        442 Nov  2 16:52 Oasisia.001.filterOverlaps.thr001.num000.log
-rw-r--r-- 1 btx604 qmul        439 Nov  2 16:52 Oasisia.001.filterOverlaps.thr002.num000.log
-rw-r--r-- 1 btx604 qmul        203 Nov  2 16:52 Oasisia.001.filterOverlaps.thr003.num000.log
-rw-r--r-- 1 btx604 qmul        493 Nov  2 16:52 Oasisia.001.filterOverlaps.thr004.num000.log
-rw-r--r-- 1 btx604 qmul        388 Nov  2 16:52 Oasisia.001.filterOverlaps.thr005.num000.log
-rw-r--r-- 1 btx604 qmul        362 Nov  2 16:52 Oasisia.001.filterOverlaps.thr006.num000.log
-rw-r--r-- 1 btx604 qmul        593 Nov  2 16:52 Oasisia.001.filterOverlaps.thr007.num000.log
-rw-r--r-- 1 btx604 qmul        205 Nov  2 16:52 Oasisia.001.filterOverlaps.thr008.num000.log
-rw-r--r-- 1 btx604 qmul        387 Nov  2 16:52 Oasisia.001.filterOverlaps.thr009.num000.log
-rw-r--r-- 1 btx604 qmul        154 Nov  2 16:52 Oasisia.001.filterOverlaps.thr010.num000.log
-rw-r--r-- 1 btx604 qmul        571 Nov  2 16:52 Oasisia.001.filterOverlaps.thr011.num000.log
-rw-r--r-- 1 btx604 qmul        569 Nov  2 16:52 Oasisia.001.filterOverlaps.thr012.num000.log
-rw-r--r-- 1 btx604 qmul        460 Nov  2 16:52 Oasisia.001.filterOverlaps.thr013.num000.log
-rw-r--r-- 1 btx604 qmul        308 Nov  2 16:52 Oasisia.001.filterOverlaps.thr014.num000.log
-rw-r--r-- 1 btx604 qmul        333 Nov  2 16:52 Oasisia.001.filterOverlaps.thr015.num000.log
-rw-r--r-- 1 btx604 qmul        648 Nov  2 16:52 Oasisia.001.filterOverlaps.thr016.num000.log
-rw-r--r-- 1 btx604 qmul       1066 Nov  2 16:52 Oasisia.003.buildGreedy.sizes
-rw-r--r-- 1 btx604 qmul      30414 Nov  2 16:52 Oasisia.003.buildGreedy.thr000.num000.log
-rw-r--r-- 1 btx604 qmul       1066 Nov  2 16:52 Oasisia.004.buildGreedyOpt.sizes
-rw-r--r-- 1 btx604 qmul       1066 Nov  2 16:52 Oasisia.005.placeContains.sizes
-rw-r--r-- 1 btx604 qmul       1066 Nov  2 16:54 Oasisia.006.placeContainsOpt.sizes
-rw-r--r-- 1 btx604 qmul      37472 Nov  2 16:54 Oasisia.006.placeContainsOpt.thr003.num000.log
-rw-r--r-- 1 btx604 qmul      25606 Nov  2 16:54 Oasisia.006.placeContainsOpt.thr004.num000.log
-rw-r--r-- 1 btx604 qmul       4849 Nov  2 16:54 Oasisia.006.placeContainsOpt.thr005.num000.log
-rw-r--r-- 1 btx604 qmul      97843 Nov  2 16:54 Oasisia.006.placeContainsOpt.thr006.num000.log
-rw-r--r-- 1 btx604 qmul       6964 Nov  2 16:54 Oasisia.006.placeContainsOpt.thr009.num000.log
-rw-r--r-- 1 btx604 qmul      18665 Nov  2 16:54 Oasisia.006.placeContainsOpt.thr012.num000.log
-rw-r--r-- 1 btx604 qmul     543343 Nov  2 16:54 Oasisia.006.placeContainsOpt.thr015.num000.log
-rw-r--r-- 1 btx604 qmul      29071 Nov  2 16:54 Oasisia.006.placeContainsOpt.thr016.num000.log
-rw-r--r-- 1 btx604 qmul       1066 Nov  2 16:54 Oasisia.007.mergeOrphans.sizes
-rw-r--r-- 1 btx604 qmul    1177902 Nov  2 16:54 Oasisia.007.mergeOrphans.thr000.num000.log
-rw-r--r-- 1 btx604 qmul       9139 Nov  2 16:54 Oasisia.007.mergeOrphans.unassembled
-rw-r--r-- 1 btx604 qmul       1148 Nov  2 17:01 Oasisia.009.breakRepeats.sizes
-rw-r--r-- 1 btx604 qmul   25988190 Nov  2 17:01 Oasisia.009.breakRepeats.thr000.num000.log
-rw-r--r-- 1 btx604 qmul     594706 Nov  2 17:01 Oasisia.010.cleanupMistakes.thr000.num000.log
-rw-r--r-- 1 btx604 qmul       3502 Nov  2 17:04 Oasisia.011.generateOutputs.overlaps
-rw-r--r-- 1 btx604 qmul       5307 Nov  2 17:04 Oasisia.011.generateOutputs.sizes
-rw-r--r-- 1 btx604 qmul         59 Nov  2 17:05 Oasisia.011.generateOutputs.thr000.num000.log
-rw-r--r-- 1 btx604 qmul  536871060 Nov  2 17:07 Oasisia.013.generateUnitigs.thr000.num000.log
-rw-r--r-- 1 btx604 qmul  106241112 Nov  2 17:08 Oasisia.013.generateUnitigs.thr000.num001.log
-rw-r--r-- 1 btx604 qmul        221 Nov  2 17:08 Oasisia.013.generateUnitigs.thr003.num000.log
-rw-r--r-- 1 btx604 qmul        141 Nov  2 17:08 Oasisia.013.generateUnitigs.thr011.num000.log
-rw-r--r-- 1 btx604 qmul   56002032 Nov  2 16:52 Oasisia.best.contains.histogram
-rw-r--r-- 1 btx604 qmul  199134288 Nov  2 16:52 Oasisia.best.edges
-rw-r--r-- 1 btx604 qmul   14911599 Nov  2 16:52 Oasisia.best.edges.gfa
-rw-r--r-- 1 btx604 qmul    2680569 Nov  2 16:52 Oasisia.best.edges.histogram
-rw-r--r-- 1 btx604 qmul     513669 Nov  2 16:52 Oasisia.best.edges.suspicious
-rw-r--r-- 1 btx604 qmul   19185131 Nov  2 16:52 Oasisia.best.singletons
-rw-r--r-- 1 btx604 qmul   34913839 Nov  2 16:48 Oasisia.best.spurs
-rw-r--r-- 1 btx604 qmul    9030940 Nov  4 14:00 Oasisia.contigs.aligned.gfa
-rw-r--r-- 1 btx604 qmul        722 Nov  4 14:00 Oasisia.contigs.aligned.gfa.err
-rw-r--r-- 1 btx604 qmul          0 Nov  2 17:04 Oasisia.contigs.bed
-rw-r--r-- 1 btx604 qmul     247066 Nov  2 17:05 Oasisia.contigs.gfa
-rw-r--r-- 1 btx604 qmul  526167160 Nov  2 17:04 Oasisia.final.assembly.gfa
-rw-r--r-- 1 btx604 qmul 1422712957 Nov  2 17:00 Oasisia.initial.assembly.gfa
-rw-r--r-- 1 btx604 qmul        627 Nov  4 14:56 Oasisia.unitigs.aligned.bed.err
-rw-r--r-- 1 btx604 qmul        642 Nov  4 14:56 Oasisia.unitigs.aligned.gfa.err
-rw-r--r-- 1 btx604 qmul     166421 Nov  2 17:08 Oasisia.unitigs.bed
-rw-r--r-- 1 btx604 qmul     936309 Nov  2 17:08 Oasisia.unitigs.gfa
-rw-r--r-- 1 btx604 qmul        355 Nov  2 16:24 unitigger.1.out
-rw-r--r-- 1 btx604 qmul      14907 Nov  2 17:08 unitigger.err
-rw-r--r-- 1 btx604 qmul        140 Nov  2 16:24 unitigger.jobSubmit-01.out
-rwxr-xr-x 1 btx604 qmul        249 Nov  2 16:24 unitigger.jobSubmit-01.sh
-rwxr-xr-x 1 btx604 qmul       2491 Nov  2 16:23 unitigger.sh
-rw-r--r-- 1 btx604 qmul          0 Nov  2 17:09 unitigger.success

This is my last canu.out:


Found perl:
   /bin/perl
   This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux-thread-multi

Found java:
   /share/apps/centos7/java/1.8.0_152/bin/java
   java version "1.8.0_152"

Found canu:
   /data/SBCS-MartinDuranLab/03-Giacomo/src/canu-1.8/Linux-amd64/bin/canu
   Canu 1.8

-- Canu 1.8
--
-- CITATIONS
--
-- Koren S, Walenz BP, Berlin K, Miller JR, Phillippy AM.
-- Canu: scalable and accurate long-read assembly via adaptive k-mer weighting and repeat separation.
-- Genome Res. 2017 May;27(5):722-736.
-- http://doi.org/10.1101/gr.215087.116
-- 
-- Koren S, Rhie A, Walenz BP, Dilthey AT, Bickhart DM, Kingan SB, Hiendleder S, Williams JL, Smith TPL, Phillippy AM.
-- De novo assembly of haplotype-resolved genomes with trio binning.
-- Nat Biotechnol. 2018
-- https//doi.org/10.1038/nbt.4277
-- 
-- Read and contig alignments during correction, consensus and GFA building use:
--   Šošic M, Šikic M.
--   Edlib: a C/C ++ library for fast, exact sequence alignment using edit distance.
--   Bioinformatics. 2017 May 1;33(9):1394-1395.
--   http://doi.org/10.1093/bioinformatics/btw753
-- 
-- Overlaps are generated using:
--   Berlin K, et al.
--   Assembling large genomes with single-molecule sequencing and locality-sensitive hashing.
--   Nat Biotechnol. 2015 Jun;33(6):623-30.
--   http://doi.org/10.1038/nbt.3238
-- 
--   Myers EW, et al.
--   A Whole-Genome Assembly of Drosophila.
--   Science. 2000 Mar 24;287(5461):2196-204.
--   http://doi.org/10.1126/science.287.5461.2196
-- 
-- Corrected read consensus sequences are generated using an algorithm derived from FALCON-sense:
--   Chin CS, et al.
--   Phased diploid genome assembly with single-molecule real-time sequencing.
--   Nat Methods. 2016 Dec;13(12):1050-1054.
--   http://doi.org/10.1038/nmeth.4035
-- 
-- Contig consensus sequences are generated using an algorithm derived from pbdagcon:
--   Chin CS, et al.
--   Nonhybrid, finished microbial genome assemblies from long-read SMRT sequencing data.
--   Nat Methods. 2013 Jun;10(6):563-9
--   http://doi.org/10.1038/nmeth.2474
-- 
-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '1.8.0_152' (from '/share/apps/centos7/java/1.8.0_152/bin/java') with -d64 support.
-- Detected gnuplot version '5.2 patchlevel 0   ' (from '/share/apps/centos7/gnuplot/5.2.0/bin/gnuplot') and image format 'png'.
-- Detected 48 CPUs and 504 gigabytes of memory.
-- Detected Sun Grid Engine in '/opt/sge/default'.
-- User supplied Grid Engine environment '-pe smp THREADS'.
-- User supplied Grid Engine consumable '-l h_vmem=MEMORY'.
-- 
-- Found 150 hosts with  12 cores and   23 GB memory under Sun Grid Engine control.
-- Found   2 hosts with  36 cores and  755 GB memory under Sun Grid Engine control.
-- Found   1 host  with  40 cores and  755 GB memory under Sun Grid Engine control.
-- Found   2 hosts with  16 cores and   31 GB memory under Sun Grid Engine control.
-- Found  32 hosts with  16 cores and   62 GB memory under Sun Grid Engine control.
-- Found   2 hosts with  32 cores and  377 GB memory under Sun Grid Engine control.
-- Found  63 hosts with  24 cores and  188 GB memory under Sun Grid Engine control.
-- Found  46 hosts with  32 cores and  251 GB memory under Sun Grid Engine control.
-- Found  11 hosts with  48 cores and  503 GB memory under Sun Grid Engine control.
-- Found   1 host  with  32 cores and  503 GB memory under Sun Grid Engine control.
--
--                     (tag)Threads
--            (tag)Memory         |
--        (tag)         |         |  algorithm
--        -------  ------  --------  -----------------------------
-- Grid:  meryl     31 GB    8 CPUs  (k-mer counting)
-- Grid:  hap       16 GB   16 CPUs  (read-to-haplotype assignment)
-- Grid:  cormhap   23 GB   12 CPUs  (overlap detection with mhap)
-- Grid:  obtovl    16 GB   12 CPUs  (overlap detection)
-- Grid:  utgovl    16 GB   12 CPUs  (overlap detection)
-- Grid:  ovb        4 GB    1 CPU   (overlap store bucketizer)
-- Grid:  ovs       32 GB    1 CPU   (overlap store sorting)
-- Grid:  red       11 GB    6 CPUs  (read error detection)
-- Grid:  oea        4 GB    1 CPU   (overlap error adjustment)
-- Grid:  bat      256 GB   16 CPUs  (contig construction with bogart)
-- Grid:  gfa       16 GB   16 CPUs  (GFA alignment and processing)
--
-- In 'Oasisia.seqStore', found PacBio reads:
--   Raw:        5225176
--   Corrected:  3468687
--   Trimmed:    3276877
--
-- Generating assembly 'Oasisia' in '/data/scratch/btx604/Oasisia/canu_V8'
--
-- Parameters:
--
--  genomeSize        1000000000
--
--  Overlap Generation Limits:
--    corOvlErrorRate 0.2400 ( 24.00%)
--    obtOvlErrorRate 0.0400 (  4.00%)
--    utgOvlErrorRate 0.0400 (  4.00%)
--
--  Overlap Processing Limits:
--    corErrorRate    0.3000 ( 30.00%)
--    obtErrorRate    0.0400 (  4.00%)
--    utgErrorRate    0.0400 (  4.00%)
--    cnsErrorRate    0.0400 (  4.00%)
--
--
-- BEGIN ASSEMBLY
--
--
-- Running jobs.  First attempt out of 2.
--

CRASH:
CRASH: Canu 1.8
CRASH: Please panic, this is abnormal.
ABORT:
CRASH:   Failed to submit batch jobs.
CRASH:
CRASH: Failed at /data/SBCS-MartinDuranLab/03-Giacomo/src/canu-1.8/Linux-amd64/bin/../lib/site_perl/canu/Execution.pm line 1233.
CRASH:  canu::Execution::submitOrRunParallelJob('Oasisia', 'gfa', 'unitigging/4-unitigger', 'alignGFA', 1) called at /data/SBCS-MartinDuranLab/03-Giacomo/src/canu-1.8/Linux-amd64/bin/../lib/site_perl/canu/Consensus.pm line 765
CRASH:  canu::Consensus::alignGFA('Oasisia') called at /data/SBCS-MartinDuranLab/03-Giacomo/src/canu-1.8/Linux-amd64/bin/canu line 873
CRASH: 
CRASH: Last 50 lines of the relevant log file (unitigging/4-unitigger/alignGFA.jobSubmit-01.out):
CRASH:
CRASH: Unable to run job: error: no suitable queues.
CRASH: Exiting.
CRASH: qsub: warning: "-j" option has already been set, overriding previous setting
CRASH:
skoren commented 5 years ago

Sorry the link should be

ln -s Oasisia.unitigs.gfa Oasisia.unitigs.aligned.gfa 
ln -s Oasisia.unitigs.bed Oasisia.unitigs.aligned.bed

Your last run error is irrelevant to that and isn't a Canu error. It's an error due to the resources you specified in your options. I'd guess this: "gridOptionsGFA=-l h_vmem=300G", remove that option.

Giacomoggioli4111 commented 5 years ago

ln: failed to create symbolic link ‘Oasisia.unitigs.gfa’: File exists

skoren commented 5 years ago

Reverse the order of the ln arguments

skoren commented 4 years ago

Any updates?

skoren commented 4 years ago

Idle, should be fixed in tip and workaround to obtain assembly fasta (without the graph) is above.

Giacomoggioli4111 commented 4 years ago

Sorry for my late reply. Unfortunately, I did not have time to try out your suggestions yet. I am actually thinking about launching CANU again without specifying correctedErrorRate and corOutCoverage. Thank you very much