search

marbl / canu

A single molecule sequence assembler for genomes large and small.
http://canu.readthedocs.io/
658 stars 179 forks source link

Canu test on 25 fastq: Failed with 'Segmentation fault' and canu failed with 'failed to create the overlap store'. #449

Closed aminards closed 7 years ago

aminards commented 7 years ago

I am testing using Canu with 25 Nanopore 1D reads. I am getting failure notices but I am not sure how to fix the error. Here is the code I ran:

#!/bin/bash
#PBS -l nodes=n03:ppn=8
#PBS -l walltime=48:00:00
#PBS -N Canu-assembler
#PBS -q abat
#PBS -j oe
#PBS -M smitha@myinst.org
#PBS -m abe

cd /home/smitha/
/home/smitha/tools/canu-1.4/Linux-amd64/bin/canu \
-d /home/smitha/MinION/1D_WGS_data/01052017/CANU-Test/ \
-p 01052017-25reads errorRate=0.013 genomeSize=16569 useGrid=false \
-nanopore-raw /home/smitha/MinION/1D_WGS_data/01052017/CANU-Test/01052017-25reads.fastq

And here is the output from this run:

-- Canu v0.0 (+0 commits) r0 unknown-hash-tag-no-repository-available.
-- Detected Java(TM) Runtime Environment '1.8.0_92' (from 'java').
-- Detected gnuplot version '4.4 patchlevel 3' (from 'gnuplot') and image format 'png'.
-- Detected 16 CPUs and 47 gigabytes of memory.
-- Detected PBS/Torque '' with 'pbsnodes' binary in /usr/bin/pbsnodes.
-- Grid engine disabled per useGrid=false option.
--
-- Allowed to run   4 jobs concurrently, and use up to   4 compute threads and   11 GB memory for stage 'bogart (unitigger)'.
-- Allowed to run   1 job  concurrently, and use up to  16 compute threads and    6 GB memory for stage 'mhap (overlapper)'.
-- Allowed to run   1 job  concurrently, and use up to  16 compute threads and    6 GB memory for stage 'mhap (overlapper)'.
-- Allowed to run   1 job  concurrently, and use up to  16 compute threads and    6 GB memory for stage 'mhap (overlapper)'.
-- Allowed to run   4 jobs concurrently, and use up to   4 compute threads and    2 GB memory for stage 'read error detection (overlap error adjustment)'.
-- Allowed to run  16 jobs concurrently, and use up to   1 compute thread  and    1 GB memory for stage 'overlap error adjustment'.
-- Allowed to run   4 jobs concurrently, and use up to   4 compute threads and   11 GB memory for stage 'utgcns (consensus)'.
-- Allowed to run  16 jobs concurrently, and use up to   1 compute thread  and    2 GB memory for stage 'overlap store parallel bucketizer'.
-- Allowed to run  16 jobs concurrently, and use up to   1 compute thread  and    8 GB memory for stage 'overlap store parallel sorting'.
-- Allowed to run  16 jobs concurrently, and use up to   1 compute thread  and    2 GB memory for stage 'overlapper'.
-- Allowed to run   2 jobs concurrently, and use up to   8 compute threads and    8 GB memory for stage 'overlapper'.
-- Allowed to run   2 jobs concurrently, and use up to   8 compute threads and    8 GB memory for stage 'overlapper'.
-- Allowed to run   4 jobs concurrently, and use up to   4 compute threads and    8 GB memory for stage 'meryl (k-mer counting)'.
-- Allowed to run   8 jobs concurrently, and use up to   2 compute threads and    6 GB memory for stage 'falcon_sense (read correction)'.
-- Allowed to run   1 job  concurrently, and use up to  16 compute threads and    6 GB memory for stage 'minimap (overlapper)'.
-- Allowed to run   1 job  concurrently, and use up to  16 compute threads and    6 GB memory for stage 'minimap (overlapper)'.
-- Allowed to run   1 job  concurrently, and use up to  16 compute threads and    6 GB memory for stage 'minimap (overlapper)'.
--
-- This is canu parallel iteration #1, out of a maximum of 2 attempts.
--
-- Final error rates before starting pipeline:
--   
--   genomeSize          -- 16569
--   errorRate           -- 0.013
--   
--   corOvlErrorRate     -- 0.039
--   obtOvlErrorRate     -- 0.039
--   utgOvlErrorRate     -- 0.039
--   
--   obtErrorRate        -- 0.039
--   
--   cnsErrorRate        -- 0.039
--
--
-- BEGIN CORRECTION
--
----------------------------------------
-- Starting command on Wed Apr 12 09:00:15 2017 with 37265.738 GB free disk space

    /home/smith/tools/canu-1.4/Linux-amd64/bin/gatekeeperCreate \
      -minlength 1000 \
      -o /home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/01052017-25reads.gkpStore.BUILDING \
      /home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/01052017-25reads.gkpStore.gkp \
    > /home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/01052017-25reads.gkpStore.BUILDING.err 2>&1

-- Finished on Wed Apr 12 09:00:15 2017 (lickety-split) with 37265.737 GB free disk space
----------------------------------------
--
-- In gatekeeper store '/home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/01052017-25reads.gkpStore':
--   Found 23 reads.
--   Found 189913 bases (11.46 times coverage).
--
--   Read length histogram (one '*' equals 0.15 reads):
--        0    999      0 
--     1000   1999      1 ******
--     2000   2999      0 
--     3000   3999      3 *******************
--     4000   4999      1 ******
--     5000   5999      0 
--     6000   6999      0 
--     7000   7999      0 
--     8000   8999      6 **************************************
--     9000   9999      1 ******
--    10000  10999     11 **********************************************************************
-- Meryl attempt 1 begins.
----------------------------------------
-- Starting concurrent execution on Wed Apr 12 09:00:16 2017 with 37265.735 GB free disk space (1 processes; 4 concurrently)

    /home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/0-mercounts/meryl.sh 1 > /home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/0-mercounts/meryl.000001.out 2>&1

-- Finished on Wed Apr 12 09:00:17 2017 (2 seconds) with 37265.73 GB free disk space
----------------------------------------
-- Meryl finished successfully.
----------------------------------------
-- Starting command on Wed Apr 12 09:00:17 2017 with 37265.73 GB free disk space

    /home/smith/tools/canu-1.4/Linux-amd64/bin/meryl \
      -Dh \
      -s /home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/0-mercounts/01052017-25reads.ms16 \
    > /home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/0-mercounts/01052017-25reads.ms16.histogram \
    2> /home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/0-mercounts/01052017-25reads.ms16.histogram.info

-- Finished on Wed Apr 12 09:00:17 2017 (lickety-split) with 37265.73 GB free disk space
----------------------------------------
-- For mhap overlapping, set repeat k-mer threshold to 1.
--
-- Found 189568 16-mers; 158030 distinct and 142069 unique.  Largest count 11.
--
-- OVERLAPPER (mhap) (correction)
--
-- Set corMhapSensitivity=high based on read coverage of 11.
--
-- PARAMETERS: hashes=768, minMatches=2, threshold=0.73
--
-- Given 6 GB, can fit 9000 reads per block.
-- For 2 blocks, set stride to 2 blocks.
-- Logging partitioning to '/home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/1-overlapper/partitioning.log'.
-- Configured 1 mhap precompute jobs.
-- Configured 1 mhap overlap jobs.
-- mhap precompute attempt 1 begins with 0 finished, and 1 to compute.
----------------------------------------
-- Starting concurrent execution on Wed Apr 12 09:00:18 2017 with 37265.726 GB free disk space (1 processes; 1 concurrently)

    /home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/1-overlapper/precompute.sh 1 > /home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/1-overlapper/precompute.000001.out 2>&1

-- Finished on Wed Apr 12 09:00:22 2017 (4 seconds) with 37265.717 GB free disk space
----------------------------------------
-- All 1 mhap precompute jobs finished successfully.
-- mhap attempt 1 begins with 0 finished, and 1 to compute.
----------------------------------------
-- Starting concurrent execution on Wed Apr 12 09:00:22 2017 with 37265.717 GB free disk space (1 processes; 1 concurrently)

    /home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/1-overlapper/mhap.sh 1 > /home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/1-overlapper/mhap.000001.out 2>&1

-- Finished on Wed Apr 12 09:00:23 2017 (1 second) with 37265.713 GB free disk space
----------------------------------------
-- Found 1 mhap overlap output files.
----------------------------------------
-- Starting command on Wed Apr 12 09:00:24 2017 with 37265.712 GB free disk space

    /home/smith/tools/canu-1.4/Linux-amd64/bin/ovStoreBuild \
     -O /home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/01052017-25reads.ovlStore.BUILDING \
     -G /home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/01052017-25reads.gkpStore \
     -M 2-8 \
     -L /home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/1-overlapper/ovljob.files \
     > /home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/01052017-25reads.ovlStore.err 2>&1
sh: line 5:  5741 Segmentation fault      /home/smith/tools/canu-1.4/Linux-amd64/bin/ovStoreBuild -O /home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/01052017-25reads.ovlStore.BUILDING -G /home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/01052017-25reads.gkpStore -M 2-8 -L /home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/1-overlapper/ovljob.files > /home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/01052017-25reads.ovlStore.err 2>&1

-- Finished on Wed Apr 12 09:00:24 2017 (lickety-split) with 37265.712 GB free disk space
----------------------------------------
ERROR:
ERROR:  Failed with exit code 139.  (rc=35584)
ERROR:
================================================================================
Don't panic, but a mostly harmless error occurred and canu failed.

Disk space available:  37265.712 GB

Last 50 lines of the relevant log file (/home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/01052017-25reads.ovlStore.err):

Found 318 (0.00 million) overlaps.
Configuring for 2.00 GB to 8.00 GB memory and 1008 open files.
Will sort using 1 files; 58720256 (58.72 million) overlaps per bucket; 2.00 GB memory per bucket
  bucket   1 has 318 olaps.
  bucket   2 has 0 olaps.
Will sort 0.000 million overlaps per bucket, using 1 buckets 0.25 GB per bucket.

-- BUCKETIZING --

-  Bucketizing '/home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/1-overlapper/results/000001.ovb'
-- Create bucket '/home/smith/MinION/1D_WGS_data/01052017/CANU-Test//correction/01052017-25reads.ovlStore.BUILDING/tmp.sort.001'

Failed with 'Segmentation fault'

Backtrace (mangled):

/home/smith/tools/canu-1.4/Linux-amd64/bin/ovStoreBuild(_Z17AS_UTL_catchCrashiP7siginfoPv+0x24)[0x408014]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f4f310a9cb0]
/lib/x86_64-linux-gnu/libc.so.6(_IO_vfprintf+0x25c6)[0x7f4f30d26b36]
/lib/x86_64-linux-gnu/libc.so.6(+0x4da64)[0x7f4f30d29a64]
/lib/x86_64-linux-gnu/libc.so.6(_IO_vfprintf+0x21e)[0x7f4f30d2478e]
/lib/x86_64-linux-gnu/libc.so.6(__fprintf_chk+0xeb)[0x7f4f30de46fb]
/home/smith/tools/canu-1.4/Linux-amd64/bin/ovStoreBuild[0x403e3b]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f4f30cfd7ed]
/home/smith/tools/canu-1.4/Linux-amd64/bin/ovStoreBuild[0x404ac1]

Backtrace (demangled):

[0] /home/smith/tools/canu-1.4/Linux-amd64/bin/ovStoreBuild::AS_UTL_catchCrash(int, siginfo*, void*) + 0x24  [0x408014]
[1] /lib/x86_64-linux-gnu/libpthread.so.0::(null) + 0xfcb0  [0x7f4f310a9cb0]
[2] /lib/x86_64-linux-gnu/libc.so.6::(null) + 0x25c6  [0x7f4f30d26b36]
[3] /lib/x86_64-linux-gnu/libc.so.6::(null) + 0x4da64  [0x7f4f30d29a64]
[4] /lib/x86_64-linux-gnu/libc.so.6::(null) + 0x21e  [0x7f4f30d2478e]
[5] /lib/x86_64-linux-gnu/libc.so.6::(null) + 0xeb  [0x7f4f30de46fb]
[6] /home/smith/tools/canu-1.4/Linux-amd64/bin/ovStoreBuild() [0x403e3b]
[7] /lib/x86_64-linux-gnu/libc.so.6::(null) + 0xed  [0x7f4f30cfd7ed]
[8] /home/smith/tools/canu-1.4/Linux-amd64/bin/ovStoreBuild() [0x404ac1]

GDB:

canu failed with 'failed to create the overlap store'.
skoren commented 7 years ago

This is most likely a known bug in 1.4 (see issue #319 and #369). Try updating to the latest code from the repository (not a release) and re-starting the run from scratch.

brianwalenz commented 7 years ago

Assuming it's fixed, since this should take an hour to test and it's been five days.