search

marbl / canu

A single molecule sequence assembler for genomes large and small.
http://canu.readthedocs.io/
655 stars 179 forks source link

Mhap overlap jobs failed in CANU1.8 #1291

Closed biowackysci closed 5 years ago

biowackysci commented 5 years ago

Hello Everyone,

We are running Canu (version 1.8) using a hybrid of PacBio Sequel+RSII and Nanopore reads using a cluster with slurm as workflow manager.

The CANU command that I use is 

#!/bin/bash
#PBS -S /bin/bash
#PBS -V
#PBS -N Assembly_genome_CANU
#PBS -q batch
#PBS -l nodes=1:ppn=10
#PBS -l walltime=1000:00:00
#PBS -l mem=500g
#PBS -M saila.vishwanathan@ecodev.vic.gov.au 
#PBS -m abe

#cp /group/pasture/Saila/MinION/Impact04_run2/20180426_0452_impact04_SV_26april2018/fastq/pass/fastq/ $TMPDIR/

#cd $TMPDIR/
#cp /group/pasture/Saila/all_Pacbio.fastq . $TMPDIR/
#cp /group/pasture/Saila/all_oxford_reads.fastq . $TMPDIR/

#cd $PBS_O_WORKDIR

cd $SLURM_SUBMIT_DIR

module load Java
module load canu

canu -p canu -d /group/home/vici86y/canu-1.8 genomeSize=2.8g -fast maxThreads=16 useGrid=false stopOnReadQuality=false corOutCoverage=1000 minOverlapLength=1000 -nanopore-raw /group/pasture/Saila/trimmed_oxford_reads.fastq -pacbio-raw /group/pasture/Saila/all_Pacbio.fastq

#cp -r $TMPDIR/genome_assembly /group/home/vici86y/canu-1.8

the output of the run as reported 

-- Canu 1.8
--
-- CITATIONS
--
-- Koren S, Walenz BP, Berlin K, Miller JR, Phillippy AM.
-- Canu: scalable and accurate long-read assembly via adaptive k-mer weighting and repeat separation.
-- Genome Res. 2017 May;27(5):722-736.
-- http://doi.org/10.1101/gr.215087.116
-- 
-- Koren S, Rhie A, Walenz BP, Dilthey AT, Bickhart DM, Kingan SB, Hiendleder S, Williams JL, Smith TPL, Phillippy AM.
-- De novo assembly of haplotype-resolved genomes with trio binning.
-- Nat Biotechnol. 2018
-- https//doi.org/10.1038/nbt.4277
-- 
-- Read and contig alignments during correction, consensus and GFA building use:
--   Šošic M, Šikic M.
--   Edlib: a C/C ++ library for fast, exact sequence alignment using edit distance.
--   Bioinformatics. 2017 May 1;33(9):1394-1395.
--   http://doi.org/10.1093/bioinformatics/btw753
-- 
-- Overlaps are generated using:
--   Berlin K, et al.
--   Assembling large genomes with single-molecule sequencing and locality-sensitive hashing.
--   Nat Biotechnol. 2015 Jun;33(6):623-30.
--   http://doi.org/10.1038/nbt.3238
-- 
-- Corrected read consensus sequences are generated using an algorithm derived from FALCON-sense:
--   Chin CS, et al.
--   Phased diploid genome assembly with single-molecule real-time sequencing.
--   Nat Methods. 2016 Dec;13(12):1050-1054.
--   http://doi.org/10.1038/nmeth.4035
-- 
-- Contig consensus sequences are generated using an algorithm derived from pbdagcon:
--   Chin CS, et al.
--   Nonhybrid, finished microbial genome assemblies from long-read SMRT sequencing data.
--   Nat Methods. 2013 Jun;10(6):563-9
--   http://doi.org/10.1038/nmeth.2474
-- 
-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '1.8.0_172' (from '/usr/local/EasyBuild/software/Java/1.8.0_172/bin/java') with -d64 support.
--
-- WARNING:
-- WARNING:  Failed to run gnuplot using command 'gnuplot'.
-- WARNING:  Plots will be disabled.
-- WARNING:
--
-- Detected 48 CPUs and 1510 gigabytes of memory.
-- Limited to 16 CPUs from maxThreads option.
-- Detected Slurm with 'sinfo' binary in /usr/local/slurm/current/bin/sinfo.
-- Grid engine disabled per useGrid=false option.
--
--                            (tag)Concurrency
--                     (tag)Threads          |
--            (tag)Memory         |          |
--        (tag)         |         |          |     total usage     algorithm
--        -------  ------  --------   --------  -----------------  -----------------------------
-- Local: meryl     64 GB    8 CPUs x   2 jobs   128 GB   16 CPUs  (k-mer counting)
-- Local: hap       16 GB   16 CPUs x   1 job     16 GB   16 CPUs  (read-to-haplotype assignment)
-- Local: cormhap   48 GB   16 CPUs x   1 job     48 GB   16 CPUs  (overlap detection with mhap)
-- Local: obtmhap   48 GB   16 CPUs x   1 job     48 GB   16 CPUs  (overlap detection with mhap)
-- Local: utgmhap   48 GB   16 CPUs x   1 job     48 GB   16 CPUs  (overlap detection with mhap)
-- Local: ovb        4 GB    1 CPU  x  16 jobs    64 GB   16 CPUs  (overlap store bucketizer)
-- Local: ovs       32 GB    1 CPU  x  16 jobs   512 GB   16 CPUs  (overlap store sorting)
-- Local: red       16 GB    8 CPUs x   2 jobs    32 GB   16 CPUs  (read error detection)
-- Local: oea        8 GB    1 CPU  x  16 jobs   128 GB   16 CPUs  (overlap error adjustment)
-- Local: bat      512 GB   16 CPUs x   1 job    512 GB   16 CPUs  (contig construction with bogart)
-- Local: gfa       32 GB   16 CPUs x   1 job     32 GB   16 CPUs  (GFA alignment and processing)
--
-- In 'canu.seqStore', found both PacBio and Nanopore reads:
--   Raw:        3957519
--   Corrected:  0
--   Trimmed:    0
--
-- Generating assembly 'canu' in '/group/home/vici86y/canu-1.8'
--
-- Parameters:
--
--  genomeSize        2800000000
--
--  Overlap Generation Limits:
--    corOvlErrorRate 0.3200 ( 32.00%)
--    obtOvlErrorRate 0.1200 ( 12.00%)
--    utgOvlErrorRate 0.1200 ( 12.00%)
--
--  Overlap Processing Limits:
--    corErrorRate    0.5000 ( 50.00%)
--    obtErrorRate    0.1200 ( 12.00%)
--    utgErrorRate    0.1200 ( 12.00%)
--    cnsErrorRate    0.2000 ( 20.00%)
--
--
-- BEGIN CORRECTION
--
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'cormhap' concurrent execution on Fri Mar 22 11:58:13 2019 with 1888.699 GB free disk space (7 processes; 1 concurrently)

    cd correction/1-overlapper
    ./mhap.sh 87 > ./mhap.000087.out 2>&1
    ./mhap.sh 88 > ./mhap.000088.out 2>&1
    ./mhap.sh 89 > ./mhap.000089.out 2>&1
    ./mhap.sh 90 > ./mhap.000090.out 2>&1
    ./mhap.sh 91 > ./mhap.000091.out 2>&1
    ./mhap.sh 92 > ./mhap.000092.out 2>&1
    ./mhap.sh 93 > ./mhap.000093.out 2>&1

-- Finished on Fri Mar 22 12:30:32 2019 (1939 seconds) with 1887.89 GB free disk space
----------------------------------------
--
-- Mhap overlap jobs failed, retry.
--   job correction/1-overlapper/results/000087.ovb FAILED.
--   job correction/1-overlapper/results/000088.ovb FAILED.
--   job correction/1-overlapper/results/000089.ovb FAILED.
--   job correction/1-overlapper/results/000090.ovb FAILED.
--   job correction/1-overlapper/results/000091.ovb FAILED.
--   job correction/1-overlapper/results/000092.ovb FAILED.
--   job correction/1-overlapper/results/000093.ovb FAILED.
--
--
-- Running jobs.  Second attempt out of 2.
----------------------------------------
-- Starting 'cormhap' concurrent execution on Fri Mar 22 12:30:32 2019 with 1887.89 GB free disk space (7 processes; 1 concurrently)

    cd correction/1-overlapper
    ./mhap.sh 87 > ./mhap.000087.out 2>&1
    ./mhap.sh 88 > ./mhap.000088.out 2>&1
    ./mhap.sh 89 > ./mhap.000089.out 2>&1
    ./mhap.sh 90 > ./mhap.000090.out 2>&1
    ./mhap.sh 91 > ./mhap.000091.out 2>&1
    ./mhap.sh 92 > ./mhap.000092.out 2>&1
    ./mhap.sh 93 > ./mhap.000093.out 2>&1

-- Finished on Fri Mar 22 13:03:23 2019 (1971 seconds) with 1886.914 GB free disk space
----------------------------------------
--
-- Mhap overlap jobs failed, tried 2 times, giving up.
--   job correction/1-overlapper/results/000087.ovb FAILED.
--   job correction/1-overlapper/results/000088.ovb FAILED.
--   job correction/1-overlapper/results/000089.ovb FAILED.
--   job correction/1-overlapper/results/000090.ovb FAILED.
--   job correction/1-overlapper/results/000091.ovb FAILED.
--   job correction/1-overlapper/results/000092.ovb FAILED.
--   job correction/1-overlapper/results/000093.ovb FAILED.
--

ABORT:
ABORT: Canu 1.8
ABORT: Don't panic, but a mostly harmless error occurred and Canu stopped.
ABORT: Try restarting.  If that doesn't work, ask for help.
ABORT:

I have tried multiple times and it always terminates with the overlap jobs

Can someone help me move forward with this please?

Thanks heaps

Saila

brianwalenz commented 5 years ago

Look for errors in correction/1-overlapper/mhap*out for the failing jobs. Paste in mhap.000087.out, or the last 50 or so lines if it's huge.

skoren commented 5 years ago

In addition to that, make sure you're not out of quota space and remove any files named *WORKING and *mhap in correction/1-overlapper/results folder.

biowackysci commented 5 years ago

Hello again this is the output from correction/1-overlapper/mhap.000087.out 

Found perl:
   /usr/bin/perl
   This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux-thread-multi

Found java:
   /usr/local/EasyBuild/software/Java/1.8.0_172/bin/java
   java version "1.8.0_172"

Found canu:
   /usr/local/EasyBuild/software/canu/1.8-intel-2018a/bin/canu
   Canu 1.8

Running job 87 based on command line options.
Fetch blocks/000025.dat
Fetch blocks/000026.dat
Fetch blocks/000027.dat
Fetch blocks/000028.dat
Fetch blocks/000029.dat
Fetch blocks/000030.dat
Fetch blocks/000031.dat
Fetch blocks/000032.dat
Fetch blocks/000033.dat
Fetch blocks/000034.dat
Fetch blocks/000035.dat
Fetch blocks/000036.dat
Fetch blocks/000037.dat
Fetch blocks/000038.dat
Fetch blocks/000039.dat

Running block 000024 in query 000087

read2344062 read1528894 0.196038 19.000000 0  9263 12483 15297 0 156 3150 7755

INVALID LENGTHS read 2344062 (len 33554) and read 1528894 (len 51638) lengths 15297 and 7755

Looks like the error is "INVALID LENGTHS read 2344062 (len 33554) and read 1528894 (len 51638) lengths 15297 and 7755"

How do i proceed from here?

Thanks Saila

biowackysci commented 5 years ago

Hello skoren, I found the mhap files but there are no WORKING files. Does that mean something ? Thanks SAILA

skoren commented 5 years ago

The mhap files are corrupt, that indicates either out of space or disk corruption. Make sure you're not hitting a quota. Try removing the *mhap files and re-running one of the jobs by hand (e.g. mhap.sh 87 inside the 1-overlapper folder).

biowackysci commented 5 years ago

Thanks for that, will try to run one of the jobs. Meanwhile i found a different error in a different mhap file - mhap.000090.out file which is 

Found perl:
   /usr/bin/perl
   This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux-thread-multi

Found java:
   /usr/local/EasyBuild/software/Java/1.8.0_172/bin/java
   java version "1.8.0_172"

Found canu:
   /usr/local/EasyBuild/software/canu/1.8-intel-2018a/bin/canu
   Canu 1.8

Running job 90 based on command line options.
Fetch blocks/000026.dat
Fetch blocks/000027.dat
Fetch blocks/000028.dat
Fetch blocks/000029.dat
Fetch blocks/000030.dat
Fetch blocks/000031.dat
Fetch blocks/000032.dat
Fetch blocks/000033.dat
Fetch blocks/000034.dat
Fetch blocks/000035.dat
Fetch blocks/000036.dat
Fetch blocks/000037.dat
Fetch blocks/000038.dat
Fetch blocks/000039.dat
Fetch blocks/000040.dat

Running block 000025 in query 000090

./mhap.sh: line 1061: 246916 Segmentation fault      (core dumped) $bin/mhapConvert -S ../../canu.seqStore -o ./results/$qry.mhap.ovb.WORKING ./results/$qry.mhap

I dont know if this is a result of the space issue or if there is something else going on here?

Thanks in advance Saila

skoren commented 5 years ago

That looks like the same root cause, which is a corrupt mhap file.

biowackysci commented 5 years ago

Thanks for that, I have a question. You said "remove the *mhap files". So do I go ahead and remove all the mhap files or just the mhap.sh 87 and try running that job alone? Also, can you please recommend a command to run the job, as I am new to command line?.

Thanks a lot for your help Saila

skoren commented 5 years ago

You can remove all of them, to run a single job go into the 1-overlapper folder and run sh mhap.sh 87. Make sure to do this on a compute node with at least 50gb reserved not the head node of your cluster.

biowackysci commented 5 years ago

When i run the job this is what I get 

[vici86y@dev1 1-overlapper]$ sh mhap.sh 87

Found perl:
   /usr/bin/perl
   This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux-thr                                                                                                                                                             ead-multi

Found java:
   /usr/local/EasyBuild/software/Java/1.8.0_172/bin/java
   java version "1.8.0_172"

Found canu:
   /usr/local/EasyBuild/software/canu/1.8-intel-2018a/bin/canu
   /usr/local/EasyBuild/software/canu/1.8-intel-2018a/bin/sqStoreCreate: error w                                                                                                                                                             hile loading shared libraries: libiomp5.so: cannot open shared object file: No s                                                                                                                                                             uch file or directory

Running job 87 based on command line options.
Fetch blocks/000025.dat
Fetch blocks/000026.dat
Fetch blocks/000027.dat
Fetch blocks/000028.dat
Fetch blocks/000029.dat
Fetch blocks/000030.dat
Fetch blocks/000031.dat
Fetch blocks/000032.dat
Fetch blocks/000033.dat
Fetch blocks/000034.dat
Fetch blocks/000035.dat
Fetch blocks/000036.dat
Fetch blocks/000037.dat
Fetch blocks/000038.dat
Fetch blocks/000039.dat

Running block 000024 in query 000087

/usr/local/EasyBuild/software/canu/1.8-intel-2018a/bin/mhapConvert: error while                                                                                                                                                              loading shared libraries: libiomp5.so: cannot open shared object file: No such f                                                                                                                                                             ile or directory

Thanks Saila

skoren commented 5 years ago

You are running on a different system than you compiled canu on, this is a system issue. You'll have to figure out what you need to do in the environment to make it run (however it was being submitted gives the right environment). Also, are you sure the mhap files are removed, what does ls results report?

biowackysci commented 5 years ago

Thanks, I ran the mhap 87 file and this is what I got [vici86y@dev1 1-overlapper]$ sh mhap.sh 87

Found perl:
   /usr/bin/perl
   This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux-thread-multi

Found java:
   /usr/local/EasyBuild/software/Java/1.8.0_172/bin/java
   java version "1.8.0_172"

Found canu:
   /usr/local/EasyBuild/software/canu/1.8-intel-2018a/bin/canu
   Canu 1.8

Running job 87 based on command line options.
Fetch blocks/000025.dat
Fetch blocks/000026.dat
Fetch blocks/000027.dat
Fetch blocks/000028.dat
Fetch blocks/000029.dat
Fetch blocks/000030.dat
Fetch blocks/000031.dat
Fetch blocks/000032.dat
Fetch blocks/000033.dat
Fetch blocks/000034.dat
Fetch blocks/000035.dat
Fetch blocks/000036.dat
Fetch blocks/000037.dat
Fetch blocks/000038.dat
Fetch blocks/000039.dat

Running block 000024 in query 000087

read2344062 read1528894 0.196038 19.000000 0  9263 12483 15297 0 156 3150 7755

INVALID LENGTHS read 2344062 (len 33554) and read 1528894 (len 51638) lengths 15297 and 7755

Can you help me go further from here? Thanks Saila

skoren commented 5 years ago

That output is incomplete, it doesn't look like the mhap files got removed since it's the exact same error you got originally, what's in your results folder as I asked before?

biowackysci commented 5 years ago

The results folder has the following files .mhap.ovb.WORKING .ovb .oc

skoren commented 5 years ago

That doesn't make sense, what happened to all your successful jobs you had before (from 1-86), are you in the same folder you were running the assembly before?

Do an ls /group/home/vici86y/canu-1.8/correction, ls /group/home/vici86y/canu-1.8/correction/1-overlapper, ls /group/home/vici86y/canu-1.8/1-overlapper/results, and ls /group/home/vici86y/canu-1.8/blocks.

biowackysci commented 5 years ago

I delete all the mhap files from the 1-overlapper file.

[vici86y@dev1 1-overlapper]$ ls /group/home/vici86y/canu-1.8/correction 0-mercounts 1-overlapper

[vici86y@dev1 1-overlapper]$ ls /group/home/vici86y/canu-1.8/correction/1-overlapper blocks queries results

[vici86y@dev1 1-overlapper]$ ls /group/home/vici86y/canu-1.8/1-overlapper/results ls: cannot access /group/home/vici86y/canu-1.8/1-overlapper/results: No such file or directory

ls /group/home/vici86y/canu-1.8/blocks ls: cannot access /group/home/vici86y/canu-1.8/blocks: No such file or directory

This is what i get

Thanks Saila

skoren commented 5 years ago

Sorry it should be /group/home/vici86y/canu-1.8/correction/1-overlapper/blocks/* and /group/home/vici86y/canu-1.8/correction/1-overlapper/results/*. The mhap files in 1-overlapper don't matter, those are just logs but there should still be files named precompute.*.out. Is your ls aliased to show only folders?

The mhap files you want to remove are in results are the ones being output by Canu so if those are still there that would explain why your job isn't running still.

biowackysci commented 5 years ago

Hello Koren, This is what i got 

[vici86y@dev1 1-overlapper]$ ls /group/home/vici86y/canu-1.8/blocks*
ls: cannot access /group/home/vici86y/canu-1.8/blocks*: No such file or directory
[vici86y@dev1 1-overlapper]$ ls /group/home/vici86y/canu-1.8/correction/1-overlapper/blocks.*
ls: cannot access /group/home/vici86y/canu-1.8/correction/1-overlapper/blocks.*: No such file or directory
[vici86y@dev1 1-overlapper]$ ls /group/home/vici86y/canu-1.8/correction/1-overlapper/results/*
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000001.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000053.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000105.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000001.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000053.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000105.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000002.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000054.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000106.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000002.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000054.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000106.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000003.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000055.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000107.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000003.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000055.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000107.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000004.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000056.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000108.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000004.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000056.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000108.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000005.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000057.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000109.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000005.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000057.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000109.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000006.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000058.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000110.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000006.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000058.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000110.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000007.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000059.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000111.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000007.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000059.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000111.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000008.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000060.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000112.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000008.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000060.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000112.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000009.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000061.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000113.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000009.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000061.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000113.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000010.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000062.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000114.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000010.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000062.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000114.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000011.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000063.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000115.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000011.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000063.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000115.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000012.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000064.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000116.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000012.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000064.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000116.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000013.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000065.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000117.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000013.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000065.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000117.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000014.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000066.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000118.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000014.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000066.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000118.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000015.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000067.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000119.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000015.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000067.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000119.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000016.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000068.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000120.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000016.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000068.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000120.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000017.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000069.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000121.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000017.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000069.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000121.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000018.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000070.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000122.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000018.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000070.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000122.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000019.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000071.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000123.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000019.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000071.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000123.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000020.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000072.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000124.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000020.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000072.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000124.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000021.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000073.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000125.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000021.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000073.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000125.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000022.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000074.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000126.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000022.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000074.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000126.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000023.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000075.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000127.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000023.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000075.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000127.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000024.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000076.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000128.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000024.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000076.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000128.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000025.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000077.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000129.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000025.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000077.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000129.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000026.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000078.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000130.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000026.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000078.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000130.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000027.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000079.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000131.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000027.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000079.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000131.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000028.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000080.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000132.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000028.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000080.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000132.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000029.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000081.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000133.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000029.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000081.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000133.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000030.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000082.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000134.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000030.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000082.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000134.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000031.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000083.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000135.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000031.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000083.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000135.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000032.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000084.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000136.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000032.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000084.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000136.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000033.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000085.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000137.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000033.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000085.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000137.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000034.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000086.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000138.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000034.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000086.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000138.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000035.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000087.mhap              /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000139.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000035.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000087.mhap.ovb.WORKING  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000139.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000036.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000088.mhap              /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000140.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000036.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000088.mhap.ovb.WORKING  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000140.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000037.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000089.mhap              /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000141.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000037.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000089.mhap.ovb.WORKING  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000141.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000038.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000090.mhap              /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000142.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000038.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000090.mhap.ovb.WORKING  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000142.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000039.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000091.mhap              /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000143.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000039.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000091.mhap.ovb.WORKING  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000143.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000040.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000092.mhap              /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000144.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000040.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000092.mhap.ovb.WORKING  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000144.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000041.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000093.mhap              /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000145.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000041.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000093.mhap.ovb.WORKING  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000145.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000042.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000094.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000146.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000042.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000094.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000146.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000043.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000095.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000147.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000043.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000095.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000147.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000044.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000096.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000148.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000044.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000096.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000148.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000045.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000097.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000149.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000045.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000097.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000149.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000046.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000098.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000150.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000046.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000098.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000150.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000047.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000099.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000151.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000047.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000099.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000151.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000048.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000100.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000152.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000048.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000100.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000152.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000049.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000101.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000153.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000049.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000101.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000153.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000050.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000102.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000154.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000050.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000102.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000154.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000051.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000103.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000155.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000051.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000103.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000155.ovb
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000052.oc   /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000104.oc                /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000156.oc
/group/home/vici86y/canu-1.8/correction/1-overlapper/results/000052.ovb  /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000104.ovb               /group/home/vici86y/canu-1.8/correction/1-overlapper/results/000156.ovb
skoren commented 5 years ago

Right, so it seems your ls is set to filter unless you add the * on the end. You have lots of mhap and WORKING files in results. Remove those before re-trying the failed job.

biowackysci commented 5 years ago

I think i made a mistake and deleted everything in the results file and now this happens sh: mhap.sh: No such file or directory [vici86y@dev1 correction]$ cd 1-overlapper/ [vici86y@dev1 1-overlapper]$ sh mhap.sh 87 sh: mhap.sh: No such file or directory

skoren commented 5 years ago

That file can be regenerated by Canu so it's not a big deal. Make sure you actually removed the .mhap and .WORKING files in 1-overlapper/results/:

rm /group/home/vici86y/canu-1.8/correction/1-overlapper/results/*.WORKING
rm /group/home/vici86y/canu-1.8/correction/1-overlapper/results/*.mhap
ls /group/home/vici86y/canu-1.8/correction/1-overlapper/results/*

Then re-run your original Canu command. It will re-create the shell script and run the failed jobs again.

biowackysci commented 5 years ago

okay will do and let you know how it goes