search

marbl / verkko

Telomere-to-telomere assembly of accurate long reads (PacBio HiFi, Oxford Nanopore Duplex, HERRO corrected Oxford Nanopore Simplex) and Oxford Nanopore ultra-long reads.
305 stars 30 forks source link

SLURM sbatch Error #295

Closed nickgladman closed 1 week ago

nickgladman commented 1 month ago

Hello,

Thanks again for your great resource!

I am running into a --slurm error that I think may be slightly different than the other --slurm issues. The run seems to almost finish then yields the following error code:

sbatch: error: Batch job submission failed: Requested time limit is invalid (missing or exceeds some limit) sbatch: error: Batch job submission failed: Requested time limit is invalid (missing or exceeds some limit) Error submitting jobscript (exit code 1):

I am submitting the batch script on a medium length partition (7 day time limit) and all the automated scripts are pushed out on the short term partition (48 hours). This whole run took ~ 8 hours before it failed.

Here is my script. I'm using verkko v 2.1

#!/bin/bash
#SBATCH --job-name="verkko"
#SBATCH -p medium
#SBATCH -N 1
#SBATCH -n 1
#SBATCH --cpus-per-task 2
#SBATCH --mem=120GB
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH -o "stdout.%j.%N"
#SBATCH -e "stderr.%j.%N"
date

# activate environment
module load miniconda/24.7.1-2
source activate ~/conda/envs/verkko_2.1

srun verkko \
-d asm_seed_max_length100000 \
--seed-max-length 100000 \
--slurm \
--hifi *.fastq.gz \
--nano ont_reads_q7.fasta.gz \

Here is the end of the log file:

Submitted job 6 with external jobid '13371552'.
[Mon Oct 21 16:37:36 2024]
Finished job 6.
221 of 240 steps (92%) done
Select jobs to execute...

[Mon Oct 21 16:37:36 2024]
localrule correctHiFi:
    input: 0-correction/hifi-corrected.fasta.gz
    output: hifi-corrected.fasta.gz
    jobid: 5
    reason: Missing output files: hifi-corrected.fasta.gz; Input files updated by another job: 0-correction/hifi-corrected.fasta.gz
    resources: mem_mb=32094, mem_mib=30608, disk_mb=32094, disk_mib=30608, tmpdir=/local/bgfs/usr/13363943, n_cpus=1, mem_gb=1, time_h=1

[Mon Oct 21 16:37:36 2024]
Finished job 5.
222 of 240 steps (92%) done
Select jobs to execute...

[Mon Oct 21 16:37:36 2024]
rule buildGraph:
    input: hifi-corrected.fasta.gz
    output: 1-buildGraph/hifi-resolved.gfa, 1-buildGraph/hifi_nodecov.csv, 1-buildGraph/paths.gaf
    log: 1-buildGraph/buildGraph.err
    jobid: 4
    reason: Missing output files: 1-buildGraph/paths.gaf, 1-buildGraph/hifi-resolved.gfa, 1-buildGraph/hifi_nodecov.csv; Input files updated by another job: hifi-corrected.fasta.gz
    threads: 4
    resources: mem_mb=32094, mem_mib=30608, disk_mb=32094, disk_mib=30608, tmpdir=<TBD>, job_id=1, n_cpus=4, mem_gb=39, time_h=72

sbatch: error: Batch job submission failed: Requested time limit is invalid (missing or exceeds some limit)
sbatch: error: Batch job submission failed: Requested time limit is invalid (missing or exceeds some limit)
Error submitting jobscript (exit code 1):

Trying to restart job 4.
Select jobs to execute...

[Mon Oct 21 16:38:06 2024]
rule buildGraph:
    input: hifi-corrected.fasta.gz
    output: 1-buildGraph/hifi-resolved.gfa, 1-buildGraph/hifi_nodecov.csv, 1-buildGraph/paths.gaf
    log: 1-buildGraph/buildGraph.err
    jobid: 4
    reason: Missing output files: 1-buildGraph/paths.gaf, 1-buildGraph/hifi-resolved.gfa, 1-buildGraph/hifi_nodecov.csv; Input files updated by another job: hifi-corrected.fasta.gz
    threads: 4
    resources: mem_mb=32094, mem_mib=30608, disk_mb=32094, disk_mib=30608, tmpdir=<TBD>, job_id=1, n_cpus=4, mem_gb=49, time_h=108

sbatch: error: Batch job submission failed: Requested time limit is invalid (missing or exceeds some limit)
sbatch: error: Batch job submission failed: Requested time limit is invalid (missing or exceeds some limit)
Error submitting jobscript (exit code 1):

Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-10-21T092932.102940.snakemake.log
skoren commented 1 month ago

You can pass through extra options to snakemake to let your scheduler know you want the longer queue, see https://github.com/marbl/verkko?tab=readme-ov-file#running-on-a-grid, specifically customizing grid requests:

--snakeopts '--cluster "./slurm-sge-submit.sh {threads} {resources.mem_gb} {resources.time_h} {rulename} {resources.job_id} --partition=medium"'
nickgladman commented 1 month ago

Thank you! That resolved run issue. Unfortunately it is now triggering another error, but I don't think it's related to the --partition=medium fix. I think it might be related #84, #205, or #131? I'll post it here, but I can open another issue if that would be cleaner:

Submitted job 6 with external jobid '13375555'.
[Tue Oct 22 18:08:53 2024]
Finished job 6.
222 of 431 steps (52%) done
Select jobs to execute...

[Tue Oct 22 18:08:53 2024]
localrule correctHiFi:
    input: 0-correction/hifi-corrected.fasta.gz
    output: hifi-corrected.fasta.gz
    jobid: 5
    reason: Missing output files: hifi-corrected.fasta.gz; Input files updated by another job: 0-correction/hifi-corrected.fasta.gz
    resources: mem_mb=32094, mem_mib=30608, disk_mb=32094, disk_mib=30608, tmpdir=/local/bgfs/nicholas.gladman/13373597, n_cpus=1, mem_gb=1, time_h=1

[Tue Oct 22 18:08:53 2024]
Finished job 5.
223 of 431 steps (52%) done
Select jobs to execute...

[Tue Oct 22 18:08:53 2024]
rule buildGraph:
    input: hifi-corrected.fasta.gz
    output: 1-buildGraph/hifi-resolved.gfa, 1-buildGraph/hifi_nodecov.csv, 1-buildGraph/paths.gaf
    log: 1-buildGraph/buildGraph.err
    jobid: 4
    reason: Missing output files: 1-buildGraph/hifi_nodecov.csv, 1-buildGraph/paths.gaf, 1-buildGraph/hifi-resolved.gfa; Input files updated by another job: hifi-corrected.fasta.gz
    threads: 4
    resources: mem_mb=32094, mem_mib=30608, disk_mb=32094, disk_mib=30608, tmpdir=<TBD>, job_id=1, n_cpus=4, mem_gb=39, time_h=72

Submitted job 4 with external jobid '13375581'.
[Tue Oct 22 21:54:03 2024]
Error in rule buildGraph:
    jobid: 4
    input: hifi-corrected.fasta.gz
    output: 1-buildGraph/hifi-resolved.gfa, 1-buildGraph/hifi_nodecov.csv, 1-buildGraph/paths.gaf
    log: 1-buildGraph/buildGraph.err (check log file(s) for error details)
    shell:

cd 1-buildGraph

cat > ./buildGraph.sh <<EOF
#!/bin/sh
set -e

#  Figure out what reads we're going to use.  If correction is enabled, the
#  reads are in a single file that is a relative path away from us; but if it
#  is not enabled, the reads are in possibly multiple files that are at an
#  absolute path.
#
iopt=""

echo "Building graph with reads:"

if [ "True" = "True" ] ; then
  echo "  ../hifi-corrected.fasta.gz"
  iopt="-i ../hifi-corrected.fasta.gz"
else
  for rf in /project/_genomics/1.hifi_reads.bam.fastq.gz /project/2.hifi_reads.bam.fastq.gz ; do
    echo "  \$rf"
    iopt="\$iopt -i \$rf"
  done
fi

echo ""
echo ""

#  Build the graph.
#    (rule build_graph in the original)
#
/home/nicholas.gladman/software/.conda/envs/verkko_2.1/lib/verkko/bin/MBG \\
  \$iopt \\
  -t 4 \\
  -k 1001 \\
  -r 15000 -R 4000 \\
  -w 100 \\
  --kmer-abundance 1 \\
  --unitig-abundance 2 \\
  --error-masking=collapse-msat \\
  --output-sequence-paths ../1-buildGraph/paths.gaf \\
  --hpc-variant-onecopy-coverage 25 \\
  --do-unsafe-guesswork-resolutions \\
  --out ../1-buildGraph/hifi-resolved.gfa

#  Find coverage.
#    (rule hifi_coverage_csv in the original Snakefile)
#    (hifi-resolved.gfa -> hifi_nodecov.csv)
#    (\$6 != "" is from 9e31a602925a477a7e52c277eda143e7bd20e52b)
#
awk 'BEGIN \\
     { \\
        FS="[ \t]+"; OFS="\t"; \\
        print "node", "length", "coverage"; \\
     } \\
     \$1 == "S" \\
     { \\
        if (\$6 != "") {
          \$4 = \$6;
        }
        print \$2, length(\$3), substr(\$4, 6); \\
     }' \\
< ../1-buildGraph/hifi-resolved.gfa \\
> ../1-buildGraph/hifi_nodecov.csv
EOF

chmod +x ./buildGraph.sh

./buildGraph.sh > ../1-buildGraph/buildGraph.err 2>&1

        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
    cluster_jobid: 13375581

Error executing rule buildGraph on cluster (jobid: 4, external: 13375581, jobscript: /project/asm/.snakemake/tmp.gkqxjow0/verkko.buildGraph.4.sh). For error details see the cluster log and the log files of the involved rule(s).
Trying to restart job 4.
Select jobs to execute...

[Tue Oct 22 21:54:03 2024]
rule buildGraph:
    input: hifi-corrected.fasta.gz
    output: 1-buildGraph/hifi-resolved.gfa, 1-buildGraph/hifi_nodecov.csv, 1-buildGraph/paths.gaf
    log: 1-buildGraph/buildGraph.err
    jobid: 4
    reason: Missing output files: 1-buildGraph/hifi_nodecov.csv, 1-buildGraph/paths.gaf, 1-buildGraph/hifi-resolved.gfa; Input files updated by another job: hifi-corrected.fasta.gz
    threads: 4
    resources: mem_mb=32094, mem_mib=30608, disk_mb=32094, disk_mib=30608, tmpdir=<TBD>, job_id=1, n_cpus=4, mem_gb=49, time_h=108

Submitted job 4 with external jobid '13375920'.
[Wed Oct 23 01:54:14 2024]
Error in rule buildGraph:
    jobid: 4
    input: hifi-corrected.fasta.gz
    output: 1-buildGraph/hifi-resolved.gfa, 1-buildGraph/hifi_nodecov.csv, 1-buildGraph/paths.gaf
    log: 1-buildGraph/buildGraph.err (check log file(s) for error details)
    shell:

cd 1-buildGraph

cat > ./buildGraph.sh <<EOF
#!/bin/sh
set -e

#  Figure out what reads we're going to use.  If correction is enabled, the
#  reads are in a single file that is a relative path away from us; but if it
#  is not enabled, the reads are in possibly multiple files that are at an
#  absolute path.
#
iopt=""

echo "Building graph with reads:"

if [ "True" = "True" ] ; then
  echo "  ../hifi-corrected.fasta.gz"
  iopt="-i ../hifi-corrected.fasta.gz"
else
  for rf in /project/1.hifi_reads.bam.fastq.gz /project/2.hifi_reads.bam.fastq.gz ; do
    echo "  \$rf"
    iopt="\$iopt -i \$rf"
  done
fi

echo ""
echo ""

#  Build the graph.
#    (rule build_graph in the original)
#
/home/nicholas.gladman/software/.conda/envs/verkko_2.1/lib/verkko/bin/MBG \\
  \$iopt \\
  -t 4 \\
  -k 1001 \\
  -r 15000 -R 4000 \\
  -w 100 \\
  --kmer-abundance 1 \\
  --unitig-abundance 2 \\
  --error-masking=collapse-msat \\
  --output-sequence-paths ../1-buildGraph/paths.gaf \\
  --hpc-variant-onecopy-coverage 25 \\
  --do-unsafe-guesswork-resolutions \\
  --out ../1-buildGraph/hifi-resolved.gfa

#  Find coverage.
#    (rule hifi_coverage_csv in the original Snakefile)
#    (hifi-resolved.gfa -> hifi_nodecov.csv)
#    (\$6 != "" is from 9e31a602925a477a7e52c277eda143e7bd20e52b)
#
awk 'BEGIN \\
     { \\
        FS="[ \t]+"; OFS="\t"; \\
        print "node", "length", "coverage"; \\
     } \\
     \$1 == "S" \\
     { \\
        if (\$6 != "") {
          \$4 = \$6;
        }
        print \$2, length(\$3), substr(\$4, 6); \\
     }' \\
< ../1-buildGraph/hifi-resolved.gfa \\
> ../1-buildGraph/hifi_nodecov.csv
EOF

chmod +x ./buildGraph.sh

./buildGraph.sh > ../1-buildGraph/buildGraph.err 2>&1

        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
    cluster_jobid: 13375920

Error executing rule buildGraph on cluster (jobid: 4, external: 13375920, jobscript: /project/asm/.snakemake/tmp.gkqxjow0/verkko.buildGraph.4.sh). For error details see the cluster log and the log files of the involved rule(s).
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-10-22T112409.242824.snakemake.log
skoren commented 1 month ago

What does the buildGraph.err log have in it? It's possible it is an MBG bug but might also be fixed by a more recent version of verkko than 2.1.

nickgladman commented 1 month ago

Here is the full error log:

Building graph with reads:
  ../hifi-corrected.fasta.gz

MBG Branch  commit  
Parameters: k=1001,w=100,a=1,u=2,t=4,r=15000,R=4000,hpcvariantcov=25,errormasking=collapse-msat,endkmers=no,blunt=no,keepgaps=no,guesswork=yes,copycountfilter=no,onlylocal=no,filterwithinunitig=yes,cleaning=yes,cache=no
Collecting hpc variant k-mers
Reading sequences from ../hifi-corrected.fasta.gz
618453111 total selected k-mers in reads
64780820 distinct selected k-mers in reads
Reading sequences from ../hifi-corrected.fasta.gz
Collecting selected k-mers
Reading sequences from ../hifi-corrected.fasta.gz
618453111 total selected k-mers in reads
64781077 distinct selected k-mers in reads
Unitigifying
Filtering by unitig coverage
4846937 distinct selected k-mers in unitigs after filtering
Getting read paths
Reading sequences from ../hifi-corrected.fasta.gz
Resolving unitigs
1696694 unitigs before resolving
estimated average coverage 9.17274
4813825 raw read paths
4844567 raw read paths
3504958 read paths
removed 54478 tips
removed 208 crosslinks
removed 1556 nodes and 0 edges
try resolve k=1001, replaced 33814 nodes with 134303 nodes, unitigified 226917 nodes to 83659 nodes
removed 851 nodes and 29 edges
removed 14031 tips and 105 edges
try resolve k=1002, replaced 663 nodes with 2644 nodes, unitigified 4709 nodes to 1800 nodes
removed 17 nodes and 0 edges
removed 20 tips and 1 edges
try resolve k=1003, replaced 770 nodes with 3022 nodes, unitigified 5410 nodes to 2089 nodes
removed 21 nodes and 0 edges
removed 9 tips and 4 edges
try resolve k=1004, replaced 764 nodes with 3220 nodes, unitigified 5712 nodes to 2281 nodes
removed 14 nodes and 0 edges
removed 9 tips and 2 edges
try resolve k=1005, replaced 823 nodes with 3283 nodes, unitigified 5788 nodes to 2272 nodes
removed 15 nodes and 0 edges
removed 10 tips and 3 edges
try resolve k=1006, replaced 816 nodes with 3359 nodes, unitigified 5929 nodes to 2377 nodes
removed 24 nodes and 0 edges
removed 8 tips and 1 edges
try resolve k=1007, replaced 752 nodes with 3034 nodes, unitigified 5326 nodes to 2094 nodes
removed 17 nodes and 0 edges
removed 3 tips and 2 edges
try resolve k=1008, replaced 812 nodes with 3199 nodes, unitigified 5579 nodes to 2220 nodes
removed 19 nodes and 2 edges
removed 9 tips and 5 edges
try resolve k=1009, replaced 760 nodes with 3081 nodes, unitigified 5442 nodes to 2146 nodes
removed 14 nodes and 0 edges
removed 4 tips and 2 edges
try resolve k=1010, replaced 785 nodes with 3203 nodes, unitigified 5624 nodes to 2235 nodes
removed 24 nodes and 0 edges
removed 6 tips and 2 edges
try resolve k=1011, replaced 761 nodes with 3150 nodes, unitigified 5545 nodes to 2224 nodes
removed 18 nodes and 0 edges
removed 3 tips and 5 edges
try resolve k=1012, replaced 830 nodes with 3291 nodes, unitigified 5702 nodes to 2287 nodes
removed 16 nodes and 0 edges
removed 4 tips and 5 edges
try resolve k=1013, replaced 755 nodes with 2971 nodes, unitigified 5170 nodes to 2045 nodes
removed 11 nodes and 0 edges
removed 8 tips and 2 edges
try resolve k=1014, replaced 783 nodes with 3021 nodes, unitigified 5267 nodes to 2038 nodes
removed 14 nodes and 0 edges
removed 2 tips and 2 edges
try resolve k=1015, replaced 830 nodes with 3224 nodes, unitigified 5642 nodes to 2193 nodes
removed 13 nodes and 2 edges
removed 2 tips and 1 edges
try resolve k=1016, replaced 797 nodes with 3066 nodes, unitigified 5212 nodes to 2106 nodes
removed 14 nodes and 2 edges
removed 8 tips and 0 edges
try resolve k=1017, replaced 820 nodes with 3128 nodes, unitigified 5394 nodes to 2130 nodes
removed 11 nodes and 0 edges
removed 5 tips and 4 edges
try resolve k=1018, replaced 835 nodes with 3357 nodes, unitigified 5861 nodes to 2391 nodes
removed 16 nodes and 0 edges
removed 6 tips and 4 edges
try resolve k=1019, replaced 819 nodes with 3312 nodes, unitigified 5749 nodes to 2339 nodes
removed 25 nodes and 0 edges
removed 5 tips and 0 edges
try resolve k=1020, replaced 815 nodes with 3177 nodes, unitigified 5452 nodes to 2203 nodes
removed 24 nodes and 0 edges
removed 4 tips and 3 edges
try resolve k=1021, replaced 821 nodes with 3134 nodes, unitigified 5402 nodes to 2162 nodes
removed 18 nodes and 0 edges
removed 3 tips and 3 edges
try resolve k=1022, replaced 772 nodes with 2890 nodes, unitigified 4966 nodes to 1963 nodes
removed 15 nodes and 0 edges
removed 4 tips and 1 edges
try resolve k=1023, replaced 780 nodes with 2916 nodes, unitigified 5000 nodes to 1957 nodes
removed 11 nodes and 1 edges
removed 8 tips and 2 edges
try resolve k=1024, replaced 803 nodes with 3071 nodes, unitigified 5175 nodes to 2094 nodes
removed 14 nodes and 2 edges
removed 8 tips and 1 edges
try resolve k=1025, replaced 848 nodes with 3217 nodes, unitigified 5490 nodes to 2230 nodes
removed 17 nodes and 2 edges
removed 1 tips and 1 edges
try resolve k=1026, replaced 778 nodes with 2975 nodes, unitigified 5048 nodes to 2029 nodes
removed 12 nodes and 0 edges
removed 6 tips and 2 edges
try resolve k=1027, replaced 817 nodes with 3135 nodes, unitigified 5375 nodes to 2157 nodes
removed 25 nodes and 2 edges
removed 3 tips and 6 edges
try resolve k=1028, replaced 784 nodes with 3107 nodes, unitigified 5309 nodes to 2137 nodes
removed 14 nodes and 0 edges
removed 12 tips and 1 edges
try resolve k=1029, replaced 825 nodes with 3222 nodes, unitigified 5522 nodes to 2224 nodes
removed 16 nodes and 0 edges
removed 9 tips and 1 edges
try resolve k=1030, replaced 779 nodes with 2954 nodes, unitigified 5056 nodes to 2000 nodes
removed 18 nodes and 3 edges
removed 4 tips and 3 edges
try resolve k=1031, replaced 751 nodes with 2956 nodes, unitigified 5056 nodes to 2019 nodes
removed 14 nodes and 0 edges
removed 8 tips and 3 edges
try resolve k=1032, replaced 845 nodes with 3239 nodes, unitigified 5473 nodes to 2219 nodes
removed 12 nodes and 0 edges
removed 5 tips and 3 edges
try resolve k=1033, replaced 842 nodes with 3260 nodes, unitigified 5565 nodes to 2249 nodes
removed 12 nodes and 0 edges
removed 4 tips and 4 edges
try resolve k=1034, replaced 825 nodes with 3260 nodes, unitigified 5519 nodes to 2258 nodes
removed 12 nodes and 0 edges
removed 7 tips and 1 edges
try resolve k=1035, replaced 812 nodes with 3101 nodes, unitigified 5245 nodes to 2106 nodes
removed 8 nodes and 0 edges
removed 6 tips and 3 edges
try resolve k=1036, replaced 800 nodes with 3059 nodes, unitigified 5179 nodes to 2103 nodes
removed 16 nodes and 0 edges
removed 3 tips and 1 edges
try resolve k=1037, replaced 793 nodes with 3003 nodes, unitigified 5074 nodes to 2054 nodes
removed 20 nodes and 0 edges
removed 7 tips and 1 edges
try resolve k=1038, replaced 793 nodes with 2987 nodes, unitigified 5085 nodes to 2039 nodes
removed 8 nodes and 0 edges
removed 8 tips and 6 edges
try resolve k=1039, replaced 777 nodes with 2943 nodes, unitigified 4949 nodes to 2003 nodes
removed 8 nodes and 0 edges
removed 6 tips and 1 edges
try resolve k=1040, replaced 768 nodes with 2929 nodes, unitigified 4948 nodes to 2023 nodes
removed 15 nodes and 4 edges
removed 4 tips and 2 edges
try resolve k=1041, replaced 794 nodes with 3129 nodes, unitigified 5324 nodes to 2178 nodes
removed 11 nodes and 0 edges
removed 11 tips and 2 edges
try resolve k=1042, replaced 825 nodes with 3100 nodes, unitigified 5196 nodes to 2147 nodes
removed 14 nodes and 0 edges
removed 4 tips and 1 edges
try resolve k=1043, replaced 772 nodes with 2906 nodes, unitigified 4909 nodes to 2004 nodes
removed 15 nodes and 0 edges
removed 4 tips and 3 edges
try resolve k=1044, replaced 805 nodes with 3181 nodes, unitigified 5352 nodes to 2227 nodes
removed 12 nodes and 2 edges
removed 3 tips and 0 edges
try resolve k=1045, replaced 820 nodes with 3070 nodes, unitigified 5155 nodes to 2061 nodes
removed 10 nodes and 0 edges
removed 5 tips and 3 edges
try resolve k=1046, replaced 799 nodes with 3049 nodes, unitigified 5061 nodes to 2088 nodes
removed 11 nodes and 0 edges
removed 3 tips and 2 edges
try resolve k=1047, replaced 749 nodes with 2799 nodes, unitigified 4706 nodes to 1921 nodes
removed 18 nodes and 0 edges
removed 6 tips and 2 edges
try resolve k=1048, replaced 780 nodes with 2933 nodes, unitigified 4928 nodes to 1968 nodes
removed 17 nodes and 0 edges
removed 6 tips and 1 edges
try resolve k=1049, replaced 753 nodes with 2817 nodes, unitigified 4719 nodes to 1929 nodes
removed 11 nodes and 0 edges
removed 6 tips and 0 edges
try resolve k=1050, replaced 767 nodes with 2775 nodes, unitigified 4621 nodes to 1892 nodes
removed 11 nodes and 0 edges
removed 3 tips and 0 edges
try resolve k=1051, replaced 798 nodes with 2900 nodes, unitigified 4843 nodes to 1951 nodes
removed 8 nodes and 0 edges
removed 12 tips and 3 edges
try resolve k=1052, replaced 759 nodes with 2958 nodes, unitigified 4961 nodes to 2055 nodes
removed 13 nodes and 0 edges
removed 10 tips and 1 edges
try resolve k=1053, replaced 772 nodes with 2857 nodes, unitigified 4825 nodes to 1950 nodes
removed 8 nodes and 0 edges
removed 4 tips and 3 edges
try resolve k=1054, replaced 811 nodes with 2981 nodes, unitigified 4938 nodes to 2065 nodes
removed 6 nodes and 0 edges
removed 3 tips and 2 edges
try resolve k=1055, replaced 814 nodes with 2929 nodes, unitigified 4846 nodes to 1951 nodes
removed 16 nodes and 0 edges
removed 8 tips and 2 edges
try resolve k=1056, replaced 698 nodes with 2668 nodes, unitigified 4463 nodes to 1846 nodes
removed 10 nodes and 0 edges
removed 3 tips and 2 edges
try resolve k=1057, replaced 713 nodes with 2685 nodes, unitigified 4536 nodes to 1846 nodes
removed 7 nodes and 0 edges
removed 4 tips and 3 edges
try resolve k=1058, replaced 713 nodes with 2806 nodes, unitigified 4688 nodes to 1956 nodes
removed 7 nodes and 0 edges
removed 4 tips and 0 edges
try resolve k=1059, replaced 791 nodes with 3040 nodes, unitigified 5104 nodes to 2106 nodes
removed 14 nodes and 1 edges
removed 7 tips and 4 edges
try resolve k=1060, replaced 768 nodes with 2785 nodes, unitigified 4563 nodes to 1892 nodes
removed 13 nodes and 0 edges
removed 4 tips and 1 edges
try resolve k=1061, replaced 793 nodes with 2938 nodes, unitigified 4867 nodes to 1978 nodes
removed 9 nodes and 0 edges
removed 7 tips and 6 edges
try resolve k=1062, replaced 759 nodes with 2753 nodes, unitigified 4564 nodes to 1844 nodes
removed 12 nodes and 0 edges
removed 4 tips and 3 edges
try resolve k=1063, replaced 772 nodes with 2886 nodes, unitigified 4877 nodes to 1985 nodes
removed 11 nodes and 0 edges
removed 6 tips and 5 edges
try resolve k=1064, replaced 703 nodes with 2650 nodes, unitigified 4422 nodes to 1819 nodes
removed 10 nodes and 0 edges
removed 4 tips and 1 edges
try resolve k=1065, replaced 735 nodes with 2727 nodes, unitigified 4599 nodes to 1839 nodes
removed 7 nodes and 0 edges
removed 5 tips and 1 edges
try resolve k=1066, replaced 759 nodes with 2885 nodes, unitigified 4824 nodes to 1958 nodes
removed 9 nodes and 0 edges
removed 8 tips and 1 edges
try resolve k=1067, replaced 769 nodes with 2891 nodes, unitigified 4822 nodes to 2006 nodes
removed 13 nodes and 0 edges
removed 3 tips and 4 edges
try resolve k=1068, replaced 707 nodes with 2734 nodes, unitigified 4580 nodes to 1915 nodes
removed 10 nodes and 0 edges
removed 6 tips and 2 edges
try resolve k=1069, replaced 727 nodes with 2764 nodes, unitigified 4550 nodes to 1934 nodes
removed 2 nodes and 0 edges
removed 4 tips and 2 edges
try resolve k=1070, replaced 743 nodes with 2603 nodes, unitigified 4300 nodes to 1716 nodes
removed 7 nodes and 0 edges
removed 5 tips and 4 edges
try resolve k=1071, replaced 706 nodes with 2618 nodes, unitigified 4368 nodes to 1804 nodes
removed 8 nodes and 0 edges
removed 2 tips and 0 edges
try resolve k=1072, replaced 726 nodes with 2633 nodes, unitigified 4384 nodes to 1778 nodes
removed 13 nodes and 0 edges
removed 6 tips and 1 edges
try resolve k=1073, replaced 713 nodes with 2562 nodes, unitigified 4243 nodes to 1712 nodes
removed 9 nodes and 0 edges
removed 3 tips and 2 edges
try resolve k=1074, replaced 690 nodes with 2598 nodes, unitigified 4347 nodes to 1798 nodes
removed 15 nodes and 0 edges
removed 2 tips and 2 edges
try resolve k=1075, replaced 738 nodes with 2637 nodes, unitigified 4334 nodes to 1757 nodes
removed 8 nodes and 0 edges
removed 1 tips and 3 edges
try resolve k=1076, replaced 725 nodes with 2605 nodes, unitigified 4248 nodes to 1763 nodes
removed 12 nodes and 0 edges
removed 3 tips and 0 edges
try resolve k=1077, replaced 744 nodes with 2864 nodes, unitigified 4802 nodes to 1978 nodes
removed 10 nodes and 0 edges
removed 10 tips and 2 edges
try resolve k=1078, replaced 689 nodes with 2543 nodes, unitigified 4162 nodes to 1760 nodes
removed 7 nodes and 0 edges
removed 6 tips and 2 edges
try resolve k=1079, replaced 721 nodes with 2761 nodes, unitigified 4558 nodes to 1893 nodes
removed 11 nodes and 0 edges
removed 5 tips and 2 edges
try resolve k=1080, replaced 724 nodes with 2790 nodes, unitigified 4633 nodes to 1938 nodes
removed 8 nodes and 0 edges
removed 2 tips and 1 edges
try resolve k=1081, replaced 700 nodes with 2502 nodes, unitigified 4082 nodes to 1683 nodes
removed 6 nodes and 2 edges
removed 0 tips and 1 edges
try resolve k=1082, replaced 668 nodes with 2445 nodes, unitigified 4007 nodes to 1666 nodes
removed 5 nodes and 0 edges
removed 6 tips and 1 edges
try resolve k=1083, replaced 678 nodes with 2568 nodes, unitigified 4231 nodes to 1768 nodes
removed 11 nodes and 0 edges
removed 5 tips and 2 edges
try resolve k=1084, replaced 700 nodes with 2547 nodes, unitigified 4197 nodes to 1735 nodes
removed 5 nodes and 0 edges
removed 2 tips and 3 edges
try resolve k=1085, replaced 690 nodes with 2555 nodes, unitigified 4200 nodes to 1743 nodes
removed 16 nodes and 2 edges
removed 1 tips and 1 edges
try resolve k=1086, replaced 684 nodes with 2483 nodes, unitigified 4079 nodes to 1709 nodes
removed 11 nodes and 0 edges
removed 4 tips and 1 edges
try resolve k=1087, replaced 666 nodes with 2470 nodes, unitigified 4095 nodes to 1668 nodes
removed 12 nodes and 2 edges
removed 4 tips and 1 edges
try resolve k=1088, replaced 685 nodes with 2577 nodes, unitigified 4306 nodes to 1762 nodes
removed 10 nodes and 0 edges
removed 5 tips and 1 edges
try resolve k=1089, replaced 696 nodes with 2553 nodes, unitigified 4217 nodes to 1749 nodes
removed 12 nodes and 0 edges
removed 5 tips and 2 edges
try resolve k=1090, replaced 723 nodes with 2651 nodes, unitigified 4315 nodes to 1798 nodes
removed 6 nodes and 0 edges
removed 5 tips and 2 edges
try resolve k=1091, replaced 711 nodes with 2450 nodes, unitigified 3971 nodes to 1635 nodes
removed 9 nodes and 0 edges
removed 5 tips and 0 edges
try resolve k=1092, replaced 638 nodes with 2358 nodes, unitigified 3861 nodes to 1612 nodes
removed 6 nodes and 0 edges
removed 3 tips and 1 edges
try resolve k=1093, replaced 659 nodes with 2447 nodes, unitigified 4021 nodes to 1668 nodes
removed 10 nodes and 0 edges
removed 4 tips and 0 edges
try resolve k=1094, replaced 651 nodes with 2335 nodes, unitigified 3744 nodes to 1577 nodes
removed 6 nodes and 0 edges
removed 4 tips and 0 edges
try resolve k=1095, replaced 676 nodes with 2631 nodes, unitigified 4417 nodes to 1851 nodes
removed 12 nodes and 0 edges
removed 5 tips and 2 edges
try resolve k=1096, replaced 668 nodes with 2408 nodes, unitigified 3967 nodes to 1650 nodes
removed 7 nodes and 0 edges
removed 2 tips and 2 edges
try resolve k=1097, replaced 705 nodes with 2667 nodes, unitigified 4365 nodes to 1837 nodes
removed 4 nodes and 0 edges
removed 8 tips and 4 edges
try resolve k=1098, replaced 696 nodes with 2562 nodes, unitigified 4206 nodes to 1756 nodes
removed 8 nodes and 0 edges
removed 5 tips and 0 edges
try resolve k=1099, replaced 4095 nodes with 12879 nodes, unitigified 17830 nodes to 8534 nodes
removed 2 nodes and 0 edges
removed 12 tips and 0 edges
try resolve k=1100, replaced 7841 nodes with 28849 nodes, unitigified 46746 nodes to 19946 nodes
removed 84 nodes and 4 edges
removed 52 tips and 8 edges
try resolve k=1101, replaced 954 nodes with 3417 nodes, unitigified 5607 nodes to 2310 nodes
removed 11 nodes and 0 edges
removed 4 tips and 2 edges
try resolve k=1102, replaced 994 nodes with 3684 nodes, unitigified 6006 nodes to 2532 nodes
removed 17 nodes and 0 edges
removed 6 tips and 0 edges
try resolve k=1103, replaced 1013 nodes with 3854 nodes, unitigified 6295 nodes to 2690 nodes
removed 12 nodes and 0 edges
removed 4 tips and 4 edges
try resolve k=1104, replaced 998 nodes with 3556 nodes, unitigified 5763 nodes to 2374 nodes
removed 13 nodes and 0 edges
removed 10 tips and 0 edges
try resolve k=1105, replaced 972 nodes with 3553 nodes, unitigified 5767 nodes to 2415 nodes
removed 8 nodes and 0 edges
removed 4 tips and 1 edges
try resolve k=1106, replaced 962 nodes with 3506 nodes, unitigified 5682 nodes to 2394 nodes
removed 15 nodes and 0 edges
removed 4 tips and 2 edges
try resolve k=1107, replaced 950 nodes with 3630 nodes, unitigified 5990 nodes to 2501 nodes
removed 16 nodes and 2 edges
removed 7 tips and 1 edges
try resolve k=1108, replaced 915 nodes with 3444 nodes, unitigified 5674 nodes to 2378 nodes
removed 11 nodes and 2 edges
removed 3 tips and 2 edges
try resolve k=1109, replaced 918 nodes with 3528 nodes, unitigified 5818 nodes to 2453 nodes
removed 16 nodes and 2 edges
removed 5 tips and 1 edges
try resolve k=1110, replaced 940 nodes with 3539 nodes, unitigified 5839 nodes to 2418 nodes
removed 10 nodes and 0 edges
removed 9 tips and 1 edges
try resolve k=1111, replaced 988 nodes with 3633 nodes, unitigified 5928 nodes to 2503 nodes
removed 13 nodes and 0 edges
removed 5 tips and 0 edges
try resolve k=1112, replaced 959 nodes with 3530 nodes, unitigified 5814 nodes to 2400 nodes
removed 11 nodes and 0 edges
removed 4 tips and 1 edges
try resolve k=1113, replaced 934 nodes with 3390 nodes, unitigified 5531 nodes to 2333 nodes
removed 13 nodes and 0 edges
removed 13 tips and 2 edges
try resolve k=1114, replaced 943 nodes with 3357 nodes, unitigified 5460 nodes to 2280 nodes
removed 11 nodes and 0 edges
removed 6 tips and 4 edges
try resolve k=1115, replaced 1015 nodes with 3778 nodes, unitigified 6183 nodes to 2587 nodes
removed 20 nodes and 2 edges
removed 10 tips and 3 edges
try resolve k=1116, replaced 973 nodes with 3564 nodes, unitigified 5856 nodes to 2395 nodes
removed 8 nodes and 0 edges
removed 11 tips and 3 edges
try resolve k=1117, replaced 1013 nodes with 3878 nodes, unitigified 6389 nodes to 2684 nodes
removed 15 nodes and 2 edges
removed 10 tips and 2 edges
try resolve k=1118, replaced 1006 nodes with 3728 nodes, unitigified 6207 nodes to 2582 nodes
removed 21 nodes and 2 edges
removed 4 tips and 5 edges
try resolve k=1119./buildGraph.sh: line 41: 3666668 Killed                  /home/nicholas.gladman/software/.conda/envs/verkko_2.1/lib/verkko/bin/MBG $iopt -t 4 -k 1001 -r 15000 -R 4000 -w 100 --kmer-abundance 1 --unitig-abundance 2 --error-masking=collapse-msat --output-sequence-paths ../1-buildGraph/paths.gaf --hpc-variant-onecopy-coverage 25 --do-unsafe-guesswork-resolutions --out ../1-buildGraph/hifi-resolved.gfa
skoren commented 1 month ago

It looks like the job is getting killed by your cluster, probably for memory. It looks like it tried 50gb last time it failed. Try manually increasing the time in your verrko by adding --mbg-run 4 100 120 and see if it succeeds. This will request 4 cores (the default), 100gb or ram and 5 days. The full list of these step-specific options is on the README but usually you don't need to modify them.

nickgladman commented 1 month ago

That fixed it! I am just having issues with assembly size not matching expected genome size (~1/2 the size). I will play around the parameters. Thanks!

skoren commented 1 month ago

What's your input data + coverages? I also noticed you have adjusted the seed-mem-count from the default of 10k, have you already tried running w/default settings?

nickgladman commented 1 month ago

I am using HiFi and ONT reads (~20x and 89x coverage respectively). The --seed-max-length setting came from a prior assembly we did in another plant system. It was based on #102. That parameter was left in from when were were working with earlier versions and never removed it. I'll try running it with the defaults on the most recent version.

skoren commented 1 week ago

I'm going to close this since the original SLURM issue is addressed. If you want to open a new issue related to the assembly size vs genome size after the re-run with defaults, please do.