search

mikolmogorov / Flye

De novo assembler for single molecule sequencing reads using repeat graphs
Other
763 stars 165 forks source link

minimap2 stage crashes #501

Closed twolfe closed 1 year ago

twolfe commented 2 years ago

Hello, I am having problems with the consensus step of Flye 2.9-b1774 installed in anaconda3, where the pipeline aborts with:

[2022-04-29 09:50:49] INFO: Starting Flye 2.9-b1774 [2022-04-29 09:50:50] INFO: >>>STAGE: configure [2022-04-29 09:50:50] INFO: Configuring run [2022-04-29 09:51:14] INFO: Total read length: 5487080319 [2022-04-29 09:51:14] INFO: Reads N50/N90: 7485 / 1239 [2022-04-29 09:51:14] INFO: Minimum overlap set to 1000 [2022-04-29 09:51:14] INFO: >>>STAGE: assembly [2022-04-29 09:51:14] INFO: Assembling disjointigs [2022-04-29 09:51:14] INFO: Reading sequences [2022-04-29 09:51:45] INFO: Building minimizer index [2022-04-29 09:51:45] INFO: Pre-calculating index storage 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [2022-04-29 09:53:30] INFO: Filling index 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [2022-04-29 09:55:20] INFO: Extending reads [2022-04-29 09:55:36] INFO: Overlap-based coverage: 3 [2022-04-29 09:55:36] INFO: Median overlap divergence: 0.086225 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [2022-04-29 10:13:02] INFO: Assembled 56488 disjointigs [2022-04-29 10:13:10] INFO: Generating sequence 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [2022-04-29 10:14:40] INFO: Filtering contained disjointigs 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [2022-04-29 10:17:20] INFO: Contained seqs: 2765 [2022-04-29 10:17:45] INFO: >>>STAGE: consensus [2022-04-29 10:17:45] INFO: Running Minimap2 [2022-04-29 10:30:59] ERROR: Error running minimap2, terminating. See the alignment error log for details: /mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_nano/10-consensus/minimap.stderr [2022-04-29 10:30:59] ERROR: Command '['/bin/bash', '-c', "set -eo pipefail; flye-minimap2 '/mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_nano/00-assembly/draft_assembly.fasta' '/mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_raw_bonito.fa' -x map-ont -t 20 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G | flye-samtools view -T '/mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_nano/00-assembly/draft_assembly.fasta' -u - | flye-samtools sort -T '/mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_nano/10-consensus/sort_220429_101745' -O bam -@ 4 -l 1 -m 4G -o '/mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_nano/10-consensus/minimap.bam'"]' returned non-zero exit status 137. [2022-04-29 10:30:59] ERROR: Pipeline aborted

and the error log minimap_stderr.log

When I run the command alone, without ' ' s, it completed, but then I was not able to resume the run using --resume-from consensus. Thank you for the help.

mikolmogorov commented 2 years ago

Hi,

Looks like minimap_stderr log is from a different run (different file names). Could you submit the matching one?

twolfe commented 2 years ago

Sorry about that. I rerun flye after deleting the output folder and files. I renamed the .stderr file to be able to upload it to the thread. Here is the output from the consensus stage.

[2022-05-03 09:15:48] INFO: >>>STAGE: consensus [2022-05-03 09:15:48] INFO: Running Minimap2 [2022-05-03 09:36:37] ERROR: Error running minimap2, terminating. See the alignment error log for details: /mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_nano/10-consensus/minimap.stderr [2022-05-03 09:36:37] ERROR: Command '['/bin/bash', '-c', "set -eo pipefail; flye-minimap2 '/mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_nano/00-assembly/draft_assembly.fasta' '/mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_raw_bonito.fa' -x map-ont -t 20 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G | flye-samtools view -T '/mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_nano/00-assembly/draft_assembly.fasta' -u - | flye-samtools sort -T '/mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_nano/10-consensus/sort_220503_091548' -O bam -@ 4 -l 1 -m 4G -o '/mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_nano/10-consensus/minimap.bam'"]' returned non-zero exit status 137.

minimap.stderr.log

Cheers

JamesHowie14 commented 2 years ago

Hi fenderglass, in case it helps, I also ran an assembly on the same system as twolfe, but from a different user, and with different data, but generating the same error:

The versions: Flye = 2.9-b1768 Minimap2 = 2.24-r1122, both from conda.

The command:

flye --nano-hq 1_Cmel4_1_2_PCR1-2-3_SUP.fastq \
  --out-dir /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial3/ \
  --genome-size 438m \
  --asm-coverage 34 \
  --threads 24 \
  --iterations 1

The error message:

[2022-05-03 10:51:02] ERROR: Command '['/bin/bash', '-c', "set -eo pipefail; flye-minimap2 '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial3/00-assembly/draft_assembly.fasta' '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq' -x map-ont -t 24 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G | flye-samtools view -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial3/00-assembly/draft_assembly.fasta' -u - | flye-samtools sort -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial3/10-consensus/sort_220503_103626' -O bam -@ 4 -l 1 -m 4G -o '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial3/10-consensus/minimap.bam'"]' returned non-zero exit status 137.
[2022-05-03 10:51:02] ERROR: Pipeline aborted

The minimap_stderror.log:

[samfaipath] build FASTA index...
[M::mm_idx_gen::21.332*1.48] collected minimizers
[M::mm_idx_gen::22.612*2.35] sorted minimizers
[M::main::22.622*2.34] loaded/built the index for 129893 target sequence(s)
[M::mm_mapopt_update::23.693*2.28] mid_occ = 1171
[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 129893
[M::mm_idx_stat::24.373*2.25] distinct minimizers: 62359831 (42.64% are singletons); average occurrences: 5.557; average spacing: 5.241; total length: 1816104594
/bin/bash: line 1: 3473754 Killed                  flye-minimap2 '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial3/00-assembly/draft_assembly.fasta' '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq' -x map-ont -t 24 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G
     3473755 Done                    | flye-samtools view -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial3/00-assembly/draft_assembly.fasta' -u -
     3473756 Done                    | flye-samtools sort -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial3/10-consensus/sort_220503_103626' -O bam -@ 4 -l 1 -m 4G -o '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial3/10-consensus/minimap.bam'

Prior to the error message the readout is basically the same as in that of twolfe, and I also note that I can run flye with no problems to assemble smaller genomes in meta mode. I just add this in case it helps rule out possible options / to help debug ..

mikolmogorov commented 2 years ago

Very likely you are running out of memory. Can you share more details about the system and send flye.log? If it is a memory error, my only suggestion would be to try running with a reduced number of threads.

twolfe commented 2 years ago

Here is the fly.log. I cut out the middle part and keep the first and last parts. flye.log Thanks for having a look.

JamesHowie14 commented 2 years ago

Hi, yes, I also wondered about memory, have been working my down to fewer threads and reads, maybe still being to greedy, will try with fewer. As twolfe send a flye.log I won't also send one, but system has 125 Gb RAM, 48 cores, and a lot of hard disk space arranged in a RAID 10.

mikolmogorov commented 2 years ago

I see, I think it's a memory issue. The log says there are 128Gb in total, but only 56Gb was available at the time of run.

I'd try to run with more available RAM, and at the same time reduce the number of threads. Unfortunately I don't have much control over how much RAM minimap2 consumes..

JamesHowie14 commented 2 years ago

Ah, interesting, yes, we will report back after trying with more RAM and fewer threads etc ;)

JamesHowie14 commented 2 years ago

Alright, I've started a run in 12 threads, with 123 RAM available and using only 24x longest reads, will report on how it or even fewer core runs go ;)

so far:

nome-size 438m --asm-coverage 24 --threads 12 --iterations 1
[2022-05-03 16:53:36] root: DEBUG: Python version: 3.8.5 (default, Sep  4 2020, 
07:30:14) 
[GCC 7.3.0]
[2022-05-03 16:53:36] root: INFO: >>>STAGE: configure
[2022-05-03 16:53:36] root: INFO: Configuring run
[2022-05-03 16:55:46] root: INFO: Total read length: 23150251229
[2022-05-03 16:55:46] root: INFO: Input genome size: 438000000
[2022-05-03 16:55:46] root: INFO: Estimated coverage: 52
[2022-05-03 16:55:46] root: INFO: Reads N50/N90: 1746 / 566
[2022-05-03 16:55:46] root: INFO: Minimum overlap set to 1000
[2022-05-03 16:55:46] root: INFO: Using longest 24x reads for contig assembly
[2022-05-03 16:55:49] root: DEBUG: Min read length cutoff: 1962
[2022-05-03 16:55:49] root: INFO: >>>STAGE: assembly
[2022-05-03 16:55:49] root: INFO: Assembling disjointigs
[2022-05-03 16:55:49] root: DEBUG: -----Begin assembly log------
[2022-05-03 16:55:49] root: DEBUG: Running: flye-modules assemble --reads /mnt/r
aid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq --out-asm /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial4/00-assembly/draft_assembly.fasta --config /home/jhowie/miniconda3/lib/python3.8/site-packages/flye/config/bin_cfg/asm_nano_hq.cfg --log /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial4/flye.log --threads 12 --genome-size 438000000 --min-ovlp 1000 --min-read 1962
[2022-05-03 16:55:49] DEBUG: Build date: Feb 22 2022 03:49:40
[2022-05-03 16:55:49] DEBUG: Total RAM: 125 Gb
[2022-05-03 16:55:49] DEBUG: Available RAM: 123 Gb
[2022-05-03 16:55:49] DEBUG: Total CPUs: 48
twolfe commented 2 years ago

Hi again, So with more RAM (see flye.log), the consensus step moves a little further but still aborts (see minimap.stderr.log) flye.log minimap.stderr.log Notice I have a low coverage of around 3X for this particular R10 flow cell sequenced sample (--nano-hq parameter). Could this be the root of the problem? I have other samples with slightly higher coverage that I could combine with this sample, but they were sequenced on R9 flow cells. Cheers.

JamesHowie14 commented 2 years ago

Hi fenderglass, another bit of extra information, in case of use. I've tried a couple more times, the most recent I set the command as before, but with 4 threads. Interestingly the log shows 123 GB available RAM, and notes that peak RAM usage was only 67 GB. Nonetheless, it run died with the general error as before, although getting a bit further in minimal:

log:

[2022-05-04 01:28:20] root: INFO: >>>STAGE: configure
[2022-05-04 01:28:20] root: INFO: Configuring run
[2022-05-04 01:30:28] root: INFO: Total read length: 23150251229
[2022-05-04 01:30:28] root: INFO: Input genome size: 438000000
[2022-05-04 01:30:28] root: INFO: Estimated coverage: 52
[2022-05-04 01:30:28] root: INFO: Reads N50/N90: 1746 / 566
[2022-05-04 01:30:28] root: INFO: Minimum overlap set to 1000
[2022-05-04 01:30:28] root: INFO: Using longest 40x reads for contig assembly
[2022-05-04 01:30:32] root: DEBUG: Min read length cutoff: 915
[2022-05-04 01:30:32] root: INFO: >>>STAGE: assembly
[2022-05-04 01:30:32] root: INFO: Assembling disjointigs
[2022-05-04 01:30:32] root: DEBUG: -----Begin assembly log------
[2022-05-04 01:30:32] root: DEBUG: Running: flye-modules assemble --reads /mnt/raid10/c
acopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel
4_1_2_PCR1-2-3_SUP.fastq --out-asm /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenome
s/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/00-assembly/draft_assembly.f
asta --config /home/jhowie/miniconda3/lib/python3.8/site-packages/flye/config/bin_cfg/a
sm_nano_hq.cfg --log /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_
PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/flye.log --threads 4 --genome-size 4380000
00 --min-ovlp 1000 --min-read 915
[2022-05-04 01:30:32] DEBUG: Build date: Feb 22 2022 03:49:40
[2022-05-04 01:30:32] DEBUG: Total RAM: 125 Gb
[2022-05-04 01:30:32] DEBUG: Available RAM: 123 Gb
[2022-05-04 01:30:32] DEBUG: Total CPUs: 48

...

[2022-05-04 06:00:20] INFO: Contained seqs: 10663
[2022-05-04 06:00:22] DEBUG: Writing FASTA
[2022-05-04 06:00:32] DEBUG: Peak RAM usage: 67 Gb
-----------End assembly log------------
[2022-05-04 06:00:50] root: DEBUG: Disjointigs length: 1832130745, N50: 22326
[2022-05-04 06:00:51] root: INFO: >>>STAGE: consensus
[2022-05-04 06:00:51] root: INFO: Running Minimap2
[2022-05-04 07:22:59] root: ERROR: Error running minimap2, terminating. See the alignment error log  for details: /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/10-consensus/minimap.stderr
[2022-05-04 07:23:00] root: ERROR: Command '['/bin/bash', '-c', "set -eo pipefail; flye-minimap2 '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/00-assembly/draft_assembly.fasta' '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq' -x map-ont -t 4 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G | flye-samtools view -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/00-assembly/draft_assembly.fasta' -u - | flye-samtools sort -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/10-consensus/sort_220504_060051' -O bam -@ 4 -l 1 -m 4G -o '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/10-consensus/minimap.bam'"]' returned non-zero exit status 137.
[2022-05-04 07:23:00] root: ERROR: Pipeline aborted

minimap:

[samfaipath] build FASTA index...
[M::mm_idx_gen::21.216*1.47] collected minimizers
[M::mm_idx_gen::25.825*1.90] sorted minimizers
[M::main::25.843*1.90] loaded/built the index for 130701 target sequence(s)
[M::mm_mapopt_update::26.860*1.86] mid_occ = 1180
[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 130701
[M::mm_idx_stat::27.512*1.84] distinct minimizers: 62534741 (42.47% are singletons); average occurrences: 5.590; average spacing: 5.241; total length: 1832130745
/bin/bash: line 1: 3500177 Killed                  flye-minimap2 '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/00-assembly/draft_assembly.fasta' '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq' -x map-ont -t 4 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G
     3500178 Done                    | flye-samtools view -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/00-assembly/draft_assembly.fasta' -u -
     3500179 Done                    | flye-samtools sort -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/10-consensus/sort_220504_060051' -O bam -@ 4 -l 1 -m 4G -o '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/10-consensus/minimap.bam'

That made me think this may not be a memory issue, I wondered if you would agree, or am I missing something there? Do you perhaps have any other ideas what could be tried?

Thanks for any suggestions :)

mikolmogorov commented 2 years ago

@JamesHowie14 the minimap2 error long say that the process was killed by the system. This is typically happens when you are running out of resources for a process. This is why I suspecting memory issues. But it could be also some limitation of the system itself (e.g. memory allocation per user etc).

  1. Could you give more information how about the system and how did you install Flye (e.g. bioconda package?)
  2. On linux you can check the system log (dmesg) for out-of-memory records after the crash
  3. I see that exit code 137 is commonly associated with Docker environments - are you using Docker? It can have a substantial memory overhead.
  4. Try running minimap2 command "manually" and watch the memory usage. This might also give you a more clear error message: flye-minimap2 'path_to_draft_assembly.fasta' 'path_to_reads.fastq' -x map-ont -t 4 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G | flye-samtools view -T 'path_to_draft_assembly.fasta' -u - | flye-samtools sort -O bam -@ 4 -l 1 -m 4G -o 'minimap.bam
JamesHowie14 commented 2 years ago

Dear Fenderglass,

Thanks for your response, OK so:

1 - the system is Linux x86_64, Ubuntu 20.04.1 LTS, with two sockets of 24 cpu, for a total 48 cpu, with 2 threads per core; the cpu max-min MHz are 4568-2200. In terms of RAM we have 126 GB. For hard disk storage, we have a 2TB solid state drive, and a RAID10 array of hard disks which gives 38-40 TB space. I have run all commands reading and writing directly to the RAID10, as for some assemblers hard drive space was an issue.

Flye version = 2.9-b1768, installed via conda

2 - to make sure my response is current I re-ran the command:

flye --nano-hq 1_Cmel4_1_2_PCR1-2-3_SUP.fastq \
  --out-dir /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial8/ \
  --genome-size 438m \
  --asm-coverage 10 \
  --threads 12 \
  --iterations 1

, as expected, this recovered the error:

[2022-05-16 14:16:12] INFO: Filtering contained disjointigs
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 
[2022-05-16 14:22:56] INFO: Contained seqs: 1699
[2022-05-16 14:23:22] INFO: >>>STAGE: consensus
[2022-05-16 14:23:22] INFO: Running Minimap2
[2022-05-16 14:36:38] ERROR: Error running minimap2, terminating. See the alignment error log  for details: /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial8/10-consensus/minimap.stderr
[2022-05-16 14:36:39] ERROR: Command '['/bin/bash', '-c', "set -eo pipefail; flye-minimap2 '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial8/00-assembly/draft_assembly.fasta' '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq' -x map-ont -t 12 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G | flye-samtools view -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial8/00-assembly/draft_assembly.fasta' -u - | flye-samtools sort -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial8/10-consensus/sort_220516_142322' -O bam -@ 4 -l 1 -m 4G -o '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial8/10-consensus/minimap.bam'"]' returned non-zero exit status 137.
[2022-05-16 14:36:39] ERROR: Pipeline aborted

Given that, I checked "dmesg", which shows:

[9935276.338517] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1007.slice/session-21465.scope,task=flye-minimap2,pid=618696,uid=1007
[9935276.338536] Out of memory: Killed process 618696 (flye-minimap2) total-vm:46467540kB, anon-rss:44820240kB, file-rss:0kB, shmem-rss:0kB, UID:1007 pgtables:89228kB oom_score_adj:0
[9935278.690076] oom_reaper: reaped process 618696 (flye-minimap2), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

The full message was:

[9935276.337889] systemd invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[9935276.337897] CPU: 30 PID: 1 Comm: systemd Tainted: P           OE     5.13.0-27-generic #29~20.04.1-Ubuntu
[9935276.337900] Hardware name: Micro-Star International Co., Ltd. MS-7C60/TRX40 PRO 10G (MS-7C60), BIOS 1.60 05/13/2020
[9935276.337901] Call Trace:
[9935276.337906]  dump_stack+0x7d/0x9c
[9935276.337911]  dump_header+0x4f/0x1f6
[9935276.337915]  oom_kill_process.cold+0xb/0x10
[9935276.337918]  out_of_memory.part.0+0x1fa/0x480
[9935276.337922]  out_of_memory+0x6d/0xd0
[9935276.337924]  __alloc_pages_slowpath.constprop.0+0xc86/0xd50
[9935276.337928]  __alloc_pages+0x2ce/0x320
[9935276.337931]  alloc_pages+0x87/0xe0
[9935276.337933]  __page_cache_alloc+0x89/0xb0
[9935276.337937]  pagecache_get_page+0x218/0x4d0
[9935276.337940]  filemap_fault+0x5bf/0x9b0
[9935276.337943]  ? do_set_pte+0xc8/0x140
[9935276.337947]  ? next_uptodate_page+0x160/0x270
[9935276.337949]  ext4_filemap_fault+0x32/0x50
[9935276.337953]  __do_fault+0x3e/0xc0
[9935276.337956]  do_fault+0x1ea/0x440
[9935276.337958]  __handle_mm_fault+0x619/0x8e0
[9935276.337961]  handle_mm_fault+0xda/0x2b0
[9935276.337963]  do_user_addr_fault+0x1bb/0x650
[9935276.337966]  exc_page_fault+0x7d/0x170
[9935276.337969]  ? asm_exc_page_fault+0x8/0x30
[9935276.337972]  asm_exc_page_fault+0x1e/0x30
[9935276.337974] RIP: 0033:0x7f6a5de0a4f8
[9935276.337980] Code: Unable to access opcode bytes at RIP 0x7f6a5de0a4ce.
[9935276.337981] RSP: 002b:00007fff5a127dc0 EFLAGS: 00010207
[9935276.337983] RAX: 00007f6a5d1d6900 RBX: 00007fff5a127e07 RCX: 0000000000000001
[9935276.337985] RDX: 0000000000000065 RSI: 00007fff5a127e07 RDI: 000055734b93ee30
[9935276.337986] RBP: 000055734b93ee30 R08: 0000000000000001 R09: 0000000000000001
[9935276.337987] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[9935276.337988] R13: 0000000000100000 R14: 0000000000000006 R15: 00007fff5a127e07
[9935276.337990] Mem-Info:
[9935276.337994] active_anon:2557632 inactive_anon:29780650 isolated_anon:0
                  active_file:2292 inactive_file:2347 isolated_file:158
                  unevictable:0 dirty:1 writeback:0
                  slab_reclaimable:125241 slab_unreclaimable:146230
                  mapped:656 shmem:1064 pagetables:70424 bounce:0
                  free:145429 free_pcp:63 free_cma:0
[9935276.337998] Node 0 active_anon:10230528kB inactive_anon:119122600kB active_file:9168kB inactive_file:9388kB unevictable:0kB isolated(anon):0kB isolated(file):632kB mapped:2624kB dirty:4kB writeback:0kB shmem:4256kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:28304kB pagetables:281696kB all_unreclaimable? no
[9935276.338003] Node 0 DMA free:11280kB min:4kB low:16kB high:28kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15376kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[9935276.338008] lowmem_reserve[]: 0 2842 128608 128608 128608
[9935276.338012] Node 0 DMA32 free:504404kB min:1492kB low:4400kB high:7308kB reserved_highatomic:0KB active_anon:313668kB inactive_anon:2111728kB active_file:0kB inactive_file:28kB unevictable:0kB writepending:0kB present:3035588kB managed:2969524kB mlocked:0kB bounce:0kB free_pcp:252kB local_pcp:0kB free_cma:0kB
[9935276.338016] lowmem_reserve[]: 0 0 125765 125765 125765
[9935276.338019] Node 0 Normal free:66032kB min:66080kB low:194864kB high:323648kB reserved_highatomic:0KB active_anon:9916860kB inactive_anon:117010872kB active_file:9172kB inactive_file:8944kB unevictable:0kB writepending:4kB present:131057664kB managed:128792996kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[9935276.338024] lowmem_reserve[]: 0 0 0 0 0
[9935276.338026] Node 0 DMA: 2*4kB (U) 1*8kB (U) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11280kB
[9935276.338036] Node 0 DMA32: 850*4kB (UE) 716*8kB (UME) 1476*16kB (UME) 1365*32kB (UE) 1079*64kB (UME) 704*128kB (UME) 341*256kB (UME) 145*512kB (UME) 51*1024kB (UE) 9*2048kB (UME) 9*4096kB (UME) = 504648kB
[9935276.338050] Node 0 Normal: 3603*4kB (UME) 1701*8kB (UME) 497*16kB (UME) 575*32kB (UME) 92*64kB (UME) 9*128kB (UM) 0*256kB 3*512kB (M) 0*1024kB 2*2048kB (ME) 0*4096kB = 67044kB
[9935276.338064] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[9935276.338066] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[9935276.338068] 21108 total pagecache pages
[9935276.338068] 14730 pages in swap cache
[9935276.338069] Swap cache stats: add 126197531, delete 126187441, find 33131458/43601237
[9935276.338071] Free swap  = 0kB
[9935276.338071] Total swap = 2097148kB
[9935276.338072] 33527312 pages RAM
[9935276.338073] 0 pages HighMem/MovableOnly
[9935276.338074] 582838 pages reserved
[9935276.338074] 0 pages hwpoisoned
[9935276.338075] Tasks state (memory values in pages):
[9935276.338076] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[9935276.338089] [    785]     0   785    25415      254   180224      461          -250 systemd-journal
[9935276.338093] [    811]     0   811     6173      658    73728      367         -1000 systemd-udevd
[9935276.338097] [   1411]   102  1411    22564       27    86016      184             0 systemd-timesyn
[9935276.338100] [   1445]     0  1445    62331      804   102400       87             0 accounts-daemon
[9935276.338102] [   1446]     0  1446      637        0    45056       47             0 acpid
[9935276.338105] [   1449]   115  1449     2170      112    53248       67             0 avahi-daemon
[9935276.338107] [   1453]     0  1453     4513       26    61440       40             0 cron
[9935276.338109] [   1456]   103  1456     2562      617    61440      110          -900 dbus-daemon
[9935276.338112] [   1457]     0  1457   155830      529   184320      382             0 NetworkManager
[9935276.338114] [   1464]     0  1464    20533      109    65536       51             0 irqbalance
[9935276.338116] [   1481]     0  1481    61060        3    94208      159             0 switcheroo-cont
[9935276.338119] [   1484]     0  1484     4425      329    77824      173             0 systemd-logind
[9935276.338121] [   1490]     0  1490    98519      907   135168      223             0 udisksd
[9935276.338123] [   1495]     0  1495     3420       24    65536      135             0 wpa_supplicant
[9935276.338126] [   1513]   115  1513     2086        1    53248       81             0 avahi-daemon
[9935276.338128] [   1550]   121  1550    82281      364   122880     1078             0 colord
[9935276.338130] [   1597]     0  1597    62156        0   110592      538             0 gdm3
[9935276.338132] [   1783]     0  1783    31690        0   143360     2070             0 unattended-upgr
[9935276.338135] [   1784]     0  1784    43827      121   110592      203             0 gdm-session-wor
[9935276.338137] [   1828]   125  1828     4749      272    69632      255             0 systemd
[9935276.338139] [   1830]   125  1830    42317        1    94208      908             0 (sd-pam)
[9935276.338141] [   1887]   125  1887   241141       46   196608     1061             0 pulseaudio
[9935276.338144] [   1889]   125  1889   183309      244   221184     1962             0 tracker-miner-f
[9935276.338146] [   1894]   125  1894    43161        1    86016      172             0 gdm-x-session
[9935276.338148] [   1898]   125  1898   652280     1065   643072     7983             0 Xorg
[9935276.338150] [   1906]   125  1906     1866      109    57344       76             0 dbus-daemon
[9935276.338152] [   1909]   111  1909    38233       38    61440       37             0 rtkit-daemon
[9935276.338154] [   1928]   125  1928    62087      109    94208      145             0 gvfsd
[9935276.338156] [   1949]   125  1949    78202        0    94208      223             0 gvfsd-fuse
[9935276.338158] [   1959]   125  1959    80701      493   106496      158             0 gvfs-udisks2-vo
[9935276.338161] [   2474]   125  2474     1323        0    49152       48             0 dbus-run-sessio
[9935276.338163] [   2482]   125  2482     1891        0    57344      246             0 dbus-daemon
[9935276.338165] [   2486]   125  2486   123117      152   167936      422             0 gnome-session-b
[9935276.338167] [   2689]   125  2689    76378        0    94208      201             0 at-spi-bus-laun
[9935276.338169] [   2699]   125  2699     1810        0    53248      119             0 dbus-daemon
[9935276.338171] [   2877]   125  2877  1571953    10369  1486848    15048             0 gnome-shell
[9935276.338174] [   2946]   125  2946    61127       28    90112      169             0 gvfs-goa-volume
[9935276.338176] [   2950]   125  2950   138950      142   258048     1408             0 goa-daemon
[9935276.338178] [   2969]   125  2969    80847       56   114688      385             0 goa-identity-se
[9935276.338180] [   2974]   125  2974    61719       28    90112      241             0 gvfs-gphoto2-vo
[9935276.338182] [   2979]   125  2979    61116       52   102400      159             0 gvfs-mtp-volume
[9935276.338184] [   2989]   125  2989    81339       74   110592      214             0 gvfs-afc-volume
[9935276.338186] [   2995]     0  2995    81650      196   110592      255             0 upowerd
[9935276.338188] [   3028]   125  3028    79891        0   106496      379             0 ibus-daemon
[9935276.338191] [   3031]   125  3031    42834        0    86016      168             0 ibus-memconf
[9935276.338193] [   3034]   125  3034    49147        0   143360      521             0 ibus-x11
[9935276.338195] [   3036]   125  3036    61284        1    94208      180             0 ibus-portal
[9935276.338197] [   3046]   125  3046    40726        0    81920      230             0 at-spi2-registr
[9935276.338199] [   3050]   125  3050    61057        0    94208      155             0 xdg-permission-
[9935276.338201] [   3063]   125  3063   734706        0   282624     1538             0 gjs
[9935276.338203] [   3083]   125  3083   118318      164   126976      213             0 gsd-sharing
[9935276.338205] [   3085]   125  3085    85920        0   155648      599             0 gsd-wacom
[9935276.338207] [   3086]   125  3086   105170      253   155648      771             0 gsd-color
[9935276.338209] [   3087]   125  3087    86044      130   159744      417             0 gsd-keyboard
[9935276.338211] [   3088]   125  3088    64378      102   131072      447             0 gsd-print-notif
[9935276.338214] [   3090]   125  3090   116433      120   118784      107             0 gsd-rfkill
[9935276.338216] [   3093]   125  3093    80948        0   114688      292             0 gsd-smartcard
[9935276.338218] [   3094]   125  3094    95559        0   172032      554             0 gsd-datetime
[9935276.338220] [   3096]   125  3096   171747      147   188416      566             0 gsd-media-keys
[9935276.338222] [   3101]   125  3101    61035        0    98304      182             0 gsd-screensaver
[9935276.338224] [   3104]   125  3104    81993        0   118784      310             0 gsd-sound
[9935276.338226] [   3105]   125  3105    79572        0   106496      202             0 gsd-a11y-settin
[9935276.338228] [   3109]   125  3109    80087      108   114688      146             0 gsd-housekeepin
[9935276.338230] [   3110]   125  3110    42832        0    77824      197             0 ibus-engine-sim
[9935276.338233] [   3113]   125  3113   104705      159   159744      465             0 gsd-power
[9935276.338235] [   3152]   125  3152    87704        0   159744      544             0 gsd-printer
[9935276.338237] [   3402]   125  3402    39032        0    69632      170             0 dconf-service
[9935276.338239] [   3409]     0  3409  1989907     3474  1241088     8017          -500 dockerd
[9935276.338241] [   3419]   120  3419    81712      247   135168      315             0 whoopsie
[9935276.338243] [   3424]   116  3424     2814       25    65536       94             0 kerneloops
[9935276.338245] [   3426]   116  3426     2814       25    61440       94             0 kerneloops
[9935276.338248] [   3430]     0  3430  1078080     1084   581632      635             0 teamviewerd
[9935276.338250] [   3435]     0  3435    45749      529   135168      567             0 rserver
[9935276.338252] [   3625]     0  3625      624        0    45056       21             0 bpfilter_umh
[9935276.338254] [   3842]  1006  3842     4745      222    81920      305             0 systemd
[9935276.338256] [   3847]  1006  3847    42321        1    94208      930             0 (sd-pam)
[9935276.338258] [   3890]  1006  3890   265693      888   241664     2062             0 tracker-miner-f
[9935276.338260] [   3900]  1006  3900     1896       96    53248      102             0 dbus-daemon
[9935276.338262] [   3929]  1006  3929    62087        0    94208      282             0 gvfsd
[9935276.338264] [   3934]  1006  3934    94586        0    98304      244             0 gvfsd-fuse
[9935276.338266] [   3940]  1006  3940    80701      481   102400      171             0 gvfs-udisks2-vo
[9935276.338268] [   3960]  1006  3960    61127        0    90112      231             0 gvfs-goa-volume
[9935276.338270] [   3967]  1006  3967   138950      142   266240     1406             0 goa-daemon
[9935276.338272] [   3997]  1006  3997    80847       67   114688      374             0 goa-identity-se
[9935276.338274] [   4019]  1006  4019    61719        0    98304      273             0 gvfs-gphoto2-vo
[9935276.338276] [   4024]  1006  4024    61116       80    98304      150             0 gvfs-mtp-volume
[9935276.338278] [   4029]  1006  4029    81339       78   114688      226             0 gvfs-afc-volume
[9935276.338280] [   4156]     0  4156      827       48    45056       25             0 mdadm
[9935276.338282] [   4296]     0  4296   327885       53   413696      655             0 mk_manager_svc
[9935276.338285] [   4300]     0  4300    39006        0    69632       84             0 crashpad_handle
[9935276.338287] [   4320]     0  4320   408872      188   544768     6731             0 guppy_basecall_
[9935276.338289] [   4321]     0  4321   114800        3   143360      530             0 grpcwebproxy
[9935276.338291] [   4322]     0  4322   171301        0   282624      569             0 basecall_manage
[9935276.338293] [   4330]     0  4330    39006        0    77824       74             0 crashpad_handle
[9935276.338295] [   4343]     0  4343   114384        0   135168      519             0 grpcwebproxy
[9935276.338297] [   4421]  1000  4421     4729      276    77824      252             0 systemd
[9935276.338299] [   4422]  1000  4422    42353        1    94208      968             0 (sd-pam)
[9935276.338301] [   4430]  1000  4430   183774      663   217088     2659             0 tracker-miner-f
[9935276.338304] [   4439]  1000  4439     1900       97    53248       86             0 dbus-daemon
[9935276.338306] [   4469]  1000  4469    62086      107    98304      153             0 gvfsd
[9935276.338308] [   4474]  1000  4474    94586        0   102400      227             0 gvfsd-fuse
[9935276.338310] [   4481]  1000  4481    80703      497   118784      165             0 gvfs-udisks2-vo
[9935276.338312] [   4505]  1000  4505    61127       88    90112      105             0 gvfs-goa-volume
[9935276.338314] [   4510]  1000  4510   138952      144   249856     1404             0 goa-daemon
[9935276.338316] [   4533]  1000  4533    80847       61   110592      410             0 goa-identity-se
[9935276.338318] [   4539]  1000  4539    61718       65   106496      201             0 gvfs-gphoto2-vo
[9935276.338320] [   4544]  1000  4544    61116       83    98304      130             0 gvfs-mtp-volume
[9935276.338322] [   4553]  1000  4553    81339       74   118784      213             0 gvfs-afc-volume
[9935276.338324] [   4714]  1000  4714     5158       23    53248      652             0 tmux: server
[9935276.338326] [   4715]  1000  4715     5015        1    69632      546             0 bash
[9935276.338329] [  17574]  1006 17574   117308     3251   352256    17318             0 tracker-store
[9935276.338331] [  17644]  1000 17644    42723       51    86016      129             0 gvfsd-metadata
[9935276.338333] [  17653]   125 17653    42721       79    86016      100             0 gvfsd-metadata
[9935276.338335] [  17915]  1006 17915    42722       79    86016      119             0 gvfsd-metadata
[9935276.338337] [  24034]  1006 24034     7621     2932    81920      204             0 tmux: server
[9935276.338340] [  24035]  1006 24035     5841      202    69632      876             0 bash
[9935276.338342] [  40965]  1006 40965     5296      187    69632      668             0 bash
[9935276.338344] [ 511345]     0 511345     3044       47    61440      186         -1000 sshd
[9935276.338347] [ 699861]  1006 699861     5146      414    69632      263             0 bash
[9935276.338350] [3777985]  1006 3777985     4950      278    61440      225             0 bash
[9935276.338352] [4169811]  1006 4169811     5191        2    65536      743             0 bash
[9935276.338355] [ 424928]   107 424928     4807        0    65536       51             0 uuidd
[9935276.338357] [ 600399]  1006 600399     4999      161    69632      383             0 bash
[9935276.338359] [ 731524]     0 731524    58799      785    90112      184             0 polkitd
[9935276.338362] [ 731528]     0 731528    78465      130   114688      351             0 ModemManager
[9935276.338364] [ 752141]     0 752141  1129976     2354   729088     4073          -999 containerd
[9935276.338366] [1029020]  1006 1029020    96989        0   118784      334             0 gnome-keyring-d
[9935276.338369] [1314442]  1006 1314442     2943      486    61440       73             0 bash
[9935276.338371] [2539261]   101 2539261     6038      146    94208      946             0 systemd-resolve
[9935276.338374] [2708297]     0 2708297     3497        3    69632      370             0 sshd
[9935276.338376] [2708421]  1006 2708421     3842      108    73728      694             0 sshd
[9935276.338378] [2708422]  1006 2708422     2835        2    61440      446             0 bash
[9935276.338380] [2708460]  1006 2708460     2520       12    53248       77             0 tmux: client
[9935276.338382] [3491607]  1007 3491607     4717      266    73728      248             0 systemd
[9935276.338384] [3491608]  1007 3491608    42668       97    98304     1206             0 (sd-pam)
[9935276.338387] [3491617]  1007 3491617   181696      680   212992     1955             0 tracker-miner-f
[9935276.338389] [3491642]  1007 3491642     1844       96    53248       58             0 dbus-daemon
[9935276.338391] [3491663]  1007 3491663    60012      119   102400      130             0 gvfsd
[9935276.338394] [3491674]  1007 3491674    94586        0    90112      194             0 gvfsd-fuse
[9935276.338396] [3491686]  1007 3491686    78629      463   114688      152             0 gvfs-udisks2-vo
[9935276.338398] [3491712]  1007 3491712    59053        0    86016      234             0 gvfs-goa-volume
[9935276.338400] [3491716]  1007 3491716   137415      137   262144     1464             0 goa-daemon
[9935276.338402] [3491725]  1007 3491725    78773       48   114688      275             0 goa-identity-se
[9935276.338405] [3491730]  1007 3491730    59644        0    94208      271             0 gvfs-gphoto2-vo
[9935276.338407] [3491735]  1007 3491735    59041       74    94208      131             0 gvfs-mtp-volume
[9935276.338409] [3491739]  1007 3491739    79265       59   110592      219             0 gvfs-afc-volume
[9935276.338411] [3492310]  1007 3492310    40646       67    86016       96             0 gvfsd-metadata
[9935276.338413] [3507273]  1007 3507273     8968       71   106496     5717             0 perl
[9935276.338415] [3558222]     0 3558222    10058        8   118784     2199             0 networkd-dispat
[9935276.338417] [3582405]  1006 3582405  3732302  2772533 26415104   398436             0 whatshap
[9935276.338420] [3640664]   104 3640664    56088      301    86016        0             0 rsyslogd
[9935276.338423] [ 197286]  1007 197286     2765      323    61440       20             0 tmux: server
[9935276.338425] [ 197287]  1007 197287     2876      293    61440      209             0 bash
[9935276.338427] [ 213280]  1007 213280     2875      199    65536      273             0 bash
[9935276.338430] [ 508741]     0 508741     7277      472    90112       19             0 cupsd
[9935276.338432] [ 508743]     0 508743    44761      573   114688        0             0 cups-browsed
[9935276.338434] [ 508756]     7 508756     3833      190    69632        9             0 dbus
[9935276.338437] [ 586286]  1007 586286     2876        1    57344      485             0 bash
[9935276.338440] [ 604958]     0 604958     3431      368    69632        0             0 sshd
[9935276.338442] [ 604991]     0 604991   698302     3447   466944        0          -900 snapd
[9935276.338444] [ 605033]  1007 605033    15313    10041   172032     1228             0 flye
[9935276.338447] [ 605656]  1006 605656     3506      371    69632        0             0 sshd
[9935276.338449] [ 605657]  1006 605657     2835      436    61440        0             0 bash
[9935276.338451] [ 606603]     0 606603     3503      366    65536        0             0 sshd
[9935276.338453] [ 606669]  1007 606669     3503      373    65536        0             0 sshd
[9935276.338455] [ 606670]  1007 606670     2864      468    57344        0             0 bash
[9935276.338457] [ 607423]  1007 607423     2525       71    53248        0             0 tmux: client
[9935276.338459] [ 608715]     0 608715     3504      367    65536        0             0 sshd
[9935276.338461] [ 608773]  1007 608773     3504      375    65536        0             0 sshd
[9935276.338464] [ 608774]  1007 608774     2864      468    53248        0             0 bash
[9935276.338466] [ 609926]  1007 609926     2525       71    53248        0             0 tmux: client
[9935276.338468] [ 614478]  1007 614478  1607528     9870   462848        0             0 circlator
[9935276.338471] [ 616253]  1007 616253      656       20    45056        0             0 sh
[9935276.338473] [ 616254]  1007 616254      656       30    45056        0             0 mhap.sh
[9935276.338475] [ 616286]  1007 616286      656       30    45056        0             0 mhap.sh
[9935276.338478] [ 616287]  1007 616287   122345   121115  1011712        0             0 mhapConvert
[9935276.338480] [ 616288]  1007 616288 16052969  8997646 73019392        0             0 java
[9935276.338482] [ 618695]  1007 618695     2461       61    53248        0             0 bash
[9935276.338485] [ 618696]  1007 618696 11616885 11205060 91369472        0             0 flye-minimap2
[9935276.338487] [ 618697]  1007 618697     1804      442    53248        0             0 flye-samtools
[9935276.338489] [ 618698]  1007 618698     1291       31    45056        0             0 flye-samtools
[9935276.338492] [ 618791]  1007 618791      656       19    40960        0             0 sh
[9935276.338494] [ 618792]  1007 618792      656       19    49152        0             0 spades.py
[9935276.338496] [ 618794]  1007 618794     5874     1940    81920        0             0 spades.py
[9935276.338498] [ 619278]  1007 619278      656       20    45056        0             0 sh
[9935276.338500] [ 619279]  1007 619279      656       28    45056        0             0 mhap.sh
[9935276.338502] [ 619311]  1007 619311      656       30    45056        0             0 mhap.sh
[9935276.338504] [ 619312]  1007 619312   122345   121116  1024000        0             0 mhapConvert
[9935276.338506] [ 619313]  1007 619313 16036328  8995063 73019392        0             0 java
[9935276.338509] [ 619897]  1007 619897    11162     6284   118784        0             0 spades-correcto
[9935276.338511] [ 619916]  1007 619916      656       20    40960        0             0 sh
[9935276.338513] [ 619917]  1007 619917    76500    23563   282624        0             0 bwa
[9935276.338515] [ 619955]     0 619955     4513       26    53248       40             0 cron
[9935276.338517] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1007.slice/session-21465.scope,task=flye-minimap2,pid=618696,uid=1007
[9935276.338536] Out of memory: Killed process 618696 (flye-minimap2) total-vm:46467540kB, anon-rss:44820240kB, file-rss:0kB, shmem-rss:0kB, UID:1007 pgtables:89228kB oom_score_adj:0
[9935278.690076] oom_reaper: reaped process 618696 (flye-minimap2), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

, which looks to me pretty relevant. The job is killed due to memory, as you suspected. Although, being unfamiliar with this error read out, I am not sure if this is an actual hard limit, or if the mems_allowed=0 somehow limits the job instead.

3 - interesting to know, but no, I am not using docker; just running the command directly.

4 - I have also tried to run the command directly, as you suggested, which yeilds the error: 

flye-minimap2 '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial8/00-assembly/draft_assembly.fasta' '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq' -x map-ont -t 4 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G | flye-samtools view -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial8/00-assembly/draft_assembly.fasta' -u - | flye-samtools sort -O bam -@ 4 -l 1 -m 4G -o 'minimap.bam'

I watched, as the job ran and and memory slowly increased from around 24 % while it is showing "[M::mm_idx_stat::26.837*1.99] distinct minimizers" to 25, 26, .. 30, .. 34%.

I looked away for a couple of minutes, then the job died.

Notably there were several other processes running which had 27.4 + 27.4 + 8.7 = 63.2 % of RAM used, meaning 36.8% remained free, so I guess we then reached that limit while I looked away.

In terms of clearer error messages from the direct command,

I have tried before, when the other processes were not present, which led to the same error, presumably after reaching a higher memory allocation rather than clipping out at 37% or so.

All together then, it looks like memory, as you suggested.

Given that, I wondered if you have a suggestion on roughly how much memory is needed for a 450 Mb genome that is highly repetitive. I had thought we would have enough given the general guidelines, but I can look to run on a cluster node with more RAM.

mikolmogorov commented 2 years ago

@JamesHowie14 normally minimap2 rarely takes more than 64Gb RAM even for a human-sized genome. I think the problem is that your genome just happened to be very repetitive, and this resulted in a lot of duplication in disjointigs, which the results in high memory usage by minimap2.

What kind of genome is it? In the flye.log file you sent previously, your genome size is set to 2.5G, is this accurate (since you mentioned it should be 450mb)?

Your read mean error rate is about 8%, which is closer to nano-raw mode. I would try running with --nano-raw, and maybe adding -m 5000, which should reduce duplications.

JamesHowie14 commented 2 years ago

Hi fendeglass, thanks for your reply. OK, interesting. Yes, from another Illumina dataset we estimate the genome as about 50% repetitive. The estimated genome size is about 450mb, I think my Flye log it is set to 438 [the exact "expected" size]. Tom's one is 2.5G. Mine is a psyllid, sort of like a leaf hopper. I'll try again with nano-raw, as you suggest, and also with -m 5000, and get back to you ;) Cheers!

JamesHowie14 commented 2 years ago

Dear Fenderglass, so I've now tried that. First, I ran:

flye --nano-raw 1_Cmel4_1_2_PCR1-2-3_SUP.fastq \
  --out-dir /mnt/raid10/.../Cmel4_FlyeTrial13/ \
  --genome-size 438m \
  --asm-coverage 50 \
  -m 5000 \
  --threads 12 \
  --iterations 1

It runs, but then at the minimap2 step the memory use just goes up, up, up, till it maxes out and crashes. I also tried a 2 thread version, with the same result, just after a slower memory build up.

The 12 thread readout was:

[2022-06-10 15:25:10] root: INFO: Starting Flye 2.9-b1768
[2022-06-10 15:25:10] root: DEBUG: Cmd: /home/jhowie/miniconda3/bin/flye --nano-raw 1_Cmel4_1_2_PCR1-2-3_SUP.fastq --out-dir /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial13/ --genome-size 438m --asm-coverage 50 -m 5000 --threads 12 --iterations 1
[2022-06-10 15:25:10] root: DEBUG: Python version: 3.8.5 (default, Sep  4 2020, 07:30:14)
[GCC 7.3.0]
[2022-06-10 15:25:10] root: INFO: >>>STAGE: configure
[2022-06-10 15:25:10] root: INFO: Configuring run
[2022-06-10 15:27:18] root: INFO: Total read length: 23150251229
[2022-06-10 15:27:18] root: INFO: Input genome size: 438000000
[2022-06-10 15:27:18] root: INFO: Estimated coverage: 52
[2022-06-10 15:27:18] root: INFO: Reads N50/N90: 1746 / 566
[2022-06-10 15:27:18] root: INFO: Selected minimum overlap: 5000
[2022-06-10 15:27:18] root: INFO: Using longest 50x reads for contig assembly
[2022-06-10 15:27:23] root: DEBUG: Min read length cutoff: 444
[2022-06-10 15:27:23] root: INFO: >>>STAGE: assembly
[2022-06-10 15:27:23] root: INFO: Assembling disjointigs
[2022-06-10 15:27:23] root: DEBUG: -----Begin assembly log------
[2022-06-10 15:27:23] root: DEBUG: Running: flye-modules assemble --reads /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq --out-asm /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial13/00-assembly/draft_assembly.fasta --config /home/jhowie/miniconda3/lib/python3.8/site-packages/flye/config/bin_cfg/asm_raw_reads.cfg --log /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial13/flye.log --threads 12 --genome-size 438000000 --min-ovlp 5000 --min-read 444
[2022-06-10 15:27:23] DEBUG: Build date: Feb 22 2022 03:49:40
[2022-06-10 15:27:23] DEBUG: Total RAM: 125 Gb
[2022-06-10 15:27:23] DEBUG: Available RAM: 123 Gb
[2022-06-10 15:27:23] DEBUG: Total CPUs: 48
[2022-06-10 15:27:23] DEBUG: Loading /home/jhowie/miniconda3/lib/python3.8/site-packages/flye/config/bin_cfg/asm_raw_reads.cfg
[2022-06-10 15:27:23] DEBUG: Loading /home/jhowie/miniconda3/lib/python3.8/site-packages/flye/config/bin_cfg/asm_defaults.cfg
[2022-06-10 15:27:23] DEBUG:    big_genome_threshold=29000000
[2022-06-10 15:27:23] DEBUG:    meta_read_filter_kmer_freq=100
[2022-06-10 15:27:23] DEBUG:    chain_large_gap_penalty=2
[2022-06-10 15:27:23] DEBUG:    chain_small_gap_penalty=0.5
[2022-06-10 15:27:23] DEBUG:    chain_gap_jump_threshold=100
[2022-06-10 15:27:23] DEBUG:    max_coverage_drop_rate=5
[2022-06-10 15:27:23] DEBUG:    max_extensions_drop_rate=5
[2022-06-10 15:27:23] DEBUG:    chimera_window=100
[2022-06-10 15:27:23] DEBUG:    chimera_overhang=1000
[2022-06-10 15:27:23] DEBUG:    min_reads_in_disjointig=4
[2022-06-10 15:27:23] DEBUG:    max_inner_reads=10
[2022-06-10 15:27:23] DEBUG:    max_inner_fraction=0.25
[2022-06-10 15:27:23] DEBUG:    max_separation=500
[2022-06-10 15:27:23] DEBUG:    unique_edge_length=50000
[2022-06-10 15:27:23] DEBUG:    min_repeat_res_support=0.51
[2022-06-10 15:27:23] DEBUG:    out_paths_ratio=5
[2022-06-10 15:27:23] DEBUG:    graph_cov_drop_rate=5
[2022-06-10 15:27:23] DEBUG:    coverage_estimate_window=100
[2022-06-10 15:27:23] DEBUG:    max_bubble_length=50000
[2022-06-10 15:27:23] DEBUG:    loop_coverage_rate=1.5
[2022-06-10 15:27:23] DEBUG:    repeat_edge_cov_mult=1.75
[2022-06-10 15:27:23] DEBUG:    weak_detach_rate=5
[2022-06-10 15:27:23] DEBUG:    tip_coverage_rate=2
[2022-06-10 15:27:23] DEBUG:    tip_length_rate=2
[2022-06-10 15:27:23] DEBUG:    output_gfa_before_rr=0
[2022-06-10 15:27:23] DEBUG:    low_cutoff_warning=1
[2022-06-10 15:27:23] DEBUG:    kmer_size=17
[2022-06-10 15:27:23] DEBUG:    use_minimizers=0
[2022-06-10 15:27:23] DEBUG:    reads_base_alignment=0
[2022-06-10 15:27:23] DEBUG:    meta_read_top_kmer_rate=0.40
[2022-06-10 15:27:23] DEBUG:    maximum_jump=1500
[2022-06-10 15:27:23] DEBUG:    maximum_overhang=1500
[2022-06-10 15:27:23] DEBUG:    repeat_kmer_rate=100
[2022-06-10 15:27:23] DEBUG:    assemble_ovlp_divergence=0.10
[2022-06-10 15:27:23] DEBUG:    assemble_divergence_relative=1
[2022-06-10 15:27:23] DEBUG:    repeat_graph_ovlp_divergence=0.08
[2022-06-10 15:27:23] DEBUG:    read_align_ovlp_divergence=0.25
[2022-06-10 15:27:23] DEBUG:    hpc_scoring_on=0
[2022-06-10 15:27:23] DEBUG:    add_unassembled_reads=0
[2022-06-10 15:27:23] DEBUG:    extend_contigs_with_repeats=0
[2022-06-10 15:27:23] DEBUG:    min_read_cov_cutoff=3
[2022-06-10 15:27:23] DEBUG:    short_tip_length=20000
[2022-06-10 15:27:23] DEBUG:    long_tip_length=100000
[2022-06-10 15:27:23] DEBUG: Running with k-mer size: 17
[2022-06-10 15:27:23] DEBUG: Running with minimum overlap 5000
[2022-06-10 15:27:23] DEBUG: Metagenome mode: N
[2022-06-10 15:27:23] DEBUG: Short mode: N
[2022-06-10 15:27:23] INFO: Reading sequences
[2022-06-10 15:28:28] DEBUG: Building positional index
[2022-06-10 15:28:31] DEBUG: Total sequence: 3915919669 bp
[2022-06-10 15:28:32] INFO: Counting k-mers:
[2022-06-10 15:29:47] DEBUG: Updating k-mer histogram
[2022-06-10 15:30:52] DEBUG: Hash size: 23322863
[2022-06-10 15:30:52] DEBUG: Total k-mers 904133692
[2022-06-10 15:30:53] INFO: Filling index table (1/2)
[2022-06-10 15:33:02] DEBUG: Mean k-mer frequency: 30.1135
[2022-06-10 15:33:02] DEBUG: Repetitive k-mer frequency: 3011
[2022-06-10 15:33:02] DEBUG: Filtered 382356617 repetitive k-mers (0.24751)
[2022-06-10 15:33:04] INFO: Filling index table (2/2)
[2022-06-10 15:35:19] DEBUG: Sorting k-mer index
[2022-06-10 15:35:29] DEBUG: Selected k-mers: 69477999
[2022-06-10 15:35:29] DEBUG: Index size: 1177392109
[2022-06-10 15:35:29] DEBUG: Mean k-mer index frequency: 16.9463
[2022-06-10 15:35:29] DEBUG: Peak RAM usage: 24 Gb
[2022-06-10 15:35:29] DEBUG: Estimating k-mer identity bias
[2022-06-10 15:36:07] DEBUG: Initial divergence estimate : 0.0935919
[2022-06-10 15:36:07] DEBUG: Relative threshold: Y
[2022-06-10 15:36:07] DEBUG: Max divergence threshold set to 0.193592
[2022-06-10 15:36:07] INFO: Extending reads
[2022-06-10 15:36:07] DEBUG: Estimating overlap coverage
[2022-06-10 15:40:10] INFO: Overlap-based coverage: 2
[2022-06-10 15:40:10] INFO: Median overlap divergence: 0.0949632
[2022-06-10 15:40:10] DEBUG: Sequence divergence distribution:

    |                  *                   |
    |                * *                   |
    |                ***                   |
    |                ***                   |
    |                ***                   |
    |              * *****                 |
    |              *******                 |
    |             ********                 |
    |             ********                 |
    |             **********               |
    |             ***********              |
    |             ***********              |
    |             ***********              |
    |            **************            |
    |           *****************          |
    |           *****************          |
    |          ****************** *        |
    |          ********************** **   |
    |       ***************************** *| *   *
    |     ****************************************  *********** *   * **  **** ** *
    ----------------------------------------------------------------------------------------------------
    0%        5%        10%       15%       20%       25%       30%       35%       40%       45%

    Q25 = 0.079, Q50 = 0.095, Q75 = 0.12

[2022-06-10 15:40:11] DEBUG: Assembled disjointig 1
        With 2 reads
        Start read: +656630b4-ddb7-4dbe-b7a3-0842c611b729
        At position: 1
        leftTip: 1 rightTip: 1
        Suspicious: 1
        Short ext: 1
        Mean extensions: 1
        Avg overlap len: 3312
        Min overlap len: 3312
        Inner reads: 0
        Length: 13593
[2022-06-10 15:40:11] DEBUG: Inner: 6 covered: 50 total: 868712

...

[2022-06-10 20:37:54] DEBUG: Assembled disjointig 44143
        With 4 reads
        Start read: +74589440-38af-47d4-b759-f37320e104a6
        At position: 3
        leftTip: 0 rightTip: 0
        Suspicious: 2
        Short ext: 2
        Mean extensions: 4
        Avg overlap len: 4763
        Min overlap len: 1214
        Inner reads: 0
        Length: 20675
[2022-06-10 20:37:54] DEBUG: Inner: 544386 covered: 803906 total: 868712
[2022-06-10 20:37:57] INFO: Assembled 44143 disjointigs
[2022-06-10 20:37:58] INFO: Generating sequence
[2022-06-10 20:38:15] DEBUG: Building positional index
[2022-06-10 20:38:15] DEBUG: Total sequence: 1148840534 bp
[2022-06-10 20:39:01] DEBUG: Mean k-mer frequency: 2.80929
[2022-06-10 20:39:01] DEBUG: Repetitive k-mer frequency: 280
[2022-06-10 20:39:01] DEBUG: Filtered 177019530 repetitive k-mers (0.154186)
[2022-06-10 20:39:45] DEBUG: Sorting k-mer index
[2022-06-10 20:39:53] DEBUG: Selected k-mers: 408506424
[2022-06-10 20:39:53] DEBUG: K-mer index size: 971070573
[2022-06-10 20:39:53] DEBUG: Mean k-mer frequency: 2.37712
[2022-06-10 20:39:53] DEBUG: Minimizer rate: 1.18307
[2022-06-10 20:39:53] INFO: Filtering contained disjointigs
[2022-06-10 20:50:15] DEBUG: Computing transitive closure for overlaps
[2022-06-10 20:50:16] DEBUG: Found 1001808 overlaps
[2022-06-10 20:50:16] DEBUG: Left 412352 overlaps after filtering
[2022-06-10 20:50:16] INFO: Contained seqs: 2387
[2022-06-10 20:50:18] DEBUG: Writing FASTA
[2022-06-10 20:50:25] DEBUG: Peak RAM usage: 31 Gb
-----------End assembly log------------
[2022-06-10 20:50:31] root: DEBUG: Disjointigs length: 1110222351, N50: 35360
[2022-06-10 20:50:31] root: INFO: >>>STAGE: consensus
[2022-06-10 20:50:31] root: INFO: Running Minimap2
[2022-06-10 21:32:11] root: ERROR: Error running minimap2, terminating. See the alignment error log  for details: /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial13/10-consensus/minimap.stderr
[2022-06-10 21:32:11] root: ERROR: Command '['/bin/bash', '-c', "set -eo pipefail; flye-minimap2 '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial13/00-assembly/draft_assembly.fasta' '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq' -x map-ont -t 12 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G | flye-samtools view -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial13/00-assembly/draft_assembly.fasta' -u - | flye-samtools sort -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial13/10-consensus/sort_220610_205031' -O bam -@ 4 -l 1 -m 4G -o '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial13/10-consensus/minimap.bam'"]' returned non-zero exit status 1.
[2022-06-10 21:32:11] root: ERROR: Pipeline aborted

The difference to before, is we now have : non-zero exit status 1, not 37.

For fun, and to get more reads, I then ran -m 2500; which goes back to non-zero exit status 37. The readout:

[2022-06-13 15:17:25] root: INFO: Starting Flye 2.9-b1768
[2022-06-13 15:17:25] root: DEBUG: Cmd: /home/jhowie/miniconda3/bin/flye --nano-raw 1_Cmel4_1_2_PCR1-2-3_SUP.fastq --out-dir /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial14/ --genome-size 438m --asm-coverage 50 -m 2500 --threads 24 --iterations 1
[2022-06-13 15:17:25] root: DEBUG: Python version: 3.8.5 (default, Sep  4 2020, 07:30:14) 
[GCC 7.3.0]
[2022-06-13 15:17:25] root: INFO: >>>STAGE: configure
[2022-06-13 15:17:25] root: INFO: Configuring run
[2022-06-13 15:19:34] root: INFO: Total read length: 23150251229
[2022-06-13 15:19:34] root: INFO: Input genome size: 438000000
[2022-06-13 15:19:34] root: INFO: Estimated coverage: 52
[2022-06-13 15:19:34] root: INFO: Reads N50/N90: 1746 / 566
[2022-06-13 15:19:34] root: INFO: Selected minimum overlap: 2500
[2022-06-13 15:19:34] root: INFO: Using longest 50x reads for contig assembly
[2022-06-13 15:19:39] root: DEBUG: Min read length cutoff: 444
[2022-06-13 15:19:39] root: INFO: >>>STAGE: assembly
[2022-06-13 15:19:39] root: INFO: Assembling disjointigs
[2022-06-13 15:19:39] root: DEBUG: -----Begin assembly log------
[2022-06-13 15:19:39] root: DEBUG: Running: flye-modules assemble --reads /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq --out-asm /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial14/00-assembly/draft_assembly.fasta --config /home/jhowie/miniconda3/lib/python3.8/site-packages/flye/config/bin_cfg/asm_raw_reads.cfg --log /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial14/flye.log --threads 24 --genome-size 438000000 --min-ovlp 2500 --min-read 444
[2022-06-13 15:19:39] DEBUG: Build date: Feb 22 2022 03:49:40
[2022-06-13 15:19:39] DEBUG: Total RAM: 125 Gb
[2022-06-13 15:19:39] DEBUG: Available RAM: 121 Gb
[2022-06-13 15:19:39] DEBUG: Total CPUs: 48
[2022-06-13 15:19:39] DEBUG: Loading /home/jhowie/miniconda3/lib/python3.8/site-packages/flye/config/bin_cfg/asm_raw_reads.cfg
[2022-06-13 15:19:39] DEBUG: Loading /home/jhowie/miniconda3/lib/python3.8/site-packages/flye/config/bin_cfg/asm_defaults.cfg
[2022-06-13 15:19:39] DEBUG:    big_genome_threshold=29000000
[2022-06-13 15:19:39] DEBUG:    meta_read_filter_kmer_freq=100
[2022-06-13 15:19:39] DEBUG:    chain_large_gap_penalty=2
[2022-06-13 15:19:39] DEBUG:    chain_small_gap_penalty=0.5
[2022-06-13 15:19:39] DEBUG:    chain_gap_jump_threshold=100
[2022-06-13 15:19:39] DEBUG:    max_coverage_drop_rate=5
[2022-06-13 15:19:39] DEBUG:    max_extensions_drop_rate=5
[2022-06-13 15:19:39] DEBUG:    chimera_window=100
[2022-06-13 15:19:39] DEBUG:    chimera_overhang=1000
[2022-06-13 15:19:39] DEBUG:    min_reads_in_disjointig=4
[2022-06-13 15:19:39] DEBUG:    max_inner_reads=10
[2022-06-13 15:19:39] DEBUG:    max_inner_fraction=0.25
[2022-06-13 15:19:39] DEBUG:    max_separation=500
[2022-06-13 15:19:39] DEBUG:    unique_edge_length=50000
[2022-06-13 15:19:39] DEBUG:    min_repeat_res_support=0.51
[2022-06-13 15:19:39] DEBUG:    out_paths_ratio=5
[2022-06-13 15:19:39] DEBUG:    graph_cov_drop_rate=5
[2022-06-13 15:19:39] DEBUG:    coverage_estimate_window=100
[2022-06-13 15:19:39] DEBUG:    max_bubble_length=50000
[2022-06-13 15:19:39] DEBUG:    loop_coverage_rate=1.5
[2022-06-13 15:19:39] DEBUG:    repeat_edge_cov_mult=1.75
[2022-06-13 15:19:39] DEBUG:    weak_detach_rate=5
[2022-06-13 15:19:39] DEBUG:    tip_coverage_rate=2
[2022-06-13 15:19:39] DEBUG:    tip_length_rate=2
[2022-06-13 15:19:39] DEBUG:    output_gfa_before_rr=0
[2022-06-13 15:19:39] DEBUG:    low_cutoff_warning=1
[2022-06-13 15:19:39] DEBUG:    kmer_size=17
[2022-06-13 15:19:39] DEBUG:    use_minimizers=0
[2022-06-13 15:19:39] DEBUG:    reads_base_alignment=0
[2022-06-13 15:19:39] DEBUG:    meta_read_top_kmer_rate=0.40
[2022-06-13 15:19:39] DEBUG:    maximum_jump=1500
[2022-06-13 15:19:39] DEBUG:    maximum_overhang=1500
[2022-06-13 15:19:39] DEBUG:    repeat_kmer_rate=100
[2022-06-13 15:19:39] DEBUG:    assemble_ovlp_divergence=0.10
[2022-06-13 15:19:39] DEBUG:    assemble_divergence_relative=1
[2022-06-13 15:19:39] DEBUG:    repeat_graph_ovlp_divergence=0.08
[2022-06-13 15:19:39] DEBUG:    read_align_ovlp_divergence=0.25
[2022-06-13 15:19:39] DEBUG:    hpc_scoring_on=0
[2022-06-13 15:19:39] DEBUG:    add_unassembled_reads=0
[2022-06-13 15:19:39] DEBUG:    extend_contigs_with_repeats=0
[2022-06-13 15:19:39] DEBUG:    min_read_cov_cutoff=3
[2022-06-13 15:19:39] DEBUG:    short_tip_length=20000
[2022-06-13 15:19:39] DEBUG:    long_tip_length=100000
[2022-06-13 15:19:39] DEBUG: Running with k-mer size: 17
[2022-06-13 15:19:39] DEBUG: Running with minimum overlap 2500
[2022-06-13 15:19:39] DEBUG: Metagenome mode: N
[2022-06-13 15:19:39] DEBUG: Short mode: N
[2022-06-13 15:19:39] INFO: Reading sequences
[2022-06-13 15:21:01] DEBUG: Building positional index
[2022-06-13 15:21:04] DEBUG: Total sequence: 8418638046 bp
[2022-06-13 15:21:05] INFO: Counting k-mers:
[2022-06-13 15:22:40] DEBUG: Updating k-mer histogram
[2022-06-13 15:24:09] DEBUG: Hash size: 65888693
[2022-06-13 15:24:09] DEBUG: Total k-mers 1417619498
[2022-06-13 15:24:10] INFO: Filling index table (1/2)
[2022-06-13 15:27:02] DEBUG: Mean k-mer frequency: 47.0223
[2022-06-13 15:27:02] DEBUG: Repetitive k-mer frequency: 4702
[2022-06-13 15:27:02] DEBUG: Filtered 950381572 repetitive k-mers (0.286516)
[2022-06-13 15:27:06] INFO: Filling index table (2/2)
[2022-06-13 15:30:03] DEBUG: Sorting k-mer index
[2022-06-13 15:30:23] DEBUG: Selected k-mers: 89084835
[2022-06-13 15:30:23] DEBUG: Index size: 2371221948
[2022-06-13 15:30:23] DEBUG: Mean k-mer index frequency: 26.6176
[2022-06-13 15:30:23] DEBUG: Peak RAM usage: 36 Gb
[2022-06-13 15:30:23] DEBUG: Estimating k-mer identity bias
[2022-06-13 15:30:35] DEBUG: Initial divergence estimate : 0.0942718
[2022-06-13 15:30:35] DEBUG: Relative threshold: Y
[2022-06-13 15:30:35] DEBUG: Max divergence threshold set to 0.194272
[2022-06-13 15:30:35] INFO: Extending reads
[2022-06-13 15:30:35] INFO: Extending reads
[2022-06-13 15:30:35] DEBUG: Estimating overlap coverage
[2022-06-13 15:33:26] INFO: Overlap-based coverage: 6
[2022-06-13 15:33:26] INFO: Median overlap divergence: 0.0968613
[2022-06-13 15:33:26] DEBUG: Sequence divergence distribution: 

    |                 *                    |                                                             
    |                **                    |                                                             
    |                ** *                  |                                                             
    |                ****  *               |                                                             
    |               ****** *               |                                                             
    |               ********               |                                                             
    |             * ********               |                                                             
    |             * *********              |                                                             
    |             ***********              |                                                             
    |             ***********              |                                                             
    |            ************              |                                                             
    |            *************             |                                                             
    |            *************             |                                                             
    |           ****************           |
    |           *****************   *      |                                                             
    |         ********************* *      |                                                             
    |         ********************* **     |                                                             
    |       * ************************   * |       *                                                     
    |   * *********************************|*  ******** ********  *           * *                        
    ----------------------------------------------------------------------------------------------------
    0%        5%        10%       15%       20%       25%       30%       35%       40%       45%       

    Q25 = 0.079, Q50 = 0.097, Q75 = 0.12

[2022-06-13 15:33:27] DEBUG: Assembled disjointig 1
        With 1 reads
        Start read: +90891108-1264-4a41-a871-d0bdb7bd835c
        At position: 1
        leftTip: 1 rightTip: 1
        Suspicious: 0
        Short ext: 0
        Mean extensions: 2
        Avg overlap len: 0
        Min overlap len: 0
        Inner reads: 0
        Length: 2681
[2022-06-13 15:33:27] DEBUG: Inner: 2 covered: 64 total: 3548400

...

[2022-06-13 19:45:42] DEBUG: Assembled disjointig 80208
        With 3 reads
        Start read: +77b4ed94-3970-485d-92c1-770650965ba0
        At position: 2
        leftTip: 0 rightTip: 0
        Suspicious: 2
        Short ext: 2
        Mean extensions: 288
        Avg overlap len: 1643
        Min overlap len: 1643
        Inner reads: 0
        Length: 11092

[2022-06-13 19:45:42] DEBUG: Inner: 2821538 covered: 2692332 total: 3548400
[2022-06-13 19:46:29] INFO: Assembled 80208 disjointigs
[2022-06-13 19:46:31] INFO: Generating sequence
[2022-06-13 19:46:49] DEBUG: Building positional index
[2022-06-13 19:46:49] DEBUG: Total sequence: 1500542806 bp
[2022-06-13 19:47:26] DEBUG: Mean k-mer frequency: 3.20001
[2022-06-13 19:47:26] DEBUG: Repetitive k-mer frequency: 320
[2022-06-13 19:47:26] DEBUG: Filtered 256470176 repetitive k-mers (0.171074)
[2022-06-13 19:47:58] DEBUG: Sorting k-mer index
[2022-06-13 19:48:08] DEBUG: Selected k-mers: 468286810
[2022-06-13 19:48:08] DEBUG: K-mer index size: 1242709094
[2022-06-13 19:48:08] DEBUG: Mean k-mer frequency: 2.65373
[2022-06-13 19:48:08] DEBUG: Minimizer rate: 1.20748
[2022-06-13 19:48:08] INFO: Filtering contained disjointigs
[2022-06-13 19:54:55] DEBUG: Computing transitive closure for overlaps
[2022-06-13 19:54:56] DEBUG: Found 1555226 overlaps
[2022-06-13 19:54:56] DEBUG: Left 651144 overlaps after filtering
[2022-06-13 19:54:56] INFO: Contained seqs: 9225
[2022-06-13 19:54:59] DEBUG: Writing FASTA
[2022-06-13 19:55:06] DEBUG: Peak RAM usage: 41 Gb
-----------End assembly log------------
[2022-06-13 19:55:16] root: DEBUG: Disjointigs length: 1410083426, N50: 28321
[2022-06-13 19:55:16] root: INFO: >>>STAGE: consensus
[2022-06-13 19:55:16] root: INFO: Running Minimap2
[2022-06-13 20:13:34] root: ERROR: Error running minimap2, terminating. See the alignment error log  for details: /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial14/10-consensus/minimap.stderr
[2022-06-13 20:13:34] root: ERROR: Command '['/bin/bash', '-c', "set -eo pipefail; flye-minimap2 '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial14/00-assembly/draft_assembly.fasta' '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq' -x map-ont -t 24 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G | flye-samtools view -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial14/00-assembly/draft_assembly.fasta' -u - | flye-samtools sort -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial14/10-consensus/sort_220613_195516' -O bam -@ 4 -l 1 -m 4G -o '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial14/10-consensus/minimap.bam'"]' returned non-zero exit status 137.
[2022-06-13 20:13:34] root: ERROR: Pipeline aborted

It seems it is a memory issue at the minimap2 step. Nonetheless, if you have any other ideas, I'd be happy to hear. From my side, a couple of thoughts:

As a tangential question:

I ran -m 5000, then -m 2500; I also tried for nano-raw and nano-hq; the results are:

--nano-hq -m 5000: Result = N50 33,762, 1172292954 bp = 2.67x inflated; 46,831 disjointigs --nano-raw -m 5000: Result = N50 33,630, 1110222351 bp = 2.53x inflated; 44,143 disjointigs

--nano-raw -m 2500: Result = N50 28,321, 1410083426 bp = 3.22x inflated; 80,208 disjointigs --nano-hq -m 2500: Result = N50 24,367, 1721925195 bp = 3.93x inflated; 109,982 disjointigs

From this I saw, that the switch to nano-raw gives similar N50, but fewer disjointigs and less inflated genome at this step. If I switch to -m 2500, in both cases, the N50 reduces, and the disjointig count and genome inflantion increases, at this step.

From this, I have a question:

Thanks for any help :)

mikolmogorov commented 2 years ago

@JamesHowie14 thanks for the update. I think it could be both a lot of repeats and high heterozygosity. Both a very complicated to assemble. I don't think error-corrected reads will make much difference, but maybe worth trying. Disjointigs may contain duplications and errors, so they should not be used for downstream analysis.

The best solution would really be trying to find a machine with more memory and let minimap2 finish.. One other thing to try running purge_haplotigs (https://bitbucket.org/mroachawri/purge_haplotigs/src/master/) on consensus.fasta, which should collapse the duplications. This at least will give an idea how much duplicated sequence is there.

JamesHowie14 commented 2 years ago

Dear fenderglass, that sounds like good advice to me. I'll definitely try out both! Thanks :)

mikolmogorov commented 1 year ago

Closed due to inactivity, feel free to reopen if there are any follow-ups.