search

mikolmogorov / Flye

De novo assembler for single molecule sequencing reads using repeat graphs
Other
743 stars 164 forks source link

thread exception - system ran out of memory #710

Open samuelmontgomery opened 2 weeks ago

samuelmontgomery commented 2 weeks ago

Hi,

I am running flye 2.9.4 on a bacterial genome (~7MB) and flye is crashing during the polishing step I am running the following command:

flye --nano-hq reads.fastq.gz --scaffold --genome-size 7000000 --asm-coverage 50 --out-dir flye --threads 16

The error seems to state the system has run out of memory - I have 64GB on my VM, and it is using ~8Gb of raw reads, but I have assembled fungal genomes with ~50Gb of reads with no issues

The output is below:

`[2024-06-19 04:55:08] INFO: Starting Flye 2.9.4-b1799 [2024-06-19 04:55:08] INFO: Resuming previous run [2024-06-19 04:55:08] INFO: >>>STAGE: polishing [2024-06-19 04:55:08] INFO: Polishing genome (1/1) [2024-06-19 04:55:08] INFO: Running minimap2 [2024-06-19 05:15:22] INFO: Separating alignment into bubbles Traceback (most recent call last): File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/bin/flye", line 11, in sys.exit(main()) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/main.py", line 758, in main _run(args) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/main.py", line 495, in _run jobs[i].run() File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/main.py", line 319, in run pol.polish(self.in_contigs, self.args.reads, self.polishing_dir, File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/polishing/polish.py", line 94, in polish make_bubbles(alignment_file, contigs_info, prev_assembly, File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/polishing/bubbles.py", line 135, in make_bubbles process_in_parallel(_thread_worker, (aln_reader, chunk_feeder, contigs_info, err_mode, File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/utils/utils.py", line 49, in process_in_parallel logger.error("Looks like the system ran out of memory") NameError: name 'logger' is not defined [2024-06-19 05:31:28] ERROR: Thread exception [2024-06-19 05:31:28] ERROR: Traceback (most recent call last): File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/polishing/bubbles.py", line 86, in _thread_worker mean_cov = aln_reader.get_median_depth(ctg_region.ctg_id, ctg_region.start, File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/utils/sam_parser.py", line 350, in get_median_depth contig_str = self.ref_fasta[parsed_contig] File "", line 2, in getitem File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

Process Process-6: Traceback (most recent call last): File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/polishing/bubbles.py", line 86, in _thread_worker mean_cov = aln_reader.get_median_depth(ctg_region.ctg_id, ctg_region.start, File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/utils/sam_parser.py", line 350, in get_median_depth contig_str = self.ref_fasta[parsed_contig] File "", line 2, in getitem File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/polishing/bubbles.py", line 114, in _thread_worker error_queue.put(e) File "", line 2, in put File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe [2024-06-19 05:31:28] ERROR: Thread exception [2024-06-19 05:31:28] ERROR: Traceback (most recent call last): File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/polishing/bubbles.py", line 86, in _thread_worker mean_cov = aln_reader.get_median_depth(ctg_region.ctg_id, ctg_region.start, File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/utils/sam_parser.py", line 350, in get_median_depth contig_str = self.ref_fasta[parsed_contig] File "", line 2, in getitem File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

Process Process-5: Traceback (most recent call last): File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/polishing/bubbles.py", line 86, in _thread_worker mean_cov = aln_reader.get_median_depth(ctg_region.ctg_id, ctg_region.start, File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/utils/sam_parser.py", line 350, in get_median_depth contig_str = self.ref_fasta[parsed_contig] File "", line 2, in getitem File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/polishing/bubbles.py", line 114, in _thread_worker error_queue.put(e) File "", line 2, in put File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe [2024-06-19 05:31:34] ERROR: Thread exception [2024-06-19 05:31:34] ERROR: Traceback (most recent call last): File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/polishing/bubbles.py", line 86, in _thread_worker mean_cov = aln_reader.get_median_depth(ctg_region.ctg_id, ctg_region.start, File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/utils/sam_parser.py", line 350, in get_median_depth contig_str = self.ref_fasta[parsed_contig] File "", line 2, in getitem File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

Process Process-3: Traceback (most recent call last): File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/polishing/bubbles.py", line 86, in _thread_worker mean_cov = aln_reader.get_median_depth(ctg_region.ctg_id, ctg_region.start, File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/utils/sam_parser.py", line 350, in get_median_depth contig_str = self.ref_fasta[parsed_contig] File "", line 2, in getitem File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/polishing/bubbles.py", line 114, in _thread_worker error_queue.put(e) File "", line 2, in put File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe [2024-06-19 05:31:42] ERROR: Thread exception [2024-06-19 05:31:42] ERROR: Traceback (most recent call last): File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/polishing/bubbles.py", line 86, in _thread_worker mean_cov = aln_reader.get_median_depth(ctg_region.ctg_id, ctg_region.start, File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/utils/sam_parser.py", line 350, in get_median_depth contig_str = self.ref_fasta[parsed_contig] File "", line 2, in getitem File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

Process Process-4: Traceback (most recent call last): File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/polishing/bubbles.py", line 86, in _thread_worker mean_cov = aln_reader.get_median_depth(ctg_region.ctg_id, ctg_region.start, File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/utils/sam_parser.py", line 350, in get_median_depth contig_str = self.ref_fasta[parsed_contig] File "", line 2, in getitem File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/polishing/bubbles.py", line 114, in _thread_worker error_queue.put(e) File "", line 2, in put File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe [2024-06-19 05:36:29] ERROR: Thread exception [2024-06-19 05:36:29] ERROR: Traceback (most recent call last): File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/polishing/bubbles.py", line 86, in _thread_worker mean_cov = aln_reader.get_median_depth(ctg_region.ctg_id, ctg_region.start, File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/utils/sam_parser.py", line 350, in get_median_depth contig_str = self.ref_fasta[parsed_contig] File "", line 2, in getitem File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

Process Process-7: Traceback (most recent call last): File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/polishing/bubbles.py", line 86, in _thread_worker mean_cov = aln_reader.get_median_depth(ctg_region.ctg_id, ctg_region.start, File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/utils/sam_parser.py", line 350, in get_median_depth contig_str = self.ref_fasta[parsed_contig] File "", line 2, in getitem File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/site-packages/flye/polishing/bubbles.py", line 114, in _thread_worker error_queue.put(e) File "", line 2, in put File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/home/ubuntu/scratch/software/miniforge3/envs/bacass/lib/python3.10/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe`

mikolmogorov commented 6 days ago

Could you please send the full flye.log and tell a bit more about the machine (hardware, OS version etc)? Is the issue reproducible - e.g if you run Flye on the same input, it crashes again?

samuelmontgomery commented 6 days ago

It's running on a VM using Ubuntu 22.04 with 16 cores and 64GB of RAM It was reproducible - continued to give the error

Re-running flye after filtering the reads down to ~6GB seems to resolve the issue

Unfortunately re-running flye seems to have overwritten the log with the error, I will see if I can reproduce it