Run OSError: [Errno 24] Too many open files Mac OS M

[husamia]

I am running version 1.5.0 on Mac OS with M chip the following command

dysgu run --low-mem --mode pe -o NBVY8432_GRCh38_dysgu.vcf -f vcf -p 8 --metrics --keep-small --clean Homo_sapie ns_assembly38.fasta.gz NBVY8432_GRCh38_dysgu NBVY8432_GRCh38.bam

here is the error '' 2023-08-16 11:48:55,745 [INFO ] [dysgu-run] Version: 1.5.0 2023-08-16 11:48:55,745 [INFO ] run --low-mem --mode pe -o NBVY8432_GRCh38_dysgu.vcf -f vcf -p 8 --metrics --keep-small --clean Homo_sapiens_assembly38.fasta.gz NBVY8432_GRCh38_dysgu NBVY8432_GRCh38.bam 2023-08-16 11:48:55,745 [INFO ] Destination: NBVY8432_GRCh38_dysgu 2023-08-16 11:54:07,557 [INFO ] dysgu fetch NBVY8432_GRCh38.bam written to NBVY8432_GRCh38_dysgu/NBVY8432_GRCh38.dysgu_reads.bam, n=23864581, time=0:05:11 h:m:s 2023-08-16 11:54:07,557 [INFO ] Input file is: NBVY8432_GRCh38_dysgu/NBVY8432_GRCh38.dysgu_reads.bam 2023-08-16 11:54:07,606 [WARNING] Warning: no @RG, using input file name as sample name for output: NBVY8432_GRCh38.dysgu_reads 2023-08-16 11:54:07,606 [INFO ] Sample name: NBVY8432_GRCh38.dysgu_reads 2023-08-16 11:54:07,606 [INFO ] Writing SVs to NBVY8432_GRCh38_dysgu.vcf 2023-08-16 11:54:07,607 [INFO ] Running pipeline 2023-08-16 11:54:07,894 [INFO ] Calculating insert size. Removed 31 outliers with insert size >= 887.0 2023-08-16 11:54:07,900 [INFO ] Inferred read length 150.0, insert median 399, insert stdev 90 2023-08-16 11:54:07,901 [INFO ] Max clustering dist 849 2023-08-16 11:54:07,901 [INFO ] Minimum support 3 2023-08-16 11:54:07,901 [INFO ] Building graph with clustering 849 bp Traceback (most recent call last): File "/Users/husw5y/miniconda3/bin/dysgu", line 8, in sys.exit(cli()) File "/Users/husw5y/miniconda3/lib/python3.10/site-packages/click/core.py", line 1157, in call return self.main(args, kwargs) File "/Users/husw5y/miniconda3/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/Users/husw5y/miniconda3/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/husw5y/miniconda3/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) File "/Users/husw5y/miniconda3/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(args, *kwargs) File "/Users/husw5y/miniconda3/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), args, **kwargs) File "/Users/husw5y/miniconda3/lib/python3.10/site-packages/dysgu/main.py", line 256, in run_pipeline cluster.cluster_reads(ctx.obj) File "dysgu/cluster.pyx", line 1079, in dysgu.cluster.cluster_reads File "dysgu/cluster.pyx", line 755, in dysgu.cluster.pipe1 File "dysgu/graph.pyx", line 1135, in dysgu.graph.construct_graph File "dysgu/graph.pyx", line 1151, in dysgu.graph.construct_graph File "dysgu/extra_metrics.pyx", line 30, in dysgu.extra_metrics.BadClipCounter.init OSError: [Errno 24] Too many open files: 'NBVY8432_GRCh38_dysgu/249.badclip.bin' `'

[kcleal]

Hi @husamia, I think this can be fixed by increasing the ulimit. To see what it is set at now use ulimit -n, and increase to a larger number e.g. ulimit -n 4096

[husamia]

@kcleal the ulimit -n was 256 so I ncreasted it to ulimit -n 4096

It didn;t solve my problem

'` (base) husw5y@EA22-00453 Sequencing % ulimit -n 4096 (base) husw5y@EA22-00453 Sequencing % ulimit -n 4096

(base) husw5y@EA22-00453 Sequencing % dysgu run --mode pe -o NBVY8432_GRCh38_dysgu.vcf -f vcf -p 8 --metrics --clean Homo_sapiens_assembly38.fasta.gz NBVY8432_GRCh38_dysgu NBVY8432_GRCh38.bam 2023-08-17 13:45:57,706 [INFO ] [dysgu-run] Version: 1.5.0 2023-08-17 13:45:57,706 [INFO ] run --mode pe -o NBVY8432_GRCh38_dysgu.vcf -f vcf -p 8 --metrics --clean Homo_sapiens_assembly38.fasta.gz NBVY8432_GRCh38_dysgu NBVY8432_GRCh38.bam 2023-08-17 13:45:57,706 [INFO ] Destination: NBVY8432_GRCh38_dysgu 2023-08-17 13:51:05,788 [INFO ] dysgu fetch NBVY8432_GRCh38.bam written to NBVY8432_GRCh38_dysgu/NBVY8432_GRCh38.dysgu_reads.bam, n=23864581, time=0:05:08 h:m:s 2023-08-17 13:51:05,788 [INFO ] Input file is: NBVY8432_GRCh38_dysgu/NBVY8432_GRCh38.dysgu_reads.bam 2023-08-17 13:51:05,839 [WARNING] Warning: no @RG, using input file name as sample name for output: NBVY8432_GRCh38.dysgu_reads 2023-08-17 13:51:05,839 [INFO ] Sample name: NBVY8432_GRCh38.dysgu_reads 2023-08-17 13:51:05,839 [INFO ] Writing SVs to NBVY8432_GRCh38_dysgu.vcf 2023-08-17 13:51:05,839 [INFO ] Running pipeline 2023-08-17 13:51:06,128 [INFO ] Calculating insert size. Removed 31 outliers with insert size >= 887.0 2023-08-17 13:51:06,134 [INFO ] Inferred read length 150.0, insert median 399, insert stdev 90 2023-08-17 13:51:06,135 [INFO ] Max clustering dist 849 2023-08-17 13:51:06,135 [INFO ] Minimum support 3 2023-08-17 13:51:06,135 [INFO ] Building graph with clustering 849 bp 2023-08-17 13:52:59,811 [INFO ] Total input reads 23864581 2023-08-17 13:53:05,884 [INFO ] Graph constructed Process Process-7: Process Process-2: Process Process-5: Process Process-4: Process Process-1: Process Process-8: Process Process-3: Process Process-6: zsh: segmentation fault dysgu run --mode pe -o NBVY8432_GRCh38_dysgu.vcf -f vcf -p 8 --metrics --clea (base) husw5y@EA22-00453 Sequencing % Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() Traceback (most recent call last): Traceback (most recent call last): File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, *self._kwargs) File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() Traceback (most recent call last): File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(self._args, self._kwargs) File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() Traceback (most recent call last): File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "dysgu/cluster.pyx", line 653, in dysgu.cluster.process_job File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, self._kwargs) File "dysgu/cluster.pyx", line 653, in dysgu.cluster.process_job File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, *self._kwargs) File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(self._args, self._kwargs) File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 255, in recv buf = self._recv_bytes() File "dysgu/cluster.pyx", line 653, in dysgu.cluster.process_job File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, self._kwargs) File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 255, in recv buf = self._recv_bytes() File "dysgu/cluster.pyx", line 653, in dysgu.cluster.process_job File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "dysgu/cluster.pyx", line 653, in dysgu.cluster.process_job File "dysgu/cluster.pyx", line 653, in dysgu.cluster.process_job File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 419, in _recv_bytes buf = self._recv(4) File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 255, in recv buf = self._recv_bytes() File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 419, in _recv_bytes buf = self._recv(4) File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 255, in recv buf = self._recv_bytes() File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, *self._kwargs) File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 255, in recv buf = self._recv_bytes() File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 388, in _recv raise EOFError File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 419, in _recv_bytes buf = self._recv(4) File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 388, in _recv raise EOFError Traceback (most recent call last): File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 419, in _recv_bytes buf = self._recv(4) File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 419, in _recv_bytes buf = self._recv(4) File "dysgu/cluster.pyx", line 653, in dysgu.cluster.process_job EOFError File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 388, in _recv raise EOFError File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 388, in _recv raise EOFError File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 388, in _recv raise EOFError File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 255, in recv buf = self._recv_bytes() EOFError File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() EOFError EOFError File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(self._args, self._kwargs) File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 419, in _recv_bytes buf = self._recv(4) File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 255, in recv buf = self._recv_bytes() File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 388, in _recv raise EOFError File "dysgu/cluster.pyx", line 653, in dysgu.cluster.process_job File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 419, in _recv_bytes buf = self._recv(4) EOFError File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 255, in recv buf = self._recv_bytes() EOFError File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 388, in _recv raise EOFError File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 419, in _recv_bytes buf = self._recv(4) EOFError File "/Users/husw5y/miniconda3/lib/python3.10/multiprocessing/connection.py", line 388, in _recv raise EOFError EOFError** `'

[kcleal]

That looks like a different error. Could you upgrade to v1.6.0 and try again? Thanks!

[husamia]

@kcleal the problem is solved with v1.6.0