too many open files error

[KangchengHou]

because of [Errno 24] Too many open files: '/u/project/pasaniuc/pasaniucdata/admixture/projects/admix-prs-uncertainty/experiments/02_simulation/submitit-logs/506017_0_result.pkl.save_tmp' submitit ERROR (2021-11-07 16:37:29,729) - Could not dump error: b'Failed to open /u/project/pasaniuc/pasaniucdata/admixture/projects/admix-prs-uncertainty/data/PLINK/eur_val/merged.bed : Too many open files.\n'

because of [Errno 24] Too many open files: '/u/project/pasaniuc/pasaniucdata/admixture/projects/admix-prs-uncertainty/experiments/02_simulation/submitit-logs/506017_0_result.pkl.save_tmp' submitit ERROR (2021-11-07 16:37:29,729) - Submitted job triggered an exception submitit ERROR (2021-11-07 16:37:29,729) - Submitted job triggered an exception Traceback (most recent call last): File "/u/project/pasaniuc/kangchen/software/miniconda3/lib/python3.8/runpy.py", line 194, in _run_module_as_main File "/u/project/pasaniuc/kangchen/software/miniconda3/lib/python3.8/runpy.py", line 87, in _run_code File "/u/project/pasaniuc/kangchen/software/submitit-sge/submitit/core/_submit.py", line 11, in File "/u/project/pasaniuc/kangchen/software/submitit-sge/submitit/core/submission.py", line 72, in submitit_main File "/u/project/pasaniuc/kangchen/software/submitit-sge/submitit/core/submission.py", line 65, in process_job File "/u/project/pasaniuc/kangchen/software/submitit-sge/submitit/core/submission.py", line 54, in process_job File "/u/project/pasaniuc/kangchen/software/submitit-sge/submitit/core/utils.py", line 122, in result File "", line 11, in simulate_pheno_gwas File "/u/project/pasaniuc/pasaniucdata/admixture/projects/admix-prs-uncertainty/admix_prs/init.py", line 48, in simulate_quant_pheno File "/u/project/pasaniuc/kangchen/admix-tools/admix/simulate/_pheno.py", line 416, in quant_pheno_1pop File "/u/project/pasaniuc/kangchen/admix-tools/admix/data/_geno.py", line 132, in geno_mult_mat File "/u/project/pasaniuc/kangchen/software/miniconda3/lib/python3.8/site-packages/dask/base.py", line 286, in compute File "/u/project/pasaniuc/kangchen/software/miniconda3/lib/python3.8/site-packages/dask/base.py", line 568, in compute File "/u/project/pasaniuc/kangchen/software/miniconda3/lib/python3.8/site-packages/dask/threaded.py", line 79, in get File "/u/project/pasaniuc/kangchen/software/miniconda3/lib/python3.8/site-packages/dask/local.py", line 514, in get_async File "/u/project/pasaniuc/kangchen/software/miniconda3/lib/python3.8/site-packages/dask/local.py", line 325, in reraise File "/u/project/pasaniuc/kangchen/software/miniconda3/lib/python3.8/site-packages/dask/local.py", line 223, in execute_task File "/u/project/pasaniuc/kangchen/software/miniconda3/lib/python3.8/site-packages/dask/core.py", line 121, in _execute_task File "/u/project/pasaniuc/kangchen/software/miniconda3/lib/python3.8/site-packages/dask/utils.py", line 35, in apply File "/u/project/pasaniuc/kangchen/software/dask-pgen/dapgen/_read.py", line 162, in _read_pgen_chunk File "pgenlib.pyx", line 271, in pgenlib.PgenReader.cinit RuntimeError: b'Failed to open /u/project/pasaniuc/pasaniucdata/admixture/projects/admix-prs-uncertainty/data/PLINK/eur_val/merged.bed : Too many open files.\n'

[KangchengHou]

seems to also come with memory leak. figure out why

[KangchengHou]

simple way to debug is to open a small file with many chunks

[KangchengHou]

this is pgenlib bug, fixed in pgenlib