Open jung233 opened 1 month ago
It is possible that you ran out of file descriptors (open files
)?
# Soft limit
ulimit -aS
# Hard limit
ulimit -aH
[user@login03 ~]$ ulimit -aS
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 512871
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
[user@login03 ~]$ ulimit -aH
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 512871
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 512871
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Thanks for your replying. But it seems impossible to run out. How can I further check it?
open files (-n) 1024
is not a lot.
Dask will open a lot of files. In the past we ran into problems with only 1024 allowed open files in combination with dask and had the cluster people change the hard limit to 16384
.
You can try to see if 4k open filehandle is enough (your current hard limit). If you need more, you will have to ask your cluster adminstrators to change the hard limit:
ulimit -S -n 4096
I tried ulimit -S -n 4096
and ulimit -S -n 131072
. It doesn't seem to be working.
#!/bin/bash
#SBATCH --job-name=pyscenic_1
#SBATCH --nodes=1
#SBATCH -c 16
#SBATCH --partition=node
source /share/home/user/.bash_profile
mamba activate pyscenic2
echo "Set open files: ulimit -S -n 131072"
ulimit -S -n 131072
echo "CHECK: ulimit -aS"
# Soft limit
ulimit -aS
echo "CHECK: ulimit -aH"
# Hard limit
ulimit -aH
echo "Pyscenic starting"
pyscenic grn --num_workers 16 \
--sparse \
--method grnboost2 \
--output /share/home/user/pyscenic/sce.adj.csv \
/share/home/user/pyscenic/expression_matrix.loom \
/share/home/user/pyscenic/hs_hgnc_tfs.txt
Set open files: ulimit -S -n 131072
CHECK: ulimit -aS
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 766890
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) 5120000
open files (-n) 131072
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
CHECK: ulimit -aH
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 766890
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) 5120000
open files (-n) 131072
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 766890
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Pyscenic starting
2024-07-19 18:43:56,086 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.
2024-07-19 18:45:00,128 - pyscenic.cli.pyscenic - INFO - Inferring regulatory networks.
2024-07-19 18:45:16,009 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 247.68 MiB -- Worker memory limit: 312.50 MiB
2024-07-19 18:45:16,010 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 229.58 MiB -- Worker memory limit: 312.50 MiB
2024-07-19 18:45:16,013 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 236.26 MiB -- Worker memory limit: 312.50 MiB
2024-07-19 18:45:16,115 - distributed.worker.memory - WARNING - Worker is at 80% memory usage. Pausing worker. Process memory: 251.70 MiB -- Worker memory limit: 312.50 MiB
2024-07-19 18:45:16,115 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 251.70 MiB -- Worker memory limit: 312.50 MiB
2024-07-19 18:45:16,117 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 245.62 MiB -- Worker memory limit: 312.50 MiB
2024-07-19 18:45:16,170 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 239.96 MiB -- Worker memory limit: 312.50 MiB
2024-07-19 18:45:16,173 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 234.43 MiB -- Worker memory limit: 312.50 MiB
2024-07-19 18:45:16,209 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 239.17 MiB -- Worker memory limit: 312.50 MiB
2024-07-19 18:45:16,247 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 236.60 MiB -- Worker memory limit: 312.50 MiB
2024-07-19 18:45:16,247 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 233.76 MiB -- Worker memory limit: 312.50 MiB
2024-07-19 18:45:16,298 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 237.86 MiB -- Worker memory limit: 312.50 MiB
2024-07-19 18:45:16,319 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 234.42 MiB -- Worker memory limit: 312.50 MiB
2024-07-19 18:45:16,324 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 232.02 MiB -- Worker memory limit: 312.50 MiB
2024-07-19 18:45:16,325 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 238.15 MiB -- Worker memory limit: 312.50 MiB
2024-07-19 18:45:16,325 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 235.00 MiB -- Worker memory limit: 312.50 MiB
2024-07-19 18:45:16,330 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 237.53 MiB -- Worker memory limit: 312.50 MiB
2024-07-19 18:45:17,655 - distributed.nanny.memory - WARNING - Worker tcp://127.0.0.1:33408 (pid=78441) exceeded 95% memory budget. Restarting...
2024-07-19 18:45:17,782 - distributed.nanny - WARNING - Restarting worker
preparing dask client
parsing input
creating dask graph
not shutting down client, client was created externally
finished
Traceback (most recent call last):
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read
frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/share/home/user/miniconda3/envs/pyscenic2/bin/pyscenic", line 8, in <module>
sys.exit(main())
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/pyscenic/cli/pyscenic.py", line 713, in main
args.func(args)
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/pyscenic/cli/pyscenic.py", line 97, in find_adjacencies_command
network = method(
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/arboreto/algo.py", line 39, in grnboost2
return diy(expression_data=expression_data, regressor_type='GBM', regressor_kwargs=SGBM_KWARGS,
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/arboreto/algo.py", line 120, in diy
graph = create_graph(expression_matrix,
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/arboreto/core.py", line 419, in create_graph
future_tf_matrix = client.scatter(tf_matrix, broadcast=True)
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/distributed/client.py", line 2668, in scatter
return self.sync(
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/distributed/utils.py", line 364, in sync
return sync(
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/distributed/utils.py", line 440, in sync
raise error
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/distributed/utils.py", line 414, in f
result = yield future
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/tornado/gen.py", line 766, in run
value = future.result()
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/distributed/client.py", line 2538, in _scatter
await self.scheduler.scatter(
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/distributed/core.py", line 1398, in send_recv_from_rpc
return await send_recv(comm=comm, op=key, **kwargs)
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/distributed/core.py", line 1157, in send_recv
response = await comm.read(deserializers=deserializers)
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/distributed/comm/tcp.py", line 236, in read
convert_stream_closed_error(self, e)
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/distributed/comm/tcp.py", line 142, in convert_stream_closed_error
raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) ConnectionPool.scatter local=tcp://127.0.0.1:55942 remote=tcp://127.0.0.1:33143>: Stream is closed
Something interesting happened when I tried running it with 8 cores ( the above was 16 cores). It threw a different error, I don't know if that's helpful.
2024-07-19 19:01:22,669 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.
2024-07-19 19:02:26,356 - pyscenic.cli.pyscenic - INFO - Inferring regulatory networks.
2024-07-19 19:02:40,816 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 481.24 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,131 - distributed.worker.memory - WARNING - Worker is at 82% memory usage. Pausing worker. Process memory: 515.96 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,132 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 515.96 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,136 - distributed.worker.memory - WARNING - Worker is at 82% memory usage. Pausing worker. Process memory: 516.37 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,136 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 516.37 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,151 - distributed.worker.memory - WARNING - Worker is at 68% memory usage. Resuming worker. Process memory: 425.24 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,158 - distributed.worker.memory - WARNING - Worker is at 67% memory usage. Resuming worker. Process memory: 424.90 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,481 - distributed.worker.memory - WARNING - Worker is at 84% memory usage. Pausing worker. Process memory: 529.61 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,482 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 529.61 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,484 - distributed.worker.memory - WARNING - Worker is at 81% memory usage. Pausing worker. Process memory: 506.55 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,484 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 506.55 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,484 - distributed.worker.memory - WARNING - Worker is at 83% memory usage. Pausing worker. Process memory: 518.85 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,484 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 518.85 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,490 - distributed.worker.memory - WARNING - Worker is at 82% memory usage. Pausing worker. Process memory: 515.44 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,490 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 515.44 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,494 - distributed.worker.memory - WARNING - Worker is at 82% memory usage. Pausing worker. Process memory: 517.31 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,494 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 517.31 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,499 - distributed.worker.memory - WARNING - Worker is at 68% memory usage. Resuming worker. Process memory: 427.87 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,529 - distributed.worker.memory - WARNING - Worker is at 68% memory usage. Resuming worker. Process memory: 426.35 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,547 - distributed.worker.memory - WARNING - Worker is at 67% memory usage. Resuming worker. Process memory: 424.48 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,562 - distributed.worker.memory - WARNING - Worker is at 70% memory usage. Resuming worker. Process memory: 438.62 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,687 - distributed.worker.memory - WARNING - Worker is at 82% memory usage. Pausing worker. Process memory: 516.21 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,699 - distributed.worker.memory - WARNING - Worker is at 82% memory usage. Pausing worker. Process memory: 515.88 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,749 - distributed.worker.memory - WARNING - Worker is at 68% memory usage. Resuming worker. Process memory: 425.11 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,758 - distributed.worker.memory - WARNING - Worker is at 67% memory usage. Resuming worker. Process memory: 424.77 MiB -- Worker memory limit: 625.00 MiB
2024-07-19 19:02:41,762 - distributed.worker.memory - WARNING - Worker is at 37% memory usage. Resuming worker. Process memory: 233.81 MiB -- Worker memory limit: 625.00 MiB
preparing dask client
parsing input
creating dask graph
not shutting down client, client was created externally
finished
Traceback (most recent call last):
File "/share/home/user/miniconda3/envs/pyscenic2/bin/pyscenic", line 8, in <module>
sys.exit(main())
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/pyscenic/cli/pyscenic.py", line 713, in main
args.func(args)
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/pyscenic/cli/pyscenic.py", line 97, in find_adjacencies_command
network = method(
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/arboreto/algo.py", line 39, in grnboost2
return diy(expression_data=expression_data, regressor_type='GBM', regressor_kwargs=SGBM_KWARGS,
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/arboreto/algo.py", line 120, in diy
graph = create_graph(expression_matrix,
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/arboreto/core.py", line 450, in create_graph
all_meta_df = from_delayed(delayed_meta_dfs, meta=_META_SCHEMA)
File "/share/home/user/miniconda3/envs/pyscenic2/lib/python3.10/site-packages/dask_expr/io/_delayed.py", line 115, in from_delayed
raise TypeError("Must supply at least one delayed object")
TypeError: Must supply at least one delayed object
ERR output:
Environment
Slurm Script:
Node:
Sample
About 60000 cells with 50000 genes
Thank you very much.