Hi,
this is maybe a bit round-about and not something I've found elsewhere in issues. The case is that our cluster runs BeeGFS, which makes using --workDir on that filesystem out of the question (it doesn't support hard links between folders). The /tmp folder is too small on most nodes (about 15-20 GB, just a few with up to 300 GB). I can run CAT/Cactus on the few nodes with large /tmp, but I want to be more flexible.
Each node do have some local storage, which I can ask for from SLURM and get access to through a variable. So far so good. However, when I try doing this and setting up a workDir on that local scratch system I get these errors:
`Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/cat-2.0-py3.8.egg/tools/fileOps.py", line 73, in ensure_dir
os.makedirs(d)
File "/usr/lib/python3.8/os.py", line 213, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/usr/lib/python3.8/os.py", line 213, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/usr/lib/python3.8/os.py", line 223, in makedirs
mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/localscratch'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/luigi/worker.py", line 191, in run
new_deps = self._run_get_new_deps()
File "/usr/local/lib/python3.8/dist-packages/luigi/worker.py", line 133, in _run_get_new_deps
task_gen = self.task.run()
File "/usr/local/lib/python3.8/dist-packages/cat-2.0-py3.8.egg/cat/init.py", line 1741, in run
toil_options = self.prepare_toil_options(toil_work_dir)
File "/usr/local/lib/python3.8/dist-packages/cat-2.0-py3.8.egg/cat/init.py", line 573, in prepare_toil_options
tools.fileOps.ensure_dir(toil_args.workDir)
File "/usr/local/lib/python3.8/dist-packages/cat-2.0-py3.8.egg/tools/fileOps.py", line 80, in ensure_dir
raise RuntimeError("Unable to create directory {}".format(d))`
(I run a singularity container created from the development branch singularity pull --name cat.develop.sif docker://quay.io/ucsc_cgl/cat:develop).
I don't really understand this code. os.makedirs tries to create all parent folders, but stalls because one of them is read-only? Could you maybe set exist_ok to True? (https://docs.python.org/3/library/os.html#os.makedirs) Alternatively, you could throw an error if the folder doesn't exist and get the user to create it. I guess this is not the wanted/expected behaviour of this code.
Hi, this is maybe a bit round-about and not something I've found elsewhere in issues. The case is that our cluster runs BeeGFS, which makes using --workDir on that filesystem out of the question (it doesn't support hard links between folders). The /tmp folder is too small on most nodes (about 15-20 GB, just a few with up to 300 GB). I can run CAT/Cactus on the few nodes with large /tmp, but I want to be more flexible.
Each node do have some local storage, which I can ask for from SLURM and get access to through a variable. So far so good. However, when I try doing this and setting up a workDir on that local scratch system I get these errors:
`Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/cat-2.0-py3.8.egg/tools/fileOps.py", line 73, in ensure_dir os.makedirs(d) File "/usr/lib/python3.8/os.py", line 213, in makedirs makedirs(head, exist_ok=exist_ok) File "/usr/lib/python3.8/os.py", line 213, in makedirs makedirs(head, exist_ok=exist_ok) File "/usr/lib/python3.8/os.py", line 223, in makedirs mkdir(name, mode) OSError: [Errno 30] Read-only file system: '/localscratch'
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/luigi/worker.py", line 191, in run new_deps = self._run_get_new_deps() File "/usr/local/lib/python3.8/dist-packages/luigi/worker.py", line 133, in _run_get_new_deps task_gen = self.task.run() File "/usr/local/lib/python3.8/dist-packages/cat-2.0-py3.8.egg/cat/init.py", line 1741, in run toil_options = self.prepare_toil_options(toil_work_dir) File "/usr/local/lib/python3.8/dist-packages/cat-2.0-py3.8.egg/cat/init.py", line 573, in prepare_toil_options tools.fileOps.ensure_dir(toil_args.workDir) File "/usr/local/lib/python3.8/dist-packages/cat-2.0-py3.8.egg/tools/fileOps.py", line 80, in ensure_dir raise RuntimeError("Unable to create directory {}".format(d))`
(I run a singularity container created from the development branch
singularity pull --name cat.develop.sif docker://quay.io/ucsc_cgl/cat:develop).I don't really understand this code. os.makedirs tries to create all parent folders, but stalls because one of them is read-only? Could you maybe set exist_ok to True? (https://docs.python.org/3/library/os.html#os.makedirs) Alternatively, you could throw an error if the folder doesn't exist and get the user to create it. I guess this is not the wanted/expected behaviour of this code.
Thank you.
Sincerely, Ole