Closed bgruening closed 2 months ago
Currently we use vda
as device for /tmp
which is the same as the root disk and has currently 50G,
however there is a vdb
device for /scratch
that has 1TB capacity.
Maybe it is safer to use that one?
How do you interpret the above error? I thought its not writable at all?
How do you interpret the above error? I thought its not writable at all?
I interpret it the same way. The docker is configured to use /scratch/docker
as its root, so all containers use this to store data, volumes, images, tmp, etc.
-- Edit ---
sorry, this is about singularity. I think if we want to use the /scratch
, then this needs to be bind mounted exclusively as tmp
in every singularity job. I don't know if there is a shortcut.
To avoid storage problems and arbitrary mount points, can we use the tmp
dir in the JWD, for example: $job_directory/tmp
in the extra args?
To avoid storage problems and arbitrary mount points, can we use the
tmp
dir in the JWD, for example:$job_directory/tmp
in the extra args?
+ :100: This would be best imo. For security reasons I do not think you want jobs to share /tmp
. It does not seem like Singularity provides any good alternatives. There exists --writable-tmpfs
, but it uses memory and provides too little space. Someone proposed --writable-scratch
to use disk, but it has not been implemented.
Edit: I see actually you are enabling --writable-tmpfs
via environment variable, moreover it is just for a single tool. I think it makes sense to have a look at this issue. It's ok anyway, no security concerns apply, and we are for sure better off with it enabled.
Edit: I see actually you are enabling --writable-tmpfs via environment variable, moreover it is just for a single tool. I think it makes sense to https://github.com/apptainer/singularity/issues/5718. It's ok anyway, no security concerns apply, and we are for sure better off with it enabled.
Yes. This is really just for tools that have /tmp
hardcoded or do other bad stuff. Galaxy should set by default TMPDIR and Co to the JWD already.
There exists --writable-tmpfs, but it uses memory and provides too little space.
We should just keep in mind that OOM killer can then kill the jobs if we not provision enough memory for the extra tmp.
Maybe I am blind here, but why do we not mount in the tmp
from jwd
as @sanjaysrikakulam suggested?
There exists --writable-tmpfs, but it uses memory and provides too little space.
We should just keep in mind that OOM killer can then kill the jobs if we not provision enough memory for the extra tmp.
Maybe I am blind here, but why do we not mount in the
tmp
fromjwd
as @sanjaysrikakulam suggested?
It's not either this or mounting tmp
from jwd
. Those are not mutually exclusive. You may check out apptainer/singularity#798. Running with --writable-tmpfs
means you get a throwaway overlay mounted on /
(only 64MB in size). Thus, if the program tries to write something anywhere (no matter if /tmp
or somewhere else), it will be able to do so (up to 64MB, more if that location is mounted somewhere else). I hope the following example clarifies it:
centos@vgcnbwc-worker-c36m100-0013:~$ singularity shell /cvmfs/singularity.galaxyproject.org/f/u/funannotate\:1.8.13--pyhdfd78af_0
Singularity> df -h
Filesystem Size Used Available Use% Mounted on
overlay 64.0M 12.0K 64.0M 0% /
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 48.9G 0 48.9G 0% /dev/shm
/dev/vda1 49.9G 8.9G 41.0G 18% /etc/localtime
/dev/vda1 49.9G 8.9G 41.0G 18% /etc/hosts
/dev/vda1 49.9G 8.9G 41.0G 18% /home/centos
/dev/vda1 49.9G 8.9G 41.0G 18% /tmp
/dev/vda1 49.9G 8.9G 41.0G 18% /var/tmp
tmpfs 64.0M 12.0K 64.0M 0% /etc/resolv.conf
tmpfs 64.0M 12.0K 64.0M 0% /etc/passwd
tmpfs 64.0M 12.0K 64.0M 0% /etc/group
Singularity> echo a > /test
bash: /test: Read-only file system
Singularity> exit
centos@vgcnbwc-worker-c36m100-0013:~$ singularity shell --writable-tmpfs /cvmfs/singularity.galaxyproject.org/f/u/funannotate\:1.8.13--pyhdfd78af_0
Singularity> df -h
Filesystem Size Used Available Use% Mounted on
fuse-overlayfs 64.0M 12.0K 64.0M 0% /
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 48.9G 0 48.9G 0% /dev/shm
/dev/vda1 49.9G 8.9G 41.0G 18% /etc/localtime
/dev/vda1 49.9G 8.9G 41.0G 18% /etc/hosts
/dev/vda1 49.9G 8.9G 41.0G 18% /home/centos
/dev/vda1 49.9G 8.9G 41.0G 18% /tmp
/dev/vda1 49.9G 8.9G 41.0G 18% /var/tmp
tmpfs 64.0M 12.0K 64.0M 0% /etc/resolv.conf
tmpfs 64.0M 12.0K 64.0M 0% /etc/passwd
tmpfs 64.0M 12.0K 64.0M 0% /etc/group
Singularity> echo a > /test
Singularity>
Björn is suggesting to enable this for toolshed.g2.bx.psu.edu/repos/iuc/funannotate_annotate/funannotate_annotate/.*
. I do not know if that solves the problem, but the cost of letting him try is close to zero.
Thank you, I think I understand this now better was confused, because I thought it is only about /tmp
64 MB should be fine memory wise, if the tmp files the tool creates are small enough, ofc
Thanks for merging. Can I redeploy this? We just got one bug report again with this tool and the NFS lock issue today :(
@sanjaysrikakulam @mira-miracoli anything against this?
I see some read-only /tmp in the logs: