pditommaso
commented
1 year ago Tagging @phue who is the master of Charliecloud support in Nextflow
Closed l-modolo closed 1 year ago
pditommaso
commented
1 year ago Tagging @phue who is the master of Charliecloud support in Nextflow
phue
commented
1 year ago The -w flag is required because of the way bind mounts work.
In contrast to singularity for example, there is no overlayfs support in charliecloud. That means the charliecloud runtime calls mkdir to make sure that the path for each nextflow task exists in within the container filesystem tree, which requires write permissions. (See https://github.com/hpc/charliecloud/issues/96)
I agree that this is far from optimal in the context of a shared cache directory, because each user will potentially bind mount many directories, but I'm afraid this cannot be solved in Nextflow. I would suggest opening an issue upstream, maybe the charliecloud developers have an idea how to approach this.
l-modolo
commented
1 year ago Hi @phue ,
Thank you for your answer. I agree with you that the overlayfs support is problematic in charliecloud.
The main problem for me is working with SquashFS images where you cannot mkdir within the image or plain format image where users can mess up the content (and the huge number of files associated with a plain format shared cache of images).
For the --bind option, it seems that charliecloud is fine with binding path to empty folder within the image (no mkdir is required).
In my case, we are required to work within a /scratch volume in my HPC, so if the charliecloud image has a empty /scratch folder everything works without the -w option (in SquashFS or plain format). This also work for binding to the /home/ folder or /tmp/ folder that are often empty in charliecloud images (a --bind /scratch:/home or --bind /scratch:/tmp would work).
But this also mean that if I want to bind the /scratch/Bio/lmodolo/nf-pipe/work I must use the option -b /scratch and not the full path. Otherwise charliecloud will try to mkdir the Bio/lmodolo/nf-pipe/work part of the path. Whereas with the -b /scratch will bind the content of /scratch and the content of the Bio folder and therefore give me access to the nextflow work directory within the image.
Do you think that providing the option to enable or disable the -w flag and binding only the first folder of the path to an empty directory in the image could make the process of using charliecloud with nextflow more flexible for HPC ?
phue
commented
1 year ago For the
--bindoption, it seems that charliecloud is fine with binding path to empty folder within the image (no mkdir is required). In my case, we are required to work within a/scratchvolume in my HPC, so if the charliecloud image has a empty/scratchfolder everything works without the-woption (in SquashFS or plain format). This also work for binding to the/home/folder or/tmp/folder that are often empty in charliecloud images (a--bind /scratch:/homeor--bind /scratch:/tmpwould work).
That's exactly the point, the empty folder needs to exist in the filesystem tree. This is somewhat guaranteed
for standard linux paths such as /home and /tmp, but not for non-standard paths like /scratch.
Of course, I you have control over container build-time this is a non-issue, but I would argue that a typical use case is pulling docker images from public registries that do not have such custom path.
In these cases, the image has to be writable.
Do you think that providing the option to enable or disable the -w flag and binding only the first folder of the path to an empty directory in the image could make the process of using charliecloud with nextflow more flexible for HPC ?
Yes this could be done with an additional config parameter in the charliecloud scope, in fact the implementations for singularity and docker have such a parameter already (readOnlyInputs).
But it should default to false for the reasons mentioned above
l-modolo
commented
1 year ago Hi @phue,
I finally had time to write the mentioned modification. In addition to the readOnlyInputs = true option, I can now work with SquashFS containers which will make the admin of my cluster happier.
However, during my investigation, I encountered another problem: nextflow try to make concurrent pull of images with charliecloud and throw the following error:
command: ch-image pull -s /Xnfs/abc/charliecloud_test ubuntu:20.04 > /dev/null
status : 1
message:
error: storage directory is already in use
hint: concurrent instances of ch-image cannot share the same storage directoryThis error is expected if different instances of nextflow (or different users) try to pull images in the same storage directory at the same time but not from the execution of one pipeline. I have seen some code with a mutex on line 180 of the CharliecloudCache.groovy file, and I was wondering is the mutex was there to prevent this kind of error (I am not fluent in groovy).
phue
commented
1 year ago This error is expected if different instances of nextflow (or different users) try to pull images in the same storage directory at the same time but not from the execution of one pipeline. I have seen some code with a
mutexon line 180 of theCharliecloudCache.groovyfile, and I was wondering is themutexwas there to prevent this kind of error (I am not fluent in groovy).
This is tricky, the mutex only handles that case for a single nextflow instance (i.e multiple processes trying to pull the same container at once).
For multiple instances/users, I don't think it can be solved in nextflow but rather the error is caused by charlieclouds built-in locking mechanism.
Edit: It's basically this: https://github.com/hpc/charliecloud/issues/1398
charliecloud v0.30 does have a --no-lock flag that could be added to the ch-image pull invocation in CharliecloudCache.groovy. But that seems risky as weird things could happen if multiple instances pull the same container.
l-modolo
commented
1 year ago Ok, the function is clearer now.
I understand that the multiple instances/users is difficult to handle and for me the error is expected in this case.
However, the error I have is when pulling multiple containers for a single process instance (for DSL2 nf-core pipeline with multiple containers for example):
[- ] process > NFCORE_RNASEQ:RNASEQ:MULTIQC -
Charliecloud pulling image quay.io/biocontainers/python:3.9--1 [cache /Xnfs/abc/charliecloud_test/img/quay.io%biocontainers%python+3.9--1]
Charliecloud pulling image ubuntu:20.04 [cache /Xnfs/abc/charliecloud_test/img/ubuntu+20.04]
Unexpected thread exception
Unexpected thread exception
Error executing process > 'NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:GUNZIP_GTF'
Caused by:
Charliecloud failed to pull image
command: ch-image pull -s /Xnfs/abc/charliecloud_test ubuntu:20.04 > /dev/null
status : 1
message:
error: storage directory is already in use
hint: concurrent instances of ch-image cannot share the same storage directoryIn this case nextflow tries to pull quay.io/biocontainers/python:3.9--1 and ubuntu:20.04 concurrently which results in an error for the ubuntu:20.04 pull.
phue
commented
1 year ago As far as I understand, this is because Charliecloud locks the entire (toplevel) storage directory when a container is pulled.
Could you try if the mutex works when changing the filename to something like ${localPath.parent.parent.parent}/.ch-pulling.lock?
l-modolo
commented
1 year ago Now I get the following error:
harliecloud pulling image quay.io/biocontainers/python:3.9--1 [cache /Xnfs/abc/charliecloud_test/img/quay.io%biocontainers%python+3.9--1]
Error executing process > 'NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:GUNZIP_GTF'
Caused by:
java.nio.channels.OverlappingFileLockExceptionI see.. looks like this would break it for a single nextflow instance My JVM knowledge is quite limited, but there is probably a way to make this synchronized somehow, so that there are not multiple threads trying to acquire the same lock. Would you have an idea @pditommaso ?
pditommaso
commented
1 year ago @l-modolo please include the full error stack trace
l-modolo
commented
1 year ago Hi @pditommaso,
pditommaso
commented
1 year ago I have to say I never seen this before. Don't know if it's a problem with the FileMutex implementation of the use made by Charliecloud.
stale[bot]
commented
1 year ago This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Bug report
Using nextflow with charliecloud.enabled=true with a shared
cacheDiron a HPCExpected behavior and actual behavior
By default the
ch-runcommand uses the result ofmakeVolumesas default bind (-b) option to bind the full path toward the pipeline working directory where only the root folder of this path is needed.Steps to reproduce the problem
On my HCP, the
.command.runrun something like:which returns
the following
ch-runcommand works correctly:Program output
(Copy and paste here output produced by the failing execution. Please highlight it as a code block. Whenever possible upload the
.nextflow.logfile.)Environment
Additional context
I would expect the
ch-runcommand to run without the-wtag enabled.-wstand for Mount image read-write (by default, the image is mounted read-only). This is problematic because some artifacts owned by the last user of the image can be created in the shared image cacheDir.