nextflow-io / nextflow

A DSL for data-driven computational pipelines
http://nextflow.io
Apache License 2.0
2.71k stars 621 forks source link

GCloud: Mounted directories not writable #5118

Closed VasLem closed 2 months ago

VasLem commented 2 months ago

Bug report

Hi, I have a simple scenario, which does not seem to work. I am supplying a directory in gs to my process, and I am trying to write to it. Say, that directory is called gs://my-bucket/my-directory. A simple touch inside the process is enough to raise a Permission denied error. If this is a feature, to prevent writing directly to the mount, I would expect to see it somewhere in the documentation. Similarly, I would also expect for an option to exist to permit this behavior. The publishDir way works alright, but for my use case it is far more intuitive just to write directly to the mount. (It is also something I need to have, as I am losing all progress on the files I have gotten up to the task failure point, as nothing is moved to the mount at the end, if an exception occurs.)

I recall that I had tried this in the past, like two months ago, and it worked like a charm.

Expected behavior and actual behavior

Expected: being able to write to a gs mounted directory Actual: Permission denied

Steps to reproduce the problem

Example code:

process example {
input:
 path inputdir
shell:
'''
 touch !{inputdir}/test
'''
}

workflow {
 example(file("gs://my-bucket/my-directory"))
}

Just to make sure, I made a different process with the sleep command within and entered the instance:

gcloud compute ssh --zone $MY_ZONE $INSTANCE --project $MY_PROJECT --command="dockername=\$(docker ps --format \"{{.Names}}\") && docker container exec -it \$dockername bash"

and then cd-ed to /mnt/disks/my-bucket/any-directory

and ran touch test. I got touch: failed to close 'test': Permission denied

Environment