PIPELINE TEST w/ M. genitalium - PermissionError: [Errno 13] Permission denied: '/pgap/output/debug/tmpdir

[raysully]

Trying to run the pipeline test w/ M. genitalium and get: PermissionError: [Errno 13] Permission denied: '/pgap/output/debug/tmpdir

• OS: Ubuntu 20.04.6 LTS
• PGAP version 2023-10-03.build7061
• Docker version 24.0.7, build afdd53b

cwltool.log follows:

cwltool.log

Thanks for your support!

[azat-badretdin]

Thanks for your report, Ray!

Could you please post a listing of your mg37_results/ directory?

[raysully]

Thanks for yr quick response! I sent mg37_results/cwltool.log. The only other dir is mg37_results/debug/log, but it's completely empty.

[azat-badretdin]

Thanks, but all I need is a listing, as in ls -alR

[raysully]

Sorry. Here it is:

(base) gene@precision6:~/mg37_results$ ls -alR .: total 20 drwxrwxr-x 3 gene gene 4096 Dec 13 10:51 . drwxr-xr-x 81 gene gene 4096 Dec 13 10:51 .. -rw-rw-r-- 1 gene gene 3108 Dec 13 10:51 cwltool.log drwxrwxr-x 3 gene gene 4096 Dec 13 10:51 debug -rw-rw-r-- 1 gene gene 74 Dec 13 10:51 .~lock.cwltool.log#

./debug: total 12 drwxrwxr-x 3 gene gene 4096 Dec 13 10:51 . drwxrwxr-x 3 gene gene 4096 Dec 13 10:51 .. drwxrwxr-x 2 gene gene 4096 Dec 13 10:51 log

./debug/log: total 8 drwxrwxr-x 2 gene gene 4096 Dec 13 10:51 . drwxrwxr-x 3 gene gene 4096 Dec 13 10:51 .. (base) gene@precision6:~/mg37_results$

[azat-badretdin]

Thanks!

Looks like the permissions are fine. Could you please try to run this command and post the response?

/usr/local/bin/docker run -i --rm --user 1000:1000 \
   --volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \
   --volume /home/gene:/pgap/user_input:z \
   --volume /home/gene/pgap_input__o732xtc.yaml:/pgap/user_input/pgap_input.yaml:ro,z \
   --volume /home/gene/mytmp:/tmp:rw,z \
   --volume /home/gene/mg37_results:/pgap/output:rw,z \
   --volume /home/gene/mg37_results/debug/log:/log/srv:z \
   ncbi/pgap:2023-10-03.build7061 \
 mkdir /pgap/output/debug/tmpdir

[azat-badretdin]

My bad, command line fixed now.

[raysully]

(base) gene@precision6:~/mg37_results$ /usr/local/bin/docker run -i --rm --user 1000:1000 \

--volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \ --volume /home/gene:/pgap/user_input:z \ --volume /home/gene/pgap_inputo732xtc.yaml:/pgap/user_input/pgap_input.yaml:ro,z \ --volume /home/gene/mytmp:/tmp:rw,z \ --volume /home/gene/mg37_results:/pgap/output:rw,z \ --volume /home/gene/mg37_results/debug/log:/log/srv:z \ ncbi/pgap:2023-10-03.build7061 \ mkdir /pgap/output/debug/tmpdir docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/host_mnt/home/gene/pgap_inputo732xtc.yaml" to rootfs at "/pgap/user_input/pgap_input.yaml": mount /host_mnt/home/gene/pgap_input__o732xtc.yaml:/pgap/user_input/pgap_input.yaml (via /proc/self/fd/9), flags: 0x5001: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.

[azat-badretdin]

Thanks, how about this one:

/usr/local/bin/docker run -i --rm --user 1000:1000 \
   --volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \
   --volume /home/gene:/pgap/user_input:z \
   --volume /home/gene/mytmp:/tmp:rw,z \
   --volume /home/gene/mg37_results:/pgap/output:rw,z \
   --volume /home/gene/mg37_results/debug/log:/log/srv:z \
   ncbi/pgap:2023-10-03.build7061 \
 mkdir /pgap/output/debug/tmpdir

[raysully]

(base) gene@precision6:~/mg37_results$ /usr/local/bin/docker run -i --rm --user 1000:1000 \

--volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \ --volume /home/gene:/pgap/user_input:z \ --volume /home/gene/mytmp:/tmp:rw,z \ --volume /home/gene/mg37_results:/pgap/output:rw,z \ --volume /home/gene/mg37_results/debug/log:/log/srv:z \ ncbi/pgap:2023-10-03.build7061 \ mkdir /pgap/output/debug/tmpdir mkdir: cannot create directory '/pgap/output/debug/tmpdir': Permission denied

[azat-badretdin]

Thanks. Could you please run these commands?

/usr/local/bin/docker run -i --rm --user 1000:1000 \
   --volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \
   --volume /home/gene:/pgap/user_input:z \
   --volume /home/gene/mytmp:/tmp:rw,z \
   --volume /home/gene/mg37_results:/pgap/output:rw,z \
   --volume /home/gene/mg37_results/debug/log:/log/srv:z \
   ncbi/pgap:2023-10-03.build7061 \
 ls -lanR /pgap/output/debug/

this:

/usr/local/bin/docker run -i --rm --user 1000:1000 \
   --volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \
   --volume /home/gene:/pgap/user_input:z \
   --volume /home/gene/mytmp:/tmp:rw,z \
   --volume /home/gene/mg37_results:/pgap/output:rw,z \
   --volume /home/gene/mg37_results/debug/log:/log/srv:z \
   ncbi/pgap:2023-10-03.build7061 \
  id -u

and

/usr/local/bin/docker run -i --rm --user 1000:1000 \
   --volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \
   --volume /home/gene:/pgap/user_input:z \
   --volume /home/gene/mytmp:/tmp:rw,z \
   --volume /home/gene/mg37_results:/pgap/output:rw,z \
   --volume /home/gene/mg37_results/debug/log:/log/srv:z \
   ncbi/pgap:2023-10-03.build7061 \
  id -g

[raysully]

(base) gene@precision6:~$ /usr/local/bin/docker run -i --rm --user 1000:1000 \

--volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \ --volume /home/gene:/pgap/user_input:z \ --volume /home/gene/mytmp:/tmp:rw,z \ --volume /home/gene/mg37_results:/pgap/output:rw,z \ --volume /home/gene/mg37_results/debug/log:/log/srv:z \ ncbi/pgap:2023-10-03.build7061 \ ls -lanR /pgap/output/debug/ /pgap/output/debug/: total 12 drwxrwxr-x 3 0 0 4096 Dec 13 15:51 . drwxrwxr-x 3 0 0 4096 Dec 13 15:51 .. drwxrwxr-x 2 0 0 4096 Dec 13 15:51 log

/pgap/output/debug/log: total 8 drwxrwxr-x 2 0 0 4096 Dec 13 15:51 . drwxrwxr-x 3 0 0 4096 Dec 13 15:51 .. (base) gene@precision6:~$ /usr/local/bin/docker run -i --rm --user 1000:1000 \

--volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \ --volume /home/gene:/pgap/user_input:z \ --volume /home/gene/mytmp:/tmp:rw,z \ --volume /home/gene/mg37_results:/pgap/output:rw,z \ --volume /home/gene/mg37_results/debug/log:/log/srv:z \ ncbi/pgap:2023-10-03.build7061 \ id -u 1000 (base) gene@precision6:~$ /usr/local/bin/docker run -i --rm --user 1000:1000 \ --volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \ --volume /home/gene:/pgap/user_input:z \ --volume /home/gene/mytmp:/tmp:rw,z \ --volume /home/gene/mg37_results:/pgap/output:rw,z \ --volume /home/gene/mg37_results/debug/log:/log/srv:z \ ncbi/pgap:2023-10-03.build7061 \ id -g 1000 (base) gene@precision6:~$

[azat-badretdin]

Thanks, Ray. Now we are talking.

Now I see what the problem is. docker for some reason does not recognize --user 1000:1000 setting on your machine for user and group ids and uses straight root/root (0/0) settings.

This is peculiar. Normally --user settings are recognized. I am going to research this on the Internet and I suggest that maybe you can request help of your local system folks to clarify what is going on with your docker installation

[raysully]

Thanks for your help. Unfortunately, It's my own machine and I installed docker myself so you're my only lifeline for this and I'm a novice at this.

[azat-badretdin]

No problem. I opened an internal ticket, now that we identified the specific problem with your installation. The information that it is your personal home computer is useful.

[raysully]

Thanks Azat! I phoned a friend about this and he noticed the docker id was not showing up and had me restart. It's now showing up: (base) gene@precision6:~$ id uid=1000(gene) gid=1000(gene) groups=1000(gene),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),120(lpadmin),132(lxd),133(sambashare),997(docker),998(minknow)

However, when I reran the M. genitalium test it still errors out, bottom lines follow: "sqn": { "location": "file:///var/tmp/t7v5wzep/annot.sqn", "basename": "annot.sqn", "nameroot": "annot", "nameext": ".sqn", "class": "File", "checksum": "sha1$92a7bf56b7e6b66069d38c76b84197c7e97777e2", "size": 5411919, "http://commonwl.org/cwltool#generation": 0 } } [2023-12-13 23:31:40] DEBUG [job Generate_Annotation_Reports_gff_enhanced] Removing input staging directory /var/tmp/ijibnx6m [2023-12-13 23:31:40] DEBUG [job Generate_Annotation_Reports_gffenhanced] Removing temporary directory /var/tmp/pgx8bx5 [2023-12-13 23:31:40] DEBUG Moving /var/tmp/6ifwcjq2/calls.tab to /pgap/output/calls.tab [2023-12-13 23:31:40] ERROR Unhandled error: [Errno 13] Permission denied: '/pgap/output/calls.tab' Traceback (most recent call last): File "/opt/python-3.9/lib/python3.9/shutil.py", line 825, in move os.rename(src, real_dst) OSError: [Errno 18] Invalid cross-device link: '/var/tmp/6ifwcjq2/calls.tab' -> '/pgap/output/calls.tab'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/venv/lib/python3.9/site-packages/cwltool/main.py", line 1295, in main (out, status) = real_executor( File "/root/venv/lib/python3.9/site-packages/cwltool/executors.py", line 61, in call return self.execute(process, job_order_object, runtime_context, logger) File "/root/venv/lib/python3.9/site-packages/cwltool/executors.py", line 147, in execute self.final_output[0] = relocateOutputs( File "/root/venv/lib/python3.9/site-packages/cwltool/process.py", line 383, in relocateOutputs stage_files(pm, stage_func=_relocate, symlink=False, fix_conflicts=True) File "/root/venv/lib/python3.9/site-packages/cwltool/process.py", line 281, in stage_files stage_func(entry.resolved, entry.target) File "/root/venv/lib/python3.9/site-packages/cwltool/process.py", line 356, in _relocate shutil.move(src, dst) File "/opt/python-3.9/lib/python3.9/shutil.py", line 845, in move copy_function(src, real_dst) File "/opt/python-3.9/lib/python3.9/shutil.py", line 444, in copy2 copyfile(src, dst, follow_symlinks=follow_symlinks) File "/opt/python-3.9/lib/python3.9/shutil.py", line 266, in copyfile with open(dst, 'wb') as fdst: PermissionError: [Errno 13] Permission denied: '/pgap/output/calls.tab'

full .zip attached: cwltool20231213.zip

The --debug is the same as before: cwltool.log

[azat-badretdin]

I am glad that docker problem was resolved so easily, Ray!

Now to this beast:

OSError: [Errno 18] Invalid cross-device link: '/var/tmp/6ifwcjq2/calls.tab' -> '/pgap/output/calls.tab'

I would suggest to point TMPDIR envar to a directory in the same disk as your work directory.

For example:

mkdir -p tmpdir
export TMPDIR=$(readlink -f tmpdir)

and then try to run MG37 example again.

[raysully]

Azat, reran MG37 AND output is exactly the same as the my post 3 days ago.

[azat-badretdin]

Hmm. "Exactly" immediately makes me think of the same cause: Docker shenanigans. Could you please do the id test for Docker you did before you found necessity to restart it?

[azat-badretdin]

A workaround might be just running this thing as root to use your personal machine owner privilege.

[raysully]

It's trying to reinstall everything at /root/. I stopped it. I can't afford the space of another instance of pgap.

[azat-badretdin]

I can't afford the space of another instance of pgap.

You have a massive amount of memory, Ray, seems that this should come with analogously massive amount of space. But you have only

"work disk space (GiB)": 571.5

Anyway, I would suggest to ditch the first instance then. But this is actually Plan B. Plan A is still to fix your docker installation. Could you please test what I asked you earlier?

Could you please do the id test for Docker you did before you found necessity to restart it?

[raysully]

Thanks, Azat. Installng on root now and will test.

[azat-badretdin]

I see you are going with Plan B, which seems to be more straightforward.

[raysully]

Trying to figure out how to get docker running on root. Installation of pgap failed because docker was only running on gene (my user account). Do I have to install another instance of docker on root?

[azat-badretdin]

Do I have to install another instance of docker on root?

Probably. Typically software of this kinds is installed by "admin" so more than user can use it. Since this is your personal station, this rule seems to be less obvious, but it makes sense to install it as "root".

[raysully]

Tried to install docker at root, could not get it to install properly. The desktop came up, but could not get it to turn on at root. Tried sharing dir /root in docker, but it made no difference. Reinstalled docker as user (gene). Reran MG37 AND output is exactly the same as the my post 4 days ago.

(base) gene@precision6:~$ id uid=1000(gene) gid=1000(gene) groups=1000(gene),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),120(lpadmin),132(lxd),133(sambashare),997(docker),998(minknow)

[azat-badretdin]

Another option to try is make the top of your directory "rw" for everybody: chmod -R a+rw .

[raysully]

Still doesn't seem to be able to make dir pgap?

(base) gene@precision6:~$ sudo chmod -R a+rw /home [sudo] password for gene: (base) gene@precision6:~$ ./pgap.py -r -o mg37_results -g $HOME/.pgap/test_genomes/MG37/ASM2732v1.annotation.nucleotide.1.fasta -s 'Mycoplasmoides genitalium' --debug PGAP version 2023-10-03.build7061 is up to date. Output will be placed in: /home/gene/mg37_results PGAP failed, docker exited with rc = 1 Unable to find error in log file. (base) gene@precision6:~$

cwltool.log

[azat-badretdin]

Could you please post output of this commands again?

[raysully]

(base) gene@precision6:~$ /usr/local/bin/docker run -i --rm --user 1000:1000 \

--volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \ --volume /home/gene:/pgap/user_input:z \ --volume /home/gene/mytmp:/tmp:rw,z \ --volume /home/gene/mg37_results:/pgap/output:rw,z \ --volume /home/gene/mg37_results/debug/log:/log/srv:z \ ncbi/pgap:2023-10-03.build7061 \ ls -lanR /pgap/output/debug/ /pgap/output/debug/: total 12 drwxrwxr-x 3 0 0 4096 Dec 19 15:35 . drwxrwxr-x 3 0 0 4096 Dec 19 15:49 .. drwxrwxr-x 2 0 0 4096 Dec 19 15:35 log

/pgap/output/debug/log: total 8 drwxrwxr-x 2 0 0 4096 Dec 19 15:35 . drwxrwxr-x 3 0 0 4096 Dec 19 15:35 .. (base) gene@precision6:~$

(base) gene@precision6:~$ /usr/local/bin/docker run -i --rm --user 1000:1000 \

--volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \ --volume /home/gene:/pgap/user_input:z \ --volume /home/gene/mytmp:/tmp:rw,z \ --volume /home/gene/mg37_results:/pgap/output:rw,z \ --volume /home/gene/mg37_results/debug/log:/log/srv:z \ ncbi/pgap:2023-10-03.build7061 \ id -u 1000

(base) gene@precision6:~$ /usr/local/bin/docker run -i --rm --user 1000:1000 \

--volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \ --volume /home/gene:/pgap/user_input:z \ --volume /home/gene/mytmp:/tmp:rw,z \ --volume /home/gene/mg37_results:/pgap/output:rw,z \ --volume /home/gene/mg37_results/debug/log:/log/srv:z \ ncbi/pgap:2023-10-03.build7061 \ id -g 1000

[azat-badretdin]

Thanks.

Notice that for "all" the permissions are still read-only under docker. Let's double check if chmod -R ... you did worked.

could you please run ls -lanR mg37_results (that is, sans docker)?

[raysully]

(base) gene@precision6:~$ ls -lanR mg37_results mg37_results: total 16 drwxrwxr-x 3 1000 1000 4096 Dec 19 10:49 . drwxrwxrwx 88 1000 1000 4096 Dec 19 10:35 .. -rw-rw-r-- 1 1000 1000 3107 Dec 19 10:35 cwltool.log drwxrwxr-x 3 1000 1000 4096 Dec 19 10:35 debug

mg37_results/debug: total 12 drwxrwxr-x 3 1000 1000 4096 Dec 19 10:35 . drwxrwxr-x 3 1000 1000 4096 Dec 19 10:49 .. drwxrwxr-x 2 1000 1000 4096 Dec 19 10:35 log

mg37_results/debug/log: total 8 drwxrwxr-x 2 1000 1000 4096 Dec 19 10:35 . drwxrwxr-x 3 1000 1000 4096 Dec 19 10:35 .. (base) gene@precision6:~$

[azat-badretdin]

Thanks. I know what is happening. chmod -R a... worked, but we also need to set this for the future file creations. Could you please execute umask 000 and run PGAP again?

Thanks!

[raysully]

YOU'VE DONE IT! Thanks very much, Azat!

(base) gene@precision6:~$ ./pgap.py -r -o mg37_results -g $HOME/.pgap/test_genomes/MG37/ASM2732v1.annotation.nucleotide.1.fasta -s 'Mycoplasmoides genitalium' --debug PGAP version 2023-10-03.build7061 is up to date. Output will be placed in: /home/gene/mg37_results PGAP completed successfully. (base) gene@precision6:~$

[azat-badretdin]

You are welcome, Ray. But please have in mind that this was a hacky way, a correct way is to set up docker so it will allow you to recognize user id/group settings from a user outside docker container.

[raysully]

OK thanks. I'll have to read up on docker.

[raysully]

Thanks for your help Azat!

[azat-badretdin]

You are welcome, Ray!