PIPELINE TEST w/ M. genitalium - PermissionError: [Errno 13] Permission denied: '/pgap/output/debug/tmpdir

raysully commented 7 months ago

Trying to run the pipeline test w/ M. genitalium and get: PermissionError: [Errno 13] Permission denied: '/pgap/output/debug/tmpdir

OS: Ubuntu 20.04.6 LTS
PGAP version 2023-10-03.build7061
Docker version 24.0.7, build afdd53b

cwltool.log follows:

cwltool.log

Thanks for your support!

azat-badretdin commented 7 months ago

Thanks for your report, Ray!

Could you please post a listing of your mg37_results/ directory?

raysully commented 7 months ago

Thanks for yr quick response! I sent mg37_results/cwltool.log. The only other dir is mg37_results/debug/log, but it's completely empty.

azat-badretdin commented 7 months ago

Thanks, but all I need is a listing, as in ls -alR

raysully commented 7 months ago

Sorry. Here it is:

(base) gene@precision6:~/mg37_results$ ls -alR .: total 20 drwxrwxr-x 3 gene gene 4096 Dec 13 10:51 . drwxr-xr-x 81 gene gene 4096 Dec 13 10:51 .. -rw-rw-r-- 1 gene gene 3108 Dec 13 10:51 cwltool.log drwxrwxr-x 3 gene gene 4096 Dec 13 10:51 debug -rw-rw-r-- 1 gene gene 74 Dec 13 10:51 .~lock.cwltool.log#

./debug: total 12 drwxrwxr-x 3 gene gene 4096 Dec 13 10:51 . drwxrwxr-x 3 gene gene 4096 Dec 13 10:51 .. drwxrwxr-x 2 gene gene 4096 Dec 13 10:51 log

./debug/log: total 8 drwxrwxr-x 2 gene gene 4096 Dec 13 10:51 . drwxrwxr-x 3 gene gene 4096 Dec 13 10:51 .. (base) gene@precision6:~/mg37_results$

azat-badretdin commented 7 months ago

Thanks!

Looks like the permissions are fine. Could you please try to run this command and post the response?

/usr/local/bin/docker run -i --rm --user 1000:1000 \
   --volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \
   --volume /home/gene:/pgap/user_input:z \
   --volume /home/gene/pgap_input__o732xtc.yaml:/pgap/user_input/pgap_input.yaml:ro,z \
   --volume /home/gene/mytmp:/tmp:rw,z \
   --volume /home/gene/mg37_results:/pgap/output:rw,z \
   --volume /home/gene/mg37_results/debug/log:/log/srv:z \
   ncbi/pgap:2023-10-03.build7061 \
 mkdir /pgap/output/debug/tmpdir

azat-badretdin commented 7 months ago

My bad, command line fixed now.

raysully commented 7 months ago

(base) gene@precision6:~/mg37_results$ /usr/local/bin/docker run -i --rm --user 1000:1000 \

azat-badretdin commented 7 months ago

Thanks, how about this one:

/usr/local/bin/docker run -i --rm --user 1000:1000 \
   --volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \
   --volume /home/gene:/pgap/user_input:z \
   --volume /home/gene/mytmp:/tmp:rw,z \
   --volume /home/gene/mg37_results:/pgap/output:rw,z \
   --volume /home/gene/mg37_results/debug/log:/log/srv:z \
   ncbi/pgap:2023-10-03.build7061 \
 mkdir /pgap/output/debug/tmpdir

raysully commented 7 months ago

(base) gene@precision6:~/mg37_results$ /usr/local/bin/docker run -i --rm --user 1000:1000 \

azat-badretdin commented 7 months ago

Thanks. Could you please run these commands?

/usr/local/bin/docker run -i --rm --user 1000:1000 \
   --volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \
   --volume /home/gene:/pgap/user_input:z \
   --volume /home/gene/mytmp:/tmp:rw,z \
   --volume /home/gene/mg37_results:/pgap/output:rw,z \
   --volume /home/gene/mg37_results/debug/log:/log/srv:z \
   ncbi/pgap:2023-10-03.build7061 \
 ls -lanR /pgap/output/debug/

this:

/usr/local/bin/docker run -i --rm --user 1000:1000 \
   --volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \
   --volume /home/gene:/pgap/user_input:z \
   --volume /home/gene/mytmp:/tmp:rw,z \
   --volume /home/gene/mg37_results:/pgap/output:rw,z \
   --volume /home/gene/mg37_results/debug/log:/log/srv:z \
   ncbi/pgap:2023-10-03.build7061 \
  id -u

and

/usr/local/bin/docker run -i --rm --user 1000:1000 \
   --volume /home/gene/.pgap/input-2023-10-03.build7061:/pgap/input:ro,z \
   --volume /home/gene:/pgap/user_input:z \
   --volume /home/gene/mytmp:/tmp:rw,z \
   --volume /home/gene/mg37_results:/pgap/output:rw,z \
   --volume /home/gene/mg37_results/debug/log:/log/srv:z \
   ncbi/pgap:2023-10-03.build7061 \
  id -g

raysully commented 7 months ago

(base) gene@precision6:~$ /usr/local/bin/docker run -i --rm --user 1000:1000 \

/pgap/output/debug/log: total 8 drwxrwxr-x 2 0 0 4096 Dec 13 15:51 . drwxrwxr-x 3 0 0 4096 Dec 13 15:51 .. (base) gene@precision6:~$ /usr/local/bin/docker run -i --rm --user 1000:1000 \

azat-badretdin commented 7 months ago

Thanks, Ray. Now we are talking.

Now I see what the problem is. docker for some reason does not recognize --user 1000:1000 setting on your machine for user and group ids and uses straight root/root (0/0) settings.

This is peculiar. Normally --user settings are recognized. I am going to research this on the Internet and I suggest that maybe you can request help of your local system folks to clarify what is going on with your docker installation

raysully commented 7 months ago

Thanks for your help. Unfortunately, It's my own machine and I installed docker myself so you're my only lifeline for this and I'm a novice at this.

azat-badretdin commented 7 months ago

No problem. I opened an internal ticket, now that we identified the specific problem with your installation. The information that it is your personal home computer is useful.

raysully commented 7 months ago

Thanks Azat! I phoned a friend about this and he noticed the docker id was not showing up and had me restart. It's now showing up: (base) gene@precision6:~$ id uid=1000(gene) gid=1000(gene) groups=1000(gene),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),120(lpadmin),132(lxd),133(sambashare),997(docker),998(minknow)

However, when I reran the M. genitalium test it still errors out, bottom lines follow: "sqn": { "location": "file:///var/tmp/t7v5wzep/annot.sqn", "basename": "annot.sqn", "nameroot": "annot", "nameext": ".sqn", "class": "File", "checksum": "sha1$92a7bf56b7e6b66069d38c76b84197c7e97777e2", "size": 5411919, "http://commonwl.org/cwltool#generation": 0 } } [2023-12-13 23:31:40] DEBUG [job Generate_Annotation_Reports_gff_enhanced] Removing input staging directory /var/tmp/ijibnx6m [2023-12-13 23:31:40] DEBUG [job Generate_Annotation_Reports_gffenhanced] Removing temporary directory /var/tmp/pgx8bx5 [2023-12-13 23:31:40] DEBUG Moving /var/tmp/6ifwcjq2/calls.tab to /pgap/output/calls.tab [2023-12-13 23:31:40] ERROR Unhandled error: [Errno 13] Permission denied: '/pgap/output/calls.tab' Traceback (most recent call last): File "/opt/python-3.9/lib/python3.9/shutil.py", line 825, in move os.rename(src, real_dst) OSError: [Errno 18] Invalid cross-device link: '/var/tmp/6ifwcjq2/calls.tab' -> '/pgap/output/calls.tab'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/venv/lib/python3.9/site-packages/cwltool/main.py", line 1295, in main (out, status) = real_executor( File "/root/venv/lib/python3.9/site-packages/cwltool/executors.py", line 61, in call return self.execute(process, job_order_object, runtime_context, logger) File "/root/venv/lib/python3.9/site-packages/cwltool/executors.py", line 147, in execute self.final_output[0] = relocateOutputs( File "/root/venv/lib/python3.9/site-packages/cwltool/process.py", line 383, in relocateOutputs stage_files(pm, stage_func=_relocate, symlink=False, fix_conflicts=True) File "/root/venv/lib/python3.9/site-packages/cwltool/process.py", line 281, in stage_files stage_func(entry.resolved, entry.target) File "/root/venv/lib/python3.9/site-packages/cwltool/process.py", line 356, in _relocate shutil.move(src, dst) File "/opt/python-3.9/lib/python3.9/shutil.py", line 845, in move copy_function(src, real_dst) File "/opt/python-3.9/lib/python3.9/shutil.py", line 444, in copy2 copyfile(src, dst, follow_symlinks=follow_symlinks) File "/opt/python-3.9/lib/python3.9/shutil.py", line 266, in copyfile with open(dst, 'wb') as fdst: PermissionError: [Errno 13] Permission denied: '/pgap/output/calls.tab'

full .zip attached: cwltool20231213.zip

The --debug is the same as before: cwltool.log

azat-badretdin commented 7 months ago

I am glad that docker problem was resolved so easily, Ray!

Now to this beast:

I would suggest to point TMPDIR envar to a directory in the same disk as your work directory.

For example:


mkdir -p tmpdir
export TMPDIR=$(readlink -f tmpdir)

and then try to run MG37 example again.

raysully commented 7 months ago

Azat, reran MG37 AND output is exactly the same as the my post 3 days ago.

azat-badretdin commented 7 months ago

Hmm. "Exactly" immediately makes me think of the same cause: Docker shenanigans. Could you please do the id test for Docker you did before you found necessity to restart it?

azat-badretdin commented 7 months ago

A workaround might be just running this thing as root to use your personal machine owner privilege.

raysully commented 7 months ago

It's trying to reinstall everything at /root/. I stopped it. I can't afford the space of another instance of pgap.

azat-badretdin commented 7 months ago

You have a massive amount of memory, Ray, seems that this should come with analogously massive amount of space. But you have only

"work disk space (GiB)": 571.5

Anyway, I would suggest to ditch the first instance then. But this is actually Plan B. Plan A is still to fix your docker installation. Could you please test what I asked you earlier?

raysully commented 7 months ago

Thanks, Azat. Installng on root now and will test.

azat-badretdin commented 7 months ago

I see you are going with Plan B, which seems to be more straightforward.

raysully commented 7 months ago

Trying to figure out how to get docker running on root. Installation of pgap failed because docker was only running on gene (my user account). Do I have to install another instance of docker on root?

azat-badretdin commented 7 months ago

Probably. Typically software of this kinds is installed by "admin" so more than user can use it. Since this is your personal station, this rule seems to be less obvious, but it makes sense to install it as "root".

raysully commented 7 months ago

Tried to install docker at root, could not get it to install properly. The desktop came up, but could not get it to turn on at root. Tried sharing dir /root in docker, but it made no difference. Reinstalled docker as user (gene). Reran MG37 AND output is exactly the same as the my post 4 days ago.

(base) gene@precision6:~$ id uid=1000(gene) gid=1000(gene) groups=1000(gene),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),120(lpadmin),132(lxd),133(sambashare),997(docker),998(minknow)

azat-badretdin commented 7 months ago

Another option to try is make the top of your directory "rw" for everybody: chmod -R a+rw .

raysully commented 7 months ago

Still doesn't seem to be able to make dir pgap?

(base) gene@precision6:~$ sudo chmod -R a+rw /home [sudo] password for gene: (base) gene@precision6:~$ ./pgap.py -r -o mg37_results -g $HOME/.pgap/test_genomes/MG37/ASM2732v1.annotation.nucleotide.1.fasta -s 'Mycoplasmoides genitalium' --debug PGAP version 2023-10-03.build7061 is up to date. Output will be placed in: /home/gene/mg37_results PGAP failed, docker exited with rc = 1 Unable to find error in log file. (base) gene@precision6:~$

cwltool.log

azat-badretdin commented 7 months ago

Could you please post output of this commands again?

raysully commented 7 months ago

(base) gene@precision6:~$ /usr/local/bin/docker run -i --rm --user 1000:1000 \

/pgap/output/debug/log: total 8 drwxrwxr-x 2 0 0 4096 Dec 19 15:35 . drwxrwxr-x 3 0 0 4096 Dec 19 15:35 .. (base) gene@precision6:~$

(base) gene@precision6:~$ /usr/local/bin/docker run -i --rm --user 1000:1000 \

(base) gene@precision6:~$ /usr/local/bin/docker run -i --rm --user 1000:1000 \

azat-badretdin commented 7 months ago

Thanks.

Notice that for "all" the permissions are still read-only under docker. Let's double check if chmod -R ... you did worked.

could you please run ls -lanR mg37_results (that is, sans docker)?

raysully commented 7 months ago

(base) gene@precision6:~$ ls -lanR mg37_results mg37_results: total 16 drwxrwxr-x 3 1000 1000 4096 Dec 19 10:49 . drwxrwxrwx 88 1000 1000 4096 Dec 19 10:35 .. -rw-rw-r-- 1 1000 1000 3107 Dec 19 10:35 cwltool.log drwxrwxr-x 3 1000 1000 4096 Dec 19 10:35 debug

mg37_results/debug: total 12 drwxrwxr-x 3 1000 1000 4096 Dec 19 10:35 . drwxrwxr-x 3 1000 1000 4096 Dec 19 10:49 .. drwxrwxr-x 2 1000 1000 4096 Dec 19 10:35 log

mg37_results/debug/log: total 8 drwxrwxr-x 2 1000 1000 4096 Dec 19 10:35 . drwxrwxr-x 3 1000 1000 4096 Dec 19 10:35 .. (base) gene@precision6:~$

azat-badretdin commented 7 months ago

Thanks. I know what is happening. chmod -R a... worked, but we also need to set this for the future file creations. Could you please execute umask 000 and run PGAP again?

Thanks!

raysully commented 7 months ago

YOU'VE DONE IT! Thanks very much, Azat!

(base) gene@precision6:~$ ./pgap.py -r -o mg37_results -g $HOME/.pgap/test_genomes/MG37/ASM2732v1.annotation.nucleotide.1.fasta -s 'Mycoplasmoides genitalium' --debug PGAP version 2023-10-03.build7061 is up to date. Output will be placed in: /home/gene/mg37_results PGAP completed successfully. (base) gene@precision6:~$

azat-badretdin commented 7 months ago

You are welcome, Ray. But please have in mind that this was a hacky way, a correct way is to set up docker so it will allow you to recognize user id/group settings from a user outside docker container.

raysully commented 7 months ago

OK thanks. I'll have to read up on docker.

raysully commented 7 months ago

Thanks for your help Azat!

azat-badretdin commented 7 months ago

You are welcome, Ray!

ncbi / pgap

PIPELINE TEST w/ M. genitalium - PermissionError: [Errno 13] Permission denied: '/pgap/output/debug/tmpdir #283