search

LANL-Bioinformatics / EDGE

EDGE is a highly adaptable bioinformatics platform that allows laboratories to quickly analyze and interpret genomic sequence data.
https://lanl-bioinformatics.github.io/EDGE/
GNU General Public License v3.0
73 stars 31 forks source link

Runs are failing with permission errors #63

Closed Stopi closed 1 year ago

Stopi commented 1 year ago

Hi,

I have 2 containers with the last version of EDGE.

Useful info: OS: Windows 10 + WSL 2 EDGE Version: v2.4.0 build 20230412

In the 1st container, I didn't mount external paths (for EDGE_output, EDGE_input & EDGE_report). This container is working well and very fast, but I know all the data is in the WSL image (ext4.vhdx) which is growing too much.

In the 2nd container, I've used the regular mount points:

docker run -d --volumes-from mysql_data2  \
  -v $PWD/EDGE_output:/home/edge/EDGE_output \
  -v $PWD/EDGE_input:/home/edge/EDGE_input \
  -v $PWD/EDGE_report:/home/edge/EDGE_report \
  -v $PWD/human_ref_GRCh38_all.fa.gz:/home/edge/edge/database/human_ref_GRCh38_all.fa.gz \
  -p 80:80 -p 8080:8080 --name edge_ncov2 bioedge/edge-covid19

In this container, uploading data is very slow. The runs can start and are indeed producing data in the desired folders (EDGE_output) which I can see using windows explorer (in D:\ a partition on an SSD drive). But those runs are ver slow, and they end up failing. The error messages in the log are:

Cannot change permissions of /home/edge/edge_dev/edge_ui/EDGE_output//13dde84ee501c08d5c48b0d7df21374c/QcReads to 493: Operation not permitted at /home/edge/edge/runPipeline line 5291.
Cannot change permissions of /home/edge/EDGE_output/13dde84ee501c08d5c48b0d7df21374c/Reference/ to 493: Operation not permitted at /home/edge/edge/runPipeline line 5291.
Cannot change permissions of /home/edge/edge_dev/edge_ui/EDGE_output//13dde84ee501c08d5c48b0d7df21374c/ReferenceBasedAnalysis to 493: Operation not permitted at /home/edge/edge/runPipeline line 5291.
Cannot change permissions of /home/edge/edge_dev/edge_ui/EDGE_output//13dde84ee501c08d5c48b0d7df21374c/ReadsBasedAnalysis to 493: Operation not permitted at /home/edge/edge/runPipeline line 5291.
Cannot change permissions of /home/edge/edge_dev/edge_ui/EDGE_output//13dde84ee501c08d5c48b0d7df21374c/ReadsBasedAnalysis/readsMappingToRef to 493: Operation not permitted at /home/edge/edge/runPipeline line 5291.
Cannot change permissions of /home/edge/edge_dev/edge_ui/EDGE_output//13dde84ee501c08d5c48b0d7df21374c/ReadsBasedAnalysis/readsMappingToRef/consensus_tmp to 493: Operation not permitted at /home/edge/edge/runPipeline line 5291.
sed: preserving permissions for ‘./sedjymIWm’: Operation not permitted
sed: preserving permissions for ‘./sedIiz1fx’: Operation not permitted
the command . /home/edge/edge_dev/thirdParty/Anaconda3/bin/activate /home/edge/edge_dev/thirdParty/Anaconda3/envs/pangolin; pangolin --update 1>> /home/edge/edge_dev/edge_ui/EDGE_output//13dde84ee501c08d5c48b0d7df21374c/ReadsBasedAnalysis/readsMappingToRef/pangolin.log 2>&1 | true ;pangolin --analysis-mode usher -t 5 --tempdir /home/edge/edge_dev/edge_ui/EDGE_output//13dde84ee501c08d5c48b0d7df21374c/ReadsBasedAnalysis/readsMappingToRef --outdir /home/edge/edge_dev/edge_ui/EDGE_output//13dde84ee501c08d5c48b0d7df21374c/ReadsBasedAnalysis/readsMappingToRef --outfile NC_045512.2_consensus_lineage.txt /home/edge/edge_dev/edge_ui/EDGE_output//13dde84ee501c08d5c48b0d7df21374c/ReadsBasedAnalysis/readsMappingToRef/NC_045512.2_consensus.fasta 1>> /home/edge/edge_dev/edge_ui/EDGE_output//13dde84ee501c08d5c48b0d7df21374c/ReadsBasedAnalysis/readsMappingToRef/pangolin.log 2>&1 failed

I'm not sure why this fails. I'm not even sure why it's trying to change the permissions of the files & folder. Somehow EDGE generated those files & folder, the owner is my current windows user. I don't know why it cannot change the permissions of files it created itself.

When looking in WSL terminal, those files all have all rights (rwxrwxrwx) which is normal since they are on an NTFS partition. Does anyone have an idea of what's happening and how I can fix this?

chienchi commented 1 year ago

I think it is the permission issue on the WSL subsystem in Windows. Could you try to add a user and group edge with uid 2000 and gid 2000? It is the same user/group in the docker container.

sudo groupadd -g 2000 edge sudo user add -m -u 2000 -g 2000 edge

In addition, the exFAT formatted partition will not work since it doesn't support symlink.

And, please pull the latest docker image from docker hub, we fixed a pangolin package issue on build 20230412. The latest build is 20230719.

Stopi commented 1 year ago

Hi, thanks for your quick answer.

I created the edge user/group in WSL but it didn't change anything. Also please note the output partition mounted to docker is not in exFAT, it's in NTFS (which supports links).

Anyway, with the new image (20230719) the run didn't fail, the overall result is green. But the permission errors are still present and a lot of red is showing in the report.

According to the logs, the only errors where about permissions. To be honest, I don't see how creating a user in WSL is impacting the container, it should be the other way around.

So I've thought about that and tried a change inside the container. We can add the edge user (2000) to the windows group (1000) which has the same GID as the one used by WSL.

sh-4.4# ls -lah /home/edge/EDGE_output/ |head
total 8.0K
drwxrwxrwx 1 windows windows 4.0K Jul 28 04:22 .
drwxr-xr-x 1 edge    edge    4.0K Jul 19 18:10 ..
drwxrwxrwx 1 windows windows 4.0K Jul 21 06:57 0006d65bc26fa16cbe4a0fc4fea752a6
drwxrwxrwx 1 windows windows 4.0K Jul 21 06:57 002ccb896cb8534722f26469bc69e47b
drwxrwxrwx 1 windows windows 4.0K Jul 21 06:58 003818fdda8e333e0a904e783abfe7cc
drwxrwxrwx 1 windows windows 4.0K Jul 21 06:58 0073adb1265f5298b8a3881173f099f4
drwxrwxrwx 1 windows windows 4.0K Jul 21 06:57 00ae4d918a23b938e7db2e0a45210fd9
drwxrwxrwx 1 windows windows 4.0K Jul 21 06:58 027645acb2fce4d7f545812c9f014ca0
drwxrwxrwx 1 windows windows 4.0K Jul 21 06:58 03097cf66fd7061b30b55d3fdfb099bb
sh-4.4# gpasswd -a edge windows
Adding user edge to group windows

But it doesn't change anything, the issue is still here.

So, let me show you a bit of what's wrong. In the error_log I still have:

Cannot change permissions of /home/edge/edge_dev/edge_ui/EDGE_output//7d037f75b5867321b009ad99a4ccc0f4/QcReads to 493: Operation not permitted at /home/edge/edge/runPipeline line 5291.
Cannot change permissions of /home/edge/EDGE_output/7d037f75b5867321b009ad99a4ccc0f4/Reference/ to 493: Operation not permitted at /home/edge/edge/runPipeline line 5291.
Cannot change permissions of /home/edge/edge_dev/edge_ui/EDGE_output//7d037f75b5867321b009ad99a4ccc0f4/ReferenceBasedAnalysis to 493: Operation not permitted at /home/edge/edge/runPipeline line 5291.
Cannot change permissions of /home/edge/edge_dev/edge_ui/EDGE_output//7d037f75b5867321b009ad99a4ccc0f4/ReadsBasedAnalysis to 493: Operation not permitted at /home/edge/edge/runPipeline line 5291.
Cannot change permissions of /home/edge/edge_dev/edge_ui/EDGE_output//7d037f75b5867321b009ad99a4ccc0f4/ReadsBasedAnalysis/readsMappingToRef to 493: Operation not permitted at /home/edge/edge/runPipeline line 5291.
Cannot change permissions of /home/edge/edge_dev/edge_ui/EDGE_output//7d037f75b5867321b009ad99a4ccc0f4/ReadsBasedAnalysis/readsMappingToRef/consensus_tmp to 493: Operation not permitted at /home/edge/edge/runPipeline line 5291.
sed: preserving permissions for ‘./sedYW0kvp’: Operation not permitted
sed: preserving permissions for ‘./sedhUCU9v’: Operation not permitted
Cannot change permissions of /home/edge/EDGE_output/7d037f75b5867321b009ad99a4ccc0f4/JBrowse to 493: Operation not permitted at /home/edge/edge/runPipeline line 5291.
Cannot change permissions of /home/edge/EDGE_output/7d037f75b5867321b009ad99a4ccc0f4/IGV to 493: Operation not permitted at /home/edge/edge/runPipeline line 5291.
Cannot change permissions of /home/edge/edge_dev/edge_ui/EDGE_output//7d037f75b5867321b009ad99a4ccc0f4/ReadsBasedAnalysis/readsMappingToRef//MixedInfectionAnalysis to 493: Operation not permitted at /home/edge/edge/runPipeline line 5291.
Cannot change permissions of /home/edge/EDGE_output/7d037f75b5867321b009ad99a4ccc0f4/HTML_Report/images to 493: Operation not permitted at /home/edge/edge/runPipeline line 5291.

And the web interface shows this: shot1 shot2 shot3

So the results seem to be here but I'm not sure if they are correct.

chienchi commented 1 year ago

Thanks for all the screenshots and detailed attempts you have done. They are helpful for checking the issue. I saw the permission on EDGE output directory are all full (777). I have checked the similar WSL permission issue here.

Short answer:

Turn on WSL metadata support for Windows drives by:

sudo -e /etc/wsl.conf

and adding:

[automount]
options = "metadata"

And you may need to restart the terminal and restart the docker container.

Stopi commented 1 year ago

Thank you very much for your help. Your answer fixed the permission issues, now everything is back to green.

For your information, despite all the red display and the related permission error logs, I was suspecting the results produced were correct anyway. So I run some tests, both locally and with the online version of EDGE. The results were exactly the same which means the permissions error were not impacting anything in the end, except coloring the interface in red.

You're doing a great job with this tool and your support, be sure we all appreciate.

Thanks again!