Closed chessleensingh closed 1 year ago
FankangMeng
commented
1 year ago @chessleensingh I am having the same errors as you: ' touch: cannot touch '.command.trace': Permission denied' . It happened to almost all epi2melabs workflows on my Ubuntu 22.04.2. Do you have any updates on this? Thank you!
suu-yi
commented
1 year ago @chessleensingh @FankangMeng Hi, I am also having this exact issue... have there been any updates or did anyone resolve this? Thanks!
cjw85
commented
1 year ago Hi @FankangMeng @chessleensingh @suu-yi,
I apologise that no one has reached out. We've seen this sympton on several user systems now. It appears like a simple file system permissions error, but then the workflow is trying to write a file (.command.trace) into a directory that it has itself just created.
We test workflows on a variety of systems and environments but have never observed a symptom like this. We are currently at a bit of a loss as to the circumstances under which this error can arise.
One thing that would help is to sanity check the file system permissions. For example from the error:
Command error:
touch: cannot touch '.command.trace': Permission denied
Work dir:
/home/abiplantlab/epi2melabs/instances/wf-metagenomics_30807338-7952-49c2-a7ce-28712da563b8/work/7b/a2f1a9d35dd775f165e7359897617dcould you, acting as the same user which was used to run nextflow change into that directory and run ls -la to view owners and permissions of the contents. Can you create files there? Is there a .command.trace present already owned by another user?
suu-yi
commented
1 year ago @cjw85 Hi there, thank you for response!
Using ls -la, files in that directory have -rw-rw-r-- I can create files in that directory but there is no .command.trace file that I could find.
lhuang3s
commented
1 year ago @cjw85 Hi Chris, I was having the same issue. It is weird to me that the workflow could have permission to create files from .command.begin to .command.sh under the same directory while not having permission to create the .command.trace. What would be the difference between the commands that create all other .command files and the .command.trace file?
Besides, I found people mentioned about this issue in other nextflow workflows touch: cannot touch '.command.trace': Permission denied #142, and the solution was to add docker.runOptions = '-u $(id -u):$(id -g)' to the nextflow.config file. I don't know how to check or modify the nextflow.config file in a docker image.
Could you please let me know how to do that? Or was this setting already included in the nextflow.config file? Thank you!
cjw85
commented
1 year ago All our workflows already include this in the standard profile:
rchapman2000
commented
1 year ago @lhuang3s Not sure if you have tried this or not, but I was having the same issue and just managed to get it resolved.
There is a small section at the bottom of the Manage Docker as a non-root user section of the Linux Post-Installation Steps Guide from docker that mentions deleting your /.docker/ directory after running these steps. Doing that immediately solved my issue. Docker must store some configuration so if you have run docker using sudo, even once, it messes with the container from EPI2ME.
In all, I followed the steps in the Manage Docker as a non-root user guide (linked above), deleted my /.docker/ directory, and restarted my computer after. I also made sure to delete and reinstall my workflows from the EPI2ME labs interface as well as deleting the docker containers that the workflows installed (I found them in the Docker Desktop app).
FankangMeng
commented
1 year ago @cjw85 , @chessleensingh , @lhuang3s , @rchapman2000 , greetings to you all.
I have currently transitioned from using Docker to leveraging Singularity for running the workflow in epi2me-labs, which has effectively resolved my issues.
Considering that epi2me-labs uses Docker by default, you'll need to make some modifications once you download the workflow. Navigate to the workflow directory, locate the 'nextflow.config' file, and replace the 'profiles' section with the following code. This alteration will set Singularity as the default execution environment:
profiles {
// the "standard" profile is used implicitly by Nextflow
// if no other profile is given on the CLI
standard {
singularity {
enabled = true
autoMounts = true
}
}
// using Docker
docker {
docker {
enabled = true
// this ensures the container is run as host user and group, but
// also adds host user to the within-container group
runOptions = "--user \$(id -u):\$(id -g) --group-add 100"
}
}
// using Singularity
singularity {
singularity {
enabled = true
autoMounts = true
}
}
conda {
conda.enabled = true
}
// Using AWS batch.
// May need to set aws.region and aws.batch.cliPath
awsbatch {
process {
executor = 'awsbatch'
queue = "${params.aws_queue}"
memory = '8G'
withLabel:wfplasmid {
container = "${params.aws_image_prefix}-wf-clone-validation:${params.wf.container_sha}-root"
}
withLabel:medaka {
container = "${params.aws_image_prefix}-medaka:${params.wf.container_sha_medaka}-root"
}
shell = ['/bin/bash', '-euo', 'pipefail']
}
}
aws.region = 'eu-west-1'
aws.batch.cliPath = '/home/ec2-user/miniconda/bin/aws'
// local profile for simplified development testing
local {
process.executor = 'local'
}
}There's no need to edit the nextflow.config file to achieve this change. Simply run:
nextflow run -profile singularity ...More generically you do not need to edit the nextflow.config file in the workflow repository. Nextflow allows you to override configuration by providing additional files, e.g.:
nextflow run -c my_conf.config ...@rchapman2000 Thank you for your advice! However, since I cannot find the /.docker/ folder in the root directory, I guess using sudo might not be the root cause of this issue. Besides, I have also tried the Linux Post-Installation Steps Guide, and I have no error when running the docker run hello-world without sudo, but the permission issue still occurred when running this workflow.
rchapman2000
commented
1 year ago @lhuang3s Bad formatting on my part - the .docker/ directory is found in your home directory (of whatever user you installed Docker Desktop with).
I had the same issue - I could run docker run hello-world without sudo, but still got the permission issue in the workflow. Deleting the .docker/ was what fixed it.
Hope this helps!
pvdam3
commented
1 year ago I can confirm @rchapman2000 's fix, deleting the ~/.docker folder resolved the issue for me.
cjw85
commented
1 year ago Thats @pvdam3 @rchapman2000 @lhuang3s for the feedback that deleting ~/.docker gets thinks working for each of you. We will investigate further what is being written there that could be causing the issues.
Did you all install docker through Docker Desktop or more directly through a system package manager like sudo apt install docker.io?
suu-yi
commented
1 year ago At first we had Docker Desktop installed which brought up this issue, but when I installed the workflow on another computer, I had it working without an issue with Docker Engine (installed with sudo apt-get install). It might be Docker Desktop that's causing this?
Eventually I uninstalled the Docker Desktop, removed all old files as per Docker documentation, removed ~./docker, reinstalled Docker, but this time Engine version and it worked. And yes, it doesn't work if the ~./docker isn't removed before your new install.
cjw85
commented
1 year ago That's good information, thank you. It does seem like Docker Desktop doesn't do things quite correctly during its installation process. 🤔
lhuang3s
commented
1 year ago @cjw85 I was also using docker desktop. So I uninstalled it and re-install the docker engine with sudo apt-get install. I also uninstalled the old epi2me labs and install the latest version. The good news is that the workflow runs perfectly with the demo data. However, when I tested it with my own data, the workflow gives me the following error message in the log:
This is epi2me-labs/wf-metagenomics v2.2.1.
--------------------------------------------------------------------------------
Checking inputs.
Checking fastq input.
[7d/f7da21] Submitted process > fastcat (1)
[8e/f00a8b] Submitted process > fastcat (3)
[cd/841333] Submitted process > kraken_pipeline:getVersions
[a0/4fba64] Submitted process > fastcat (2)
[ed/ef0a35] Submitted process > kraken_pipeline:getParams
[f8/7e6cfd] Submitted process > fastcat (4)
ERROR ~ Error executing process > 'fastcat (2)'
Caused by:
Process `fastcat (2)` terminated with an error exit status (127)
Command executed:
mkdir fastcat_stats
fastcat -s barcode82 -r fastcat_stats/per-read-stats.tsv -f fastcat_stats/per-file-stats.tsv barcode82 | bgzip -@ 2 > seqs.fastq.gz
Command exit status:
127
Command output:
(empty)
Command error:
.command.sh: line 3: fastcat: command not found
.command.sh: line 3: bgzip: command not found
Work dir:
/home/quintara/epi2melabs/instances/wf-metagenomics_50ebc8a0-3152-414e-927b-4f744a22b8a6/work/a0/4fba6471a500d4121491cc30f3628a
Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`
-- Check '/home/quintara/epi2melabs/instances/wf-metagenomics_50ebc8a0-3152-414e-927b-4f744a22b8a6/nextflow.log' file for details
[86/d4fb2b] Submitted process > kraken_pipeline:output (1)
WARN: Killing running tasks (5)
WARN: Graphviz is required to render the execution DAG in the given format -- See http://www.graphviz.org for more info.We have had cases of this same issues reported for other workflows and have determined that Docker Desktop is the culprit. We are going to be changing our recommendation that on Linux docker is installed through a more traditional route from the system package manager.
Stramon1um
commented
1 year ago @lhuang3s Bad formatting on my part - the .docker/ directory is found in your home directory (of whatever user you installed Docker Desktop with).
I had the same issue - I could run
docker run hello-worldwithout sudo, but still got the permission issue in the workflow. Deleting the .docker/ was what fixed it.Hope this helps!
Hi, same situation here. I can run docker run hello-world with and without sudo + Docker Desktop installed but always problems trying to run the pipeline test. After reading this issue, I removed Docker Desktop, remove the .docker and install docker directly from apt and I was at least able to run the pipeline test but still only with sudo, until I decided to remove also the folder for temporary files 'work' (which i guess had the old privileges of the previous docker installation. After the first test, the folder is recreated and it works like a charm now.
TLDR: remove and purge everything related to the installation of docker desktop. Install docker engine from apt, follow the post-installation instructions for the non-root user, and try to run 'docker run ...' or a test pipeline without sudo...et voilà!
teegman123
commented
4 months ago @cjw85 Hello running into a similar issue as this chain (error 126).
I have a few quick questions about the program: Is the docker functionality within the wf-metagenomic workflow written in such a way that it will need to have root(admin) access rather than just user access?
Thank you for your assistance :)
cjw85
commented
4 months ago This depends how docker itself has been installed. Typically the docker daemon is run as root and users must be added to a user group (typically docker) in order to have priveleges to interact with the daemon. You should never need to run nextflow as the root user.
What happened?
A bug happened!
Operating System
ubuntu 20.04
Workflow Execution
EPI2ME Labs desktop application
Workflow Execution - EPI2ME Labs Versions
EPI2ME Labs V4.1.3
Workflow Execution - CLI Execution Profile
Docker
Workflow Version
v2.0.10
Relevant log output