Docker run unknown shorthand flag: 'r' in -rm

[pinjouwu9325]

Hi icbi-lab,

When I started to run the analysis of single samples with docker, I faced an error message of docker run and here is my command:

NeoFuse \
-1 /NAS_lab/pinjouwu9325/3utr/human/drHong/data/Control_100M_R1.clean.fastq \
-2 /NAS_lab/pinjouwu9325/3utr/human/drHong/data/Control_100M_R2.clean.fastq \
-s /NAS_lab/pinjouwu9325/neo/NeoFuse/arriba/STAR_index_GRCh38_ENSEMBL93/ \
-g /NAS_lab/pinjouwu9325/neo/NeoFuse/arriba/GRCh38.fa \
-a /NAS_lab/pinjouwu9325/neo/NeoFuse/arriba/ENSEMBL93.gtf \
-d control \
-m 8 \
-M 11 \
-n 8 \
-t 500 \
--docker

unknown shorthand flag: 'r' in -rm
See 'docker run --help'.

I am wondering that the shorthand flag error is because of the version of Docker. My current Docker version is 19.03.5 And I guess that in version 19.03.5 --rm should be pass if we want the container to removed automatically after running.

Also, I am wondering if this pipeline is only recommended to use on local PC, not on a server because of the permission issues. Thank you so much in advance for your help!

All the best, Pin-Jou

[abyssum]

Hello Pin-Jou,

Thank you for pointing this out, you are right about the flag. I just updated the script, please try to clone/ pull the latest changes and tell me if that works for you.

As for your second question, NeoFuse was developed and tested for local PC and HPC infrastructures. [edit]: Sorry I thought you were talking about the licenses, my bad. For server machines, we strongly recommend using Singularity to avoid any privilege escalation

Best, Georgios

[pinjouwu9325]

Hi Georgios,

Thank you for your quick reply. I ran the same command with the updated script, but it turned out permission denied when it tried to create directories. Here is the error message:

./NeoFuse -1 /NAS_lab/pinjouwu9325/3utr/human/drHong/data/Control_100M_R1.clean.fastq \
-2 /NAS_lab/pinjouwu9325/3utr/human/drHong/data/Control_100M_R2.clean.fastq \
-s /NAS_lab/pinjouwu9325/neo/NeoFuse/arriba/STAR_index_GRCh38_ENSEMBL93/ \
-g /NAS_lab/pinjouwu9325/neo/NeoFuse/arriba/GRCh38.fa \
-a /NAS_lab/pinjouwu9325/neo/NeoFuse/arriba/ENSEMBL93.gtf \
-d control -m 8 -M 11 -n 8 -t 500 --docker
mkdir: cannot create directory '/mnt/out/./control': Permission denied
mkdir: cannot create directory '/mnt/out/./control': Permission denied
mkdir: cannot create directory '/mnt/out/./control': Permission denied
mkdir: cannot create directory '/mnt/out/./control': Permission denied
mkdir: cannot create directory '/mnt/out/./control': Permission denied
mkdir: cannot create directory '/mnt/out/./control': Permission denied
mkdir: cannot create directory '/mnt/out/./control': Permission denied
mkdir: cannot create directory '/mnt/out/./control': Permission denied
mkdir: cannot create directory '/mnt/out/./control': Permission denied
mkdir: cannot create directory '/mnt/out/./control': Permission denied
mkdir: cannot create directory '/mnt/out/./control': Permission denied
[-------------------------------- [NeoFuse] --------------------------------]

[NeoFuse]  Paired End (PE) Reads detected: commencing processing
[NeoFuse]  Processing files Control_100M_R1.clean.fastq - Control_100M_R2.clean.fastq
[NeoFuse]  STAR Run started at: 12:23:54
/usr/local/bin/NeoFuse_single: line 169: /mnt/out/./control//LOGS/control.STAR.log: No such file or directory
[NeoFuse]  Arriba Run started at: 12:23:54
/usr/local/bin/NeoFuse_single: line 191: /mnt/out/./control//LOGS/control.arriba.log: No such file or directory

EXITING because of FATAL ERROR: could not create output file: /mnt/out/./control//Arriba/control.Log.out
Check if the path /mnt/out/./control//Arriba/control. exists and you have permissions to write there

Mar 23 12:23:54 ...... FATAL ERROR, exiting
An error occured during STAR/Arriba run, check the log files in /NAS_lab/pinjouwu9325/neo/NeoFuse/NeoFuse/./control/LOGS/ for more details

I think there might be something wrong with the mount path /mnt/out/./control/, so I tried to set -o test for $OUTDIR but it was the same:

 ./NeoFuse -1 /NAS_lab/pinjouwu9325/3utr/human/drHong/data/Control_100M_R1.clean.fastq \
-2 /NAS_lab/pinjouwu9325/3utr/human/drHong/data/Control_100M_R2.clean.fastq \
-s /NAS_lab/pinjouwu9325/neo/NeoFuse/arriba/STAR_index_GRCh38_ENSEMBL93/ \
-g /NAS_lab/pinjouwu9325/neo/NeoFuse/arriba/GRCh38.fa \
-a /NAS_lab/pinjouwu9325/neo/NeoFuse/arriba/ENSEMBL93.gtf \
-d control **-o test** -m 8 -M 11 -n 8 -t 500 --docker
mkdir: cannot create directory '/mnt/out/test/control': Permission denied
mkdir: cannot create directory '/mnt/out/test/control': Permission denied
mkdir: cannot create directory '/mnt/out/test/control': Permission denied
mkdir: cannot create directory '/mnt/out/test/control': Permission denied
mkdir: cannot create directory '/mnt/out/test/control': Permission denied
mkdir: cannot create directory '/mnt/out/test/control': Permission denied
mkdir: cannot create directory '/mnt/out/test/control': Permission denied
mkdir: cannot create directory '/mnt/out/test/control': Permission denied
mkdir: cannot create directory '/mnt/out/test/control': Permission denied
mkdir: cannot create directory '/mnt/out/test/control': Permission denied
mkdir: cannot create directory '/mnt/out/test/control': Permission denied
[-------------------------------- [NeoFuse] --------------------------------]

[NeoFuse]  Paired End (PE) Reads detected: commencing processing
[NeoFuse]  Processing files Control_100M_R1.clean.fastq - Control_100M_R2.clean.fastq
[NeoFuse]  STAR Run started at: 12:28:28
/usr/local/bin/NeoFuse_single: line 169: /mnt/out/test/control//LOGS/control.STAR.log: No such file or directory
[NeoFuse]  Arriba Run started at: 12:28:28
/usr/local/bin/NeoFuse_single: line 191: /mnt/out/test/control//LOGS/control.arriba.log: No such file or directory

EXITING because of FATAL ERROR: could not create output file: /mnt/out/test/control//Arriba/control.Log.out
Check if the path /mnt/out/test/control//Arriba/control. exists and you have permissions to write there

Mar 23 12:28:28 ...... FATAL ERROR, exiting
An error occured during STAR/Arriba run, check the log files in /NAS_lab/pinjouwu9325/neo/NeoFuse/NeoFuse/test/control/LOGS/ for more details

I am not sure if this is the user and the permission issue inside the container or I didn't set the right path for docker, though the double slash in the path /mnt/out/test/control// sound weird to me.

And yes I ran it on a server (Ubuntu 18.04.3 LTS), so maybe I should go for Singularity rather than Docker. Thanks again for your help.

Best, Pin-Jou

[riederd]

Hi, do you have write permissions in the directory you started NeoFuse? Can you try to start NeoFuse with the "-o" option and specify a directory in which you have write permissions.

[abyssum]

Hello again,

The first command (where you don't specify any output-dir) is a bug and I need to look into it (it should return $PWD, not "./").

The "//" shouldn't affect anything (but I will change that anyway)

But from your second run, I can tell that the problem is with docker not having permission to "write" in the specified output-dir, try re-running the same cmd but specify explicitly the output folder (where you would normally have permissions to write) - eg.:

-o /NAS_lab/pinjouwu9325/test_out

[pinjouwu9325]

Hi again,

I tried to set -o /NAS_lab/pinjouwu9325/test_out or other dir I have the write permission but still got the same error.

./NeoFuse -1 /NAS_lab/pinjouwu9325/3utr/human/drHong/data/Control_100M_R1.clean.fastq \
-2 /NAS_lab/pinjouwu9325/3utr/human/drHong/data/Control_100M_R2.clean.fastq \
-s /NAS_lab/pinjouwu9325/neo/NeoFuse/arriba/STAR_index_GRCh38_ENSEMBL93/ \
-g /NAS_lab/pinjouwu9325/neo/NeoFuse/arriba/GRCh38.fa \
-a /NAS_lab/pinjouwu9325/neo/NeoFuse/arriba/ENSEMBL93.gtf \
-d control \
-o /NAS_lab/pinjouwu9325/neo/NeoFuse/NeoFuse/test_out \
-m 8 -M 11 -n 8 -t 500 --docker
mkdir: cannot create directory '/mnt/out/test_out/control': Permission denied
mkdir: cannot create directory '/mnt/out/test_out/control': Permission denied
mkdir: cannot create directory '/mnt/out/test_out/control': Permission denied
mkdir: cannot create directory '/mnt/out/test_out/control': Permission denied
mkdir: cannot create directory '/mnt/out/test_out/control': Permission denied
mkdir: cannot create directory '/mnt/out/test_out/control': Permission denied
mkdir: cannot create directory '/mnt/out/test_out/control': Permission denied
mkdir: cannot create directory '/mnt/out/test_out/control': Permission denied
mkdir: cannot create directory '/mnt/out/test_out/control': Permission denied
mkdir: cannot create directory '/mnt/out/test_out/control': Permission denied
mkdir: cannot create directory '/mnt/out/test_out/control': Permission denied
[-------------------------------- [NeoFuse] --------------------------------]

[NeoFuse]  Paired End (PE) Reads detected: commencing processing
[NeoFuse]  Processing files Control_100M_R1.clean.fastq - Control_100M_R2.clean.fastq
[NeoFuse]  STAR Run started at: 03:03:10
/usr/local/bin/NeoFuse_single: line 169: /mnt/out/test_out/control//LOGS/control.STAR.log: No such file or directory
[NeoFuse]  Arriba Run started at: 03:03:10
/usr/local/bin/NeoFuse_single: line 191: /mnt/out/test_out/control//LOGS/control.arriba.log: No such file or directory

EXITING because of FATAL ERROR: could not create output file: /mnt/out/test_out/control//Arriba/control.Log.out
Check if the path /mnt/out/test_out/control//Arriba/control. exists and you have permissions to write there

Mar 24 03:03:10 ...... FATAL ERROR, exiting
An error occured during STAR/Arriba run, check the log files in /NAS_lab/pinjouwu9325/neo/NeoFuse/NeoFuse/test_out/control/LOGS/ for more details

I guess that it might be the permission issue inside the container, so I started an interactive shell docker run -it --entrypoint /bin/bash icbi/neofuse and tired to create an out directory under /mnt as user neofuse. However, I cannot create it either because of permission.

neofuse@f56e1a5b8993:/mnt$ whoami
neofuse
neofuse@f56e1a5b8993:/mnt$ mkdir out
mkdir: cannot create directory 'out': Permission denied
neofuse@f56e1a5b8993:/mnt$

I am not sure if this is the place that the error came from, so I went inside the container to see what happened. Thank you again for all your help! :)

Best, Pin-Jou

[abyssum]

Hello,

Yes, that is the case since the Docker daemon will always launch containers either as root or as the user specified in the Dockerfile ("neofuse" in this case). So, with Docker there is no way to create folders/files on a fs that only you (as the user) have permissions to write, without using sudo.

I would suggest to switch to Singularity (but note that in order to get proper support on the system, you will need to ask your sys admin to install Singularity to the system as root so it can leverage a set UID code path that does not require the user namespace) and let us know how it goes.

Best, Georgios

[pinjouwu9325]

Hi Georgios,

Yes, it makes sense that it showed a permission error. I will try to run it in Singularity. If anything happens, I'll let you know. Thanks again for all your help.

Best, Pin-Jou

[abyssum]

It seems the problem is resolved, so I am closing the issue.