Open Tintest opened 6 years ago
Tintest
commented
6 years ago @jmstover smartly outlined in his post on the Singularity thread that having a tool asking for the /root directory should not be a normal behavior and might be hardcoded. That's why the command is successful when i'm using the container as root.
If you could change the /root directory with the /home/user one for the theano software, it would be great.
Thank you
samuelklee
commented
6 years ago Thanks for bringing this to our attention, @Tintest. I think that we may be able to address this by setting base_compiledir via os.environ["THEANO_FLAGS"] appropriately (see http://deeplearning.net/software/theano/library/config.html). @mbabadi @cmnbroad any thoughts?
In any case, thanks for trying out the GermlineCNVCaller pipeline. You may have to tune some parameters, depending on your data type. You may find the following discussions helpful:
https://github.com/broadinstitute/gatk/issues/4719
Note that we're still in beta, but our preliminary evaluations have demonstrated improved performance over other callers in both WES and WGS.
Tintest
commented
6 years ago Thank you for your links :)
I hoped you will be able to add the change in the next GATK version :)
If I can run this tool with singularity, I will be able to test a ton of different parameter !
cmnbroad
commented
6 years ago @samuelklee os.environ would probably work, though it might be easier to set theano.config.base_compiledir directly so you don't have to worry about clobbering any other flags.
shengqh
commented
6 years ago I downloaded the most recently gatk docker file yesterday and I met this problem today. I am wondering if you have solved this request? Thank you so much.
/gatk/gatk-package-4.0.11.0-local.jar singularity 2.5.0-dist
sooheelee
commented
6 years ago Hi @shengqh, did you activate the gatk python environment within your Singularity container with source activate gatk? GATK CNV tools require this environment and it appears that launching the Singularity container does not automatically activate this. Docker containers automatically activate this environment.
shengqh
commented
6 years ago @sooheelee
Here is my command:
singularity exec gatk.simg test.pbsHere is test.pbs:
gatk DetermineGermlineContigPloidy \
-L filtered.interval_list \
--input A1.count.hdf5 --input A2.count.hdf5 \
--contig-ploidy-priors contig_ploidy_priors_homo_sapiens.tsv \
--interval-merging-rule OVERLAPPING_ONLY \
--output out \
--output-prefix exomeseq \
--verbosity DEBUG \
--mean-bias-standard-deviation 0.01 \
--mapping-error-rate 0.01 \
--global-psi-scale 0.001 \
--sample-psi-scale 0.0001Here is the error message and I think the python environment has been activated by Singularity. The task failed when it tried to create directory of '/root/.theano'.
17:03:28.891 INFO DetermineGermlineContigPloidy - Initializing engine
17:03:28.896 DEBUG ScriptExecutor - Executing:
17:03:28.896 DEBUG ScriptExecutor - python
17:03:28.896 DEBUG ScriptExecutor - -c
17:03:28.896 DEBUG ScriptExecutor - import gcnvkernel
Traceback (most recent call last):
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/configdefaults.py", line 1738, in filter_compiledir
os.makedirs(path, 0o770) # read-write-execute for user and group
File "/opt/miniconda/envs/gatk/lib/python3.6/os.py", line 210, in makedirs
makedirs(head, mode, exist_ok)
File "/opt/miniconda/envs/gatk/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/root/.theano'You can tell if the python environment is active because your bash prompt will be prefaced with (gatk) as you see here from my testing:
shengqh
commented
6 years ago @sooheelee You are right, the python environment is not activated automatically. However, I manually activated the environment and then run my test.pbs, I still got the error.
[shengq2@cqs1 singularity]$ singularity shell gatk.simg
Singularity: Invoking an interactive shell within container...
Singularity gatk.simg:/scratch/cqs/softwares/singularity> source activate gatk
(gatk) Singularity gatk.simg:/scratch/cqs/softwares/singularity> sh test.pbs
...
21:27:03.205 INFO DetermineGermlineContigPloidy - Initializing engine
21:27:03.210 DEBUG ScriptExecutor - Executing:
21:27:03.210 DEBUG ScriptExecutor - python
21:27:03.210 DEBUG ScriptExecutor - -c
21:27:03.210 DEBUG ScriptExecutor - import gcnvkernel
Traceback (most recent call last):
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/configdefaults.py", line 1738, in filter_compiledir
os.makedirs(path, 0o770) # read-write-execute for user and group
File "/opt/miniconda/envs/gatk/lib/python3.6/os.py", line 210, in makedirs
makedirs(head, mode, exist_ok)
File "/opt/miniconda/envs/gatk/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/root/.theano'
jmstover
commented
6 years ago @shengqh
Looking at it, it seems that test.pbs is basically a shell script. So at the top of that add:
source activate gatkOr... whatever it is you need to do to activate the environment. What the exec command is going to do is launch whatever is passed in. So... something like:
singularity exec gatk.simg cat /etc/os-releaseWould show you the /etc/os-release file from within the container. exec isn't going to load a shell environment, but launch the application directly. In a shell look at: /.singularity.d/actions/exec
shengqh
commented
6 years ago @jmstover Thank you for your recommendation. I added the following line in the beginning of test.pbs
source activate gatkHowever, running the following command still threw the same error.
singularity exec gatk.simg ./test.pbsAre you building this from the docker broadinstitute/gatk image?
If so, I've found an error in it (@sooheelee, @samuelklee )... the Docker Manifest has set:
export HOME="/root"... That's going to make some things not work if you're using it as a non-root user. If you are using that, try building from a Singularity definition file that looks like:
Bootstrap: docker
From: broadinstitute/gatk
%environment
export HOME=/path/to/home
%runscript
bash -c "source activate gatk; /gatk/gatk \"$@\""I'm still testing this since I don't know the gatk software at all ... but with setting HOME in %environment you'll be overriding what the Docker manifest is setting. Setting %runscript you can then execute gatk by running:
singularity run gatk.simg DetermineGermlineContigPloidy \
-L filtered.interval_list \
--input A1.count.hdf5 --input A2.count.hdf5 \
--contig-ploidy-priors contig_ploidy_priors_homo_sapiens.tsv \
--interval-merging-rule OVERLAPPING_ONLY \
--output out \
--output-prefix exomeseq \
--verbosity DEBUG \
--mean-bias-standard-deviation 0.01 \
--mapping-error-rate 0.01 \
--global-psi-scale 0.001 \
--sample-psi-scale 0.0001Or if gatk.simg is executable, by just calling that image file if run-singularity is in your path. You just need to pass the options to gatk, not call gatk. So that would be:
./gatk.simg DetermineGermlineContigPloidy [...]@jmstover Yes, I pulled from broadinstiture/gatk.
singularity build gatk.simg docker://broadinstitute/gatkI don't have root permission to build Singularity image from definition file now. I may need to install a virtual machine first to test it. However, just like the activation of environment, I added the export command in my test.pbs and it worked. :-)
export HOME=/scratch/cqs/shengq2
source activate gatkGood to hear. :+1:
sooheelee
commented
6 years ago Yes, glad you found a workaround. 👍🏽
cmnbroad
commented
6 years ago Although there is a workaround, ideally we'd remove the assumption from our docker that it can access the root user's home dir.
samuelklee
commented
5 years ago @cmnbroad @droazen will there be any action on this? I'm not sure if #5714 might be related (although I'm not sure why we get a ulimit message there instead of a permission denied)?
droazen
commented
5 years ago @samuelklee I can have someone take another look at this, though it seems unlikely to me that it's the same problem reported in https://github.com/broadinstitute/gatk/issues/5714
samuelklee
commented
5 years ago Thanks for looking into it! I guess Cromwell also essentially sets HOME, which is probably why we don't typically run into this. Do we have any non-Cromwell Docker tests that could catch a regression?
drifty914
commented
5 years ago @samuelklee I forgot to note in #5714 that I had to manually set the path for HOME (to a mounted folder) when using the Docker image as it defaulted to ROOT and would complain otherwise.
samuelklee
commented
5 years ago OK, thanks @drifty914, that's very useful to know. Was that when using the Docker image with or without Cromwell---hopefully just the latter?
drifty914
commented
5 years ago @samuelklee that was when using the Docker image with Cromwell (v34).
Tintest
commented
5 years ago Hello,
I did encounter the same behaviour that I reported one year ago with the official Docker GATK 4.1.0.0 container converted into a singularity image.
Singularity : 2.5.1, GATK : 4.1.0.0
Singularity : singularity build gatk-4.0.4.0.img docker://broadinstitute/gatk:4.0.4.0
14:39:21.762 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gatk/gatk-package-4.1.0.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
14:39:24.079 INFO DetermineGermlineContigPloidy - ------------------------------------------------------------
14:39:24.079 INFO DetermineGermlineContigPloidy - The Genome Analysis Toolkit (GATK) v4.1.0.0
14:39:24.080 INFO DetermineGermlineContigPloidy - For support and documentation go to https://software.broadinstitute.org/gatk/
14:39:24.081 INFO DetermineGermlineContigPloidy - Executing as tintest@dahu38 on Linux v4.9.0-8-amd64 amd64
14:39:24.081 INFO DetermineGermlineContigPloidy - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_191-8u191-b12-0ubuntu0.16.04.1-b12
14:39:24.081 INFO DetermineGermlineContigPloidy - Start Date/Time: May 26, 2019 2:39:21 PM UTC
14:39:24.081 INFO DetermineGermlineContigPloidy - ------------------------------------------------------------
14:39:24.081 INFO DetermineGermlineContigPloidy - ------------------------------------------------------------
14:39:24.082 INFO DetermineGermlineContigPloidy - HTSJDK Version: 2.18.2
14:39:24.082 INFO DetermineGermlineContigPloidy - Picard Version: 2.18.25
14:39:24.083 INFO DetermineGermlineContigPloidy - HTSJDK Defaults.COMPRESSION_LEVEL : 2
14:39:24.083 INFO DetermineGermlineContigPloidy - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
14:39:24.083 INFO DetermineGermlineContigPloidy - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
14:39:24.083 INFO DetermineGermlineContigPloidy - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
14:39:24.083 INFO DetermineGermlineContigPloidy - Deflater: IntelDeflater
14:39:24.083 INFO DetermineGermlineContigPloidy - Inflater: IntelInflater
14:39:24.083 INFO DetermineGermlineContigPloidy - GCS max retries/reopens: 20
14:39:24.083 INFO DetermineGermlineContigPloidy - Requester pays: disabled
14:39:24.083 INFO DetermineGermlineContigPloidy - Initializing engine
14:39:26.111 INFO DetermineGermlineContigPloidy - Shutting down engine
[May 26, 2019 2:39:26 PM UTC] org.broadinstitute.hellbender.tools.copynumber.DetermineGermlineContigPloidy done. Elapsed time: 0.07 minutes.
Runtime.totalMemory()=1511522304
org.broadinstitute.hellbender.utils.python.PythonScriptExecutorException:
python exited with 1
Command Line: python -c import gcnvkernel
Stdout:
Stderr: Traceback (most recent call last):
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/configdefaults.py", line 1738, in filter_compiledir
os.makedirs(path, 0o770) # read-write-execute for user and group
File "/opt/miniconda/envs/gatk/lib/python3.6/os.py", line 210, in makedirs
makedirs(head, mode, exist_ok)
File "/opt/miniconda/envs/gatk/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/root/.theano'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/gcnvkernel/__init__.py", line 1, in <module>
from pymc3 import __version__ as pymc3_version
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/pymc3/__init__.py", line 5, in <module>
from .distributions import *
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/pymc3/distributions/__init__.py", line 1, in <module>
from . import timeseries
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/pymc3/distributions/timeseries.py", line 1, in <module>
import theano.tensor as tt
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/__init__.py", line 66, in <module>
from theano.compile import (
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/compile/__init__.py", line 10, in <module>
from theano.compile.function_module import *
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/compile/function_module.py", line 21, in <module>
import theano.compile.mode
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/compile/mode.py", line 10, in <module>
import theano.gof.vm
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/gof/vm.py", line 662, in <module>
from . import lazylinker_c
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/gof/lazylinker_c.py", line 42, in <module>
location = os.path.join(config.compiledir, 'lazylinker_ext')
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/configparser.py", line 333, in __get__
self.__set__(cls, val_str)
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/configparser.py", line 344, in __set__
self.val = self.filter(val)
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/configdefaults.py", line 1745, in filter_compiledir
" '%s'. Check the permissions." % path)
ValueError: Unable to create the compiledir directory '/root/.theano/compiledir_Linux-4.9--amd64-x86_64-with-debian-stretch-sid-x86_64-3.6.2-64'. Check the permissions.
at org.broadinstitute.hellbender.utils.python.PythonExecutorBase.getScriptException(PythonExecutorBase.java:75)
at org.broadinstitute.hellbender.utils.runtime.ScriptExecutor.executeCuratedArgs(ScriptExecutor.java:126)
at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeArgs(PythonScriptExecutor.java:170)
at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeCommand(PythonScriptExecutor.java:79)
at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.checkPythonEnvironmentForPackage(PythonScriptExecutor.java:192)
at org.broadinstitute.hellbender.tools.copynumber.DetermineGermlineContigPloidy.onStartup(DetermineGermlineContigPloidy.java:269)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:136)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205)
at org.broadinstitute.hellbender.Main.main(Main.java:291)
Using GATK jar /gatk/gatk-package-4.1.0.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /gatk/gatk-package-4.1.0.0-local.jar DetermineGermlineContigPloidy --input /bettik/tintest/PROJECTS/Test_nextflow_OAR/SAMPLE/test_grexome/cnv/tsv/0001/0001_grexome0505.tsv --input /bettik/tintest/PROJECTS/Test_nextflow_OAR/SAMPLE/test_grexome/cnv/tsv/0001/0001_grexome0507.tsv --input /bettik/tintest/PROJECTS/Test_nextflow_OAR/SAMPLE/test_grexome/cnv/tsv/0001/0001_grexome0498.tsv --model /bettik/tintest/PROJECTS/Test_nextflow_OAR/REF/hg38/CNV/DetermineGermlineContigPloidy/0001_DetermineGermlineContigPloidy --output /bettik/tintest/PROJECTS/Test_nextflow_OAR/SAMPLE/test_grexome/cnv/DetermineGermlineContigPloidy/0001_DetermineGermlineContigPloidy/ --output-prefix 0001_DetermineGermlineContigPloidyThank you.
shengqh
commented
5 years ago @Tintest I also met this problem. However, I solved it by "export HOME" before calling gatk command in my shell file call_gatk.sh
export HOME=/scratch/cqs/shengq2/temp/
source activate gatk
gatk --java-options "-Xmx40G" DetermineGermlineContigPloidy \
...And, here is my shell file test.sh to call singularity :
export R_LIBS=
export PYTHONPATH=
export JAVA_HOME=
singularity exec gatk.4.1.0.0.simg bash /scratch/cqs/shengq2/temp/call_gatk.shPutting the "export HOME" in test.sh with "export JAVA_HOME" before calling singularity didn't work. You have to put it in call_gatk.sh.
samuelklee
commented
3 years ago
Hello,
I'm trying to use the CNVGermline Pipeline with Nextflow and a GATK Singularity image pulled from your Docker image.
As you can see, I'm trying to run the DetermineGermlineContigPloidy with 57 samples, and I got a permission error within my Singularity container.
This error is directly related to Singularity permissions to create a directory ('/root/.theano/compiledir_Linux-4.10--generic-x86_64-with-debian-stretch-sid-x86_64-3.6.2-64'), because it's running fine with Singularity when I'm root and Docker (without root). So it's not really a GATK4 problem but more a singularity-GATK4 related problem.
Maybe your GATK4 Germline CNV calling pipeline is not designed to work as a Singularity container (which is quite understandable), because for tools like HaplotypeCaller, MarkDuplicates ... It's working fine.
I didn't know where to post this error, singularity or GATK github, so why not both !
Do you have any idea make it run properly ? Change a bit the design of your CNV pipeline to make it compatible with Singularity ?
Version of softwares:
Singularity : 2.5.1, GATK : 4.0.4.0
Command
Singularity : singularity build gatk-4.0.4.0.img docker://broadinstitute/gatk:4.0.4.0
GATK4 : java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /gatk/build/libs/gatk-package-4.0.4.0-local.jar DetermineGermlineContigPloidy --input 2044098202-8046_S5_sample.counts.hdf5 --input 2045946179-9076_S2_sample.counts.hdf5 --input 2045946166-9075_S1_sample.counts.hdf5 --input 2048220927-11022_S4_sample.counts.hdf5 --input 2045599261-9046ci_S1_sample.counts.hdf5 --input 2046745668-1007_S5_sample.counts.hdf5 --input 2044098101-8043_S2_sample.counts.hdf5 --input 2044098168-8044_S3_sample.counts.hdf5 --input 2046746598-1012_S4_sample.counts.hdf5 --input 2044395763-8064ci_S4_sample.counts.hdf5 --input 2044395647-8061ci_S1_sample.counts.hdf5 --input 70-20-CI_S3_sample.counts.hdf5 --input 70-18-CI_S1_sample.counts.hdf5 --input 2044395782-8065ci_S5_sample.counts.hdf5 --input 2048222393-11027_S3_sample.counts.hdf5 --input 2048220452-11020_S2_sample.counts.hdf5 --input 2048222667-11031_S7_sample.counts.hdf5 --input 2044395678-8062ci_S2_sample.counts.hdf5 --input 2044395611-8060_S6_sample.counts.hdf5 --input 2048221956-11025_S1_sample.counts.hdf5 --input 2045946131-9071ci_S6_sample.counts.hdf5 --input 2045599349-9047ci_S2_sample.counts.hdf5 --input 2045599158-9045ci_S6_sample.counts.hdf5 --input 2045946149-9073ci_S5_sample.counts.hdf5 --input 2048220431-11019_S1_sample.counts.hdf5 --input 2045946141-9072ci_S4_sample.counts.hdf5 --input 2046746758-1013_S5_sample.counts.hdf5 --input 2045598958-9043ci_S4_sample.counts.hdf5 --input 2046745451-1006_S4_sample.counts.hdf5 --input 2048220997-11023_S5_sample.counts.hdf5 --input 2044097928-8042_S1_sample.counts.hdf5 --input 2045598560-9041ci_S2_sample.counts.hdf5 --input 70-21-CI_S4_sample.counts.hdf5 --input 70-19-CI_S2_sample.counts.hdf5 --input 2046745923-1008_S6_sample.counts.hdf5 --input 2045598771-9042ci_S3_sample.counts.hdf5 --input 2048220861-11021_S3_sample.counts.hdf5 --input 204409765380-40Ci_S6_sample.counts.hdf5 --input 2048221977-11026_S2_sample.counts.hdf5 --input 2048222584-11029_S5_sample.counts.hdf5 --input 2046746353-1010_S2_sample.counts.hdf5 --input 2045946158-9074ci_S6_sample.counts.hdf5 --input 2045946119-9070ci_S5_sample.counts.hdf5 --input 2045599063-9044ci_S5_sample.counts.hdf5 --input 2046746482-1011_S3_sample.counts.hdf5 --input 204409784980-41ci_S7_sample.counts.hdf5 --input 2048222477-11028_S4_sample.counts.hdf5 --input 2045598355-9040ci_S1_sample.counts.hdf5 --input 2048221935-11024_S6_sample.counts.hdf5 --input 2044098183-8045_S4_sample.counts.hdf5 --input 2045946104-9069ci_S4_sample.counts.hdf5 --input 2048222618-11030_S6_sample.counts.hdf5 --input 2046747002-1014_S6_sample.counts.hdf5 --input 2044395737-8063ci_S3_sample.counts.hdf5 --input 2045946189-9077_S3_sample.counts.hdf5 --input 2045599415-9048ci_S3_sample.counts.hdf5 --input 2046746162-1009_S1_sample.counts.hdf5 --contig-ploidy-priors /home/qtestard/PROJECTS/aurexome/REF/common/prior_contig_diploidy.tsv --output /home/qtestard/PROJECTS/aurexome/SAMPLE/CNV_calling/CNV/CNV_calling_NaiveDetermineGermlineContigPloidy --output-prefix CNV_calling_NaiveDetermineGermlineContigPloidy
Actual behavior
Steps to reproduce behavior
Run the DetermineGermlineContigPloidy tool within a Singularity container without root permission.
Thank you.
My thread on Singularity Github #1557