cannot open URL 'https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh'

[YingYa]

Hi I'am running Sigfolw with singularity, but fail to run extraction with the following errors:

=================================================================

       .__        _____.__
  _____|__| _____/ ____\  |   ______  _  __
 /  ___/  |/ ___\   __\|  |  /  _ \ \/ \/ /
 \___ \|  / /_/  >  |  |  |_(  <_> )     /
/____  >__\___  /|__|  |____/\____/ \/\_/
     \/  /_____/

Name   :       sigflow
Author :       Shixiang Wang
Version:       2.0
License:       AFL 3.0
Link   :       https://github.com/ShixiangWang/sigflow
Doc    :       https://github.com/ShixiangWang/sigflow
============================== START

Parsing parameters...
------
extract         : TRUE
fit             : FALSE
bt              : FALSE
show            : FALSE
help            : FALSE
version         : FALSE
input           : /path/to/exome.merge.maf.gz
output          : /path/to/Sigflow/exome/extract_SigProfiler/DBS_20210519_m50r300
index           : ALL
mode            : DBS
manual          : FALSE
number          : 0
max             : 50
genome          : hg38
nrun            : 300
cores           : 16
refit           : FALSE
hyper           : FALSE
sigprofiler     : TRUE
isearch         : NULL
verbose         : FALSE
------

User command calling: /usr/lib/R/bin/exec/R --no-echo --no-restore --file=/opt/sigflow.R --args extract -i /path/to/exome.merge.maf.gz -o /path/to/Sigflow/exome/extract_SigProfiler/DBS_20210519_m50r300 -m DBS -g hg38 --max 50 -r 300 -T 16 --sigprofiler
------

Reading file...
------
------

Running signature extraction pipeline...
------
trying URL 'https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh'
An error is detected in extract process. Quit the program!
cannot open URL 'https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh'
Warning messages:
1: In file(con, "r") :
  URL 'https://bioconductor.org/config.yaml': status was 'Couldn't resolve host name'
2: In file(con, "r") :
  URL 'http://bioconductor.org/config.yaml': status was 'Couldn't resolve host name'
3: In download.file(url, destfile = installer, mode = "wb") :
  URL 'https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh': Timeout of 60 seconds was reached

[ShixiangWang]

@YingYa Thanks for your report. Are you using sigflow with docker or by installing it by yourself? If it is the latter, you can install miniconda firstly and re-run again.

https://docs.conda.io/en/latest/miniconda.html

[ShixiangWang]

If you are familar with R and not use sigflow docker, directly using the function https://shixiangwang.github.io/sigminer/reference/sigprofiler.html is more recommended.

[YingYa]

@YingYa Thanks for your report. Are you using sigflow with docker or by installing it by yourself? If it is the latter, you can install miniconda firstly and re-run again.

https://docs.conda.io/en/latest/miniconda.html

I installed the Sigflow by: singularity build sigflow.v2.0.simg docker://shixiangwang/sigflow:2.0

Our working cluster nodes are offline, does sigflow need to installed something by conda?

[ShixiangWang]

@YingYa It seems that your singularity build is somewhat failed at some steps. I checked the tests (test file) for v2.0, it works well.

Please retry building the image by yourself.

If it is still now working, this issue may be caused by the root user: I used root directly in dockerfile, but the singularity seeems not work for root. Please check your build logs.

You can also try run command directly instead of build to test:

singularity run docker://shixiangwang/gistic extract -i /opt/test/tcga_laml.maf.gz -o /opt/test/test_results/test_docker -m SBS -r 2 --max 10 -T 4 --sigprofiler

[YingYa]

I perform "/share/app/singularity/3.2.0/bin/singularity run docker://shixiangwang/sigflow extract -i /opt/test/tcga_laml.maf.gz -o ./test_docker3 -m SBS -r 2 --max 10 -T 2 --sigprofiler" but fail for Miniconda3 installed. My home is to small to installed the Miniconda3.

Is taht possible to include the Miniconda3 in the image? What is the conda used for in the pipeline? Could the conda be removed?

=================================================================

       .__        _____.__
  _____|__| _____/ ____\  |   ______  _  __
 /  ___/  |/ ___\   __\|  |  /  _ \ \/ \/ /
 \___ \|  / /_/  >  |  |  |_(  <_> )     /
/____  >__\___  /|__|  |____/\____/ \/\_/
     \/  /_____/

Name   :       sigflow
Author :       Shixiang Wang
Version:       2.0
License:       AFL 3.0
Link   :       https://github.com/ShixiangWang/sigflow
Doc    :       https://github.com/ShixiangWang/sigflow
============================== START

Parsing parameters...
------
extract         : TRUE
fit             : FALSE
bt              : FALSE
show            : FALSE
help            : FALSE
version         : FALSE
input           : /opt/test/tcga_laml.maf.gz
output          : ./test_docker3
index           : ALL
mode            : SBS
manual          : FALSE
number          : 0
max             : 10
genome          : hg19
nrun            : 2
cores           : 2
refit           : FALSE
hyper           : FALSE
sigprofiler     : TRUE
isearch         : NULL
verbose         : FALSE
------

User command calling: /usr/lib/R/bin/exec/R --no-echo --no-restore --file=/usr/bin/sigflow --args extract -i /opt/test/tcga_laml.maf.gz -o ./test_docker3 -m SBS -r 2 --max 10 -T 2 --sigprofiler
------

Reading file...
------
Try reading as a MAF file.
-Validating
-Silent variants: 475
-Summarizing
-Processing clinical data
--Missing clinical data
-Finished in 0.519s elapsed (0.495s cpu)
------

Running signature extraction pipeline...
------
trying URL 'https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh'
Content type 'application/x-sh' length 94235922 bytes (89.9 MB)
==================================================
downloaded 89.9 MB

PREFIX=/home/yingya/.local/share/r-miniconda
Unpacking payload ...
Traceback (most recent call last):
  File "entry_point.py", line 76, in <module>
  File "tarfile.py", line 2026, in extractall
  File "tarfile.py", line 2067, in extract
  File "tarfile.py", line 2139, in _extract_member
  File "tarfile.py", line 2188, in makefile
OSError: [Errno 122] Disk quota exceeded
[9998] Failed to execute script entry_point
concurrent.futures.process._RemoteTraceback:
'''
Traceback (most recent call last):
  File "concurrent/futures/process.py", line 368, in _queue_management_worker
  File "multiprocessing/connection.py", line 251, in recv
TypeError: __init__() missing 1 required positional argument: 'msg'
'''

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "entry_point.py", line 69, in <module>
  File "concurrent/futures/process.py", line 484, in _chain_from_iterable_of_lists
  File "concurrent/futures/_base.py", line 611, in result_iterator
  File "concurrent/futures/_base.py", line 439, in result
  File "concurrent/futures/_base.py", line 388, in __get_result
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
[10011] Failed to execute script entry_point
An error is detected in extract process. Quit the program!
miniconda installation failed [exit code 1]

[ShixiangWang]

@YingYa The miniconda is used to install a specified environment for installing SigProfiler https://github.com/AlexandrovLab/SigProfilerExtractor. It must be installed if you use the option --sigprofiler. You can also omit this option to extract signatures.

If you do need sigprofiler, please specify a location (big enough) to install the whole sigminer docker image including the conda environment for sigprofiler. I dont know much about singularity, so I can't tell you how to do this. I remember the docker image can be converted to a sif file based on singualrity, may be you can take a try?

[YingYa]

@ShixiangWang

To install the conda environment for sigprofiler other than home directory (/home/yingya) by singualrity, the following command works for me: /share/app/singularity/3.2.0/bin/singularity run -e -B /path/to/Miniconda3:/home/yingya docker://shixiangwang/sigflow extract -i /opt/test/tcga_laml.maf.gz -o ./test_docker3 -m SBS -r 2 --max 10 -T 2 --sigprofiler

and then could run with "--sigprofiler": /share/app/singularity/3.2.0/bin/singularity run -e -B /path/to/Miniconda3:/home/yingya /path/to/sigflow_latest.sif extract -i /opt/test/tcga_laml.maf.gz -o ./test_docker3 -m SBS -r 2 --max 10 -T 2 --sigprofiler

[ShixiangWang]

I am glad that you find a solution. Could you get the result outside the container now?

[YingYa]

Yes， I could get the result outside the container. Thanks for your help.

[ShixiangWang]

You're welcome. I am closing this issue. Thanks for your feedback :).