search

babelomics / impuSARS

Software tool for the imputation of whole-genome SARS-CoV-2 sequences.
MIT License
4 stars 0 forks source link

Could not install docker version #2

Open ahmetgorkemer opened 1 year ago

ahmetgorkemer commented 1 year ago

Hi,

The docker file raises an error during installation.

=> ERROR [ 3/52] RUN pip install cget 2.2s

[ 3/52] RUN pip install cget:

6 0.545 Collecting cget

6 0.871 Downloading https://files.pythonhosted.org/packages/d0/1a/1ff282ab3fb810812841b5e7d2b3e87cd3bd308e79d507da950839b8f2a7/cget-0.2.0.tar.gz

6 0.961 Collecting click>=6.6 (from cget)

6 1.070 Downloading https://files.pythonhosted.org/packages/59/87/84326af34517fca8c58418d148f2403df25303e02736832403587318e9e8/click-8.1.3.tar.gz (331kB)

6 1.215 Requested click>=6.6 from https://files.pythonhosted.org/packages/59/87/84326af34517fca8c58418d148f2403df25303e02736832403587318e9e8/click-8.1.3.tar.gz#sha256=7682dc8afb30297001674575ea00d1814d808d6a36af415a82bd481d37ba7b8e (from cget), but installing version None

6 1.222 Collecting six>=1.10 (from cget)

6 1.384 Downloading https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl

6 1.396 Collecting subprocess32 (from cget)

6 1.499 Downloading https://files.pythonhosted.org/packages/32/c8/564be4d12629b912ea431f1a50eb8b3b9d00f1a0b1ceff17f266be190007/subprocess32-3.5.4.tar.gz (97kB)

6 1.593 Collecting colorama; platform_system == "Windows" (from click>=6.6->cget)

6 1.704 Downloading https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl

6 1.713 Collecting importlib-metadata; python_version < "3.8" (from click>=6.6->cget)

6 1.830 Downloading https://files.pythonhosted.org/packages/7e/ec/97f2ce958b62961fddd7258e0ceede844953606ad09b672fa03b86c453d3/importlib_metadata-5.0.0.tar.gz (48kB)

6 1.895 Complete output from command python setup.py egg_info:

6 1.895 Traceback (most recent call last):

6 1.895 File "", line 1, in

6 1.895 IOError: [Errno 2] No such file or directory: '/tmp/pip-build-lXmN1K/importlib-metadata/setup.py'

6 1.895

6 1.895 ----------------------------------------

6 1.904 Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-lXmN1K/importlib-metadata/

6 2.131 You are using pip version 8.1.1, however version 22.3.1 is available.

6 2.131 You should consider upgrading via the 'pip install --upgrade pip' command.

executor failed running [/bin/sh -c pip install cget]: exit code: 1

I think it is related to the pip version that was coded in the docker file, it seems that it should be updated.

Best, Ahmet

fortuno commented 1 year ago

Hi Ahmet!

Sorry about that! You are right there were some issues with the pip version installed in the docker.

We have updated the base distribution in the docker to Ubuntu 20.04 and made some additional adjustments to make it work. The docker file should be working now. Please, let us know when you are able to correctly install it or you have any other issue. We will close the issue just after your confirmation.

Thank you very much for your interest!

ahmetgorkemer commented 1 year ago

Hi,

Unfortunately, it is not still working. This time, it has raised an different error:

Cloning into 'impuSARS'... remote: Enumerating objects: 346, done. remote: Counting objects: 100% (43/43), done. remote: Compressing objects: 100% (20/20), done. remote: Total 346 (delta 27), reused 33 (delta 23), pack-reused 303 Receiving objects: 100% (346/346), 87.20 MiB | 4.35 MiB/s, done. Resolving deltas: 100% (203/203), done. (base) ahmetgorkemer@DN0a259194 ~ % cd impuSARS (base) ahmetgorkemer@DN0a259194 impuSARS % ./install_impuSARS [ impuSARS ] Building impuSARS Docker image... [+] Building 161.0s (47/58)
=> [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 2.96kB 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/ubuntu:20.04 1.0s => [ 1/53] FROM docker.io/library/ubuntu:20.04@sha256:450e066588f42ebe15 1.7s => => resolve docker.io/library/ubuntu:20.04@sha256:450e066588f42ebe1551 0.0s => => sha256:450e066588f42ebe1551f3b1a535034b6aa46cd936f 1.42kB / 1.42kB 0.0s => => sha256:72dd010fadf5d9da8197f433fa6ac50179307ce0fc5ac69 529B / 529B 0.0s => => sha256:89867091bfb2626393d967e92b55842b547f249ec12 1.48kB / 1.48kB 0.0s => => sha256:4e7e0215f4adc2c48ad9cb3b3781e21d474b47758 27.20MB / 27.20MB 1.2s => => extracting sha256:4e7e0215f4adc2c48ad9cb3b3781e21d474b477587f85682 0.3s => https://github.com/cov-lineages/pangolin/archive/refs/tags/v4.1.3.tar 0.0s => CACHED https://codeload.github.com/statgen/Minimac4/tar.gz/refs/tags/ 0.0s => CACHED https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x8 0.0s => [internal] load build context 0.3s => => transferring context: 40.96MB 0.3s => CACHED https://github.com/samtools/bcftools/releases/download/1.11/bc 0.0s => CACHED https://github.com/samtools/htslib/releases/download/1.11/htsl 0.0s => [ 2/53] RUN apt-get update && apt install -y build-essential 26.4s => [ 3/53] RUN pip install cget 1.2s => [ 4/53] ADD https://github.com/samtools/bcftools/releases/download/1. 0.0s => [ 5/53] RUN tar -xf bcftools-1-11.tar.bz2 0.6s => [ 6/53] WORKDIR /bcftools-1.11 0.0s => [ 7/53] RUN ./configure --prefix=/ 2.8s => [ 8/53] RUN make 43.8s => [ 9/53] RUN make install 0.2s => [10/53] RUN rm /bcftools-1-11.tar.bz2 0.1s => [11/53] ADD https://github.com/samtools/htslib/releases/download/1.11 0.0s => [12/53] RUN tar -xf htslib-1.11.tar.bz2 0.3s => [13/53] WORKDIR /htslib-1.11 0.0s => [14/53] RUN ./configure --prefix=/ 2.2s => [15/53] RUN make 34.7s => [16/53] RUN make install 0.4s => [17/53] RUN rm /htslib-1.11.tar.bz2 0.3s => [18/53] ADD https://codeload.github.com/statgen/Minimac4/tar.gz/refs/ 0.0s => [19/53] RUN tar xvzf Minimac4-1.0.2.tar.gz 0.2s => [20/53] RUN mv Minimac4-1.0.2 Minimac4 0.3s => [21/53] WORKDIR /Minimac4 0.0s => [22/53] RUN sed -i 's/zlib/#zlib/g' requirements.txt 0.2s => [23/53] RUN bash install.sh 41.2s => [24/53] RUN curl ftp://share.sph.umich.edu/minimac3/Minimac3Executabl 1.6s => [25/53] RUN tar xvzf Minimac3Executable.tar.gz 0.3s => [26/53] COPY ./docker_files/references/SARS_CoV_2_IMPUTATION_PANEL.v4 0.1s => [27/53] COPY ./docker_files/references/SARS_CoV_2_REFERENCE.v1.0.fast 0.0s => [28/53] COPY ./docker_files/references/SARS_CoV_2_REFERENCE.v1.0.fast 0.0s => [29/53] COPY ./docker_files/references/REFERENCE_N.fa /Minimac4/relea 0.0s => [30/53] COPY ./docker_files/references/VCF_headers.txt /Minimac4/rele 0.0s => [31/53] COPY ./docker_files/impuSARS /Minimac4/release-build/ 0.0s => [32/53] COPY ./docker_files/fasta2vcf /Minimac4/release-build/ 0.0s => [33/53] COPY ./docker_files/fixFASTA.py /Minimac4/release-build/ 0.0s => [34/53] COPY ./docker_files/fixvcf.py /Minimac4/release-build/ 0.0s => [35/53] COPY ./docker_files/impuSARS_reference /Minimac4/release-buil 0.0s => [36/53] ADD https://repo.anaconda.com/miniconda/Miniconda3-latest-Lin 0.1s => [37/53] RUN chmod 755 Miniconda3.sh 0.3s => ERROR [38/53] RUN mkdir /root/.conda && bash Miniconda3.sh -b 0.6s

[38/53] RUN mkdir /root/.conda && bash Miniconda3.sh -b && rm -f Miniconda3.sh:

46 0.341 PREFIX=/root/miniconda3

46 0.572 Unpacking payload ...

46 0.576 qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory

46 0.579 qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory

executor failed running [/bin/sh -c mkdir /root/.conda && bash Miniconda3.sh -b && rm -f Miniconda3.sh]: exit code: 1 (base) ahmetgorkemer@DN0a259194 impuSARS %

By the way, conda version also could not be installed. I have tried a few different platforms and computers, but every time it stucks in installing of Minimac 4.

Checking if conda installed... Adding bioconda/conda-forge channels to conda... Warning: 'bioconda' already in 'channels' list, moving to the top Warning: 'conda-forge' already in 'channels' list, moving to the top Creating impusars conda environment... Collecting package metadata (repodata.json): done Solving environment: failed

ResolvePackageNotFound:

Warning: '/Users/ahmetgorkemer/impuSARS/.venvs/' already in 'envs_dirs' list, moving to the bottom

EnvironmentNameNotFound: Could not find conda environment: impusars You can list all discoverable environments with conda info --envs.

ln: /Users/ahmetgorkemer/opt/anaconda3/bin/gcc: File exists ln: /Users/ahmetgorkemer/opt/anaconda3/bin/g++: File exists ln: /Users/ahmetgorkemer/opt/anaconda3/bin/ar: File exists Installing Minimac imputation... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 87759 0 87759 0 0 267k 0 --:--:-- --:--:-- --:--:-- 270k % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 819k 100 819k 0 0 741k 0 0:00:01 0:00:01 --:--:-- 745k x Minimac4-1.0.2/ x Minimac4-1.0.2/.gitignore x Minimac4-1.0.2/CMakeLists.txt x Minimac4-1.0.2/LICENSE x Minimac4-1.0.2/README.md x Minimac4-1.0.2/dep/ x Minimac4-1.0.2/dep/libstatgen.cmake x Minimac4-1.0.2/install.sh x Minimac4-1.0.2/package-linux.sh x Minimac4-1.0.2/packaging-dockerfile-ubuntu16 x Minimac4-1.0.2/requirements.txt x Minimac4-1.0.2/src/ x Minimac4-1.0.2/src/Analysis.cpp x Minimac4-1.0.2/src/Analysis.h x Minimac4-1.0.2/src/AnalysisChunks.cpp x Minimac4-1.0.2/src/DosageData.cpp x Minimac4-1.0.2/src/DosageData.h x Minimac4-1.0.2/src/Estimation.cpp x Minimac4-1.0.2/src/Estimation.h x Minimac4-1.0.2/src/HaplotypeSet.cpp x Minimac4-1.0.2/src/HaplotypeSet.h x Minimac4-1.0.2/src/Imputation.cpp x Minimac4-1.0.2/src/Imputation.h x Minimac4-1.0.2/src/ImputationStatistics.cpp x Minimac4-1.0.2/src/ImputationStatistics.h x Minimac4-1.0.2/src/Main.cpp x Minimac4-1.0.2/src/MarkovModel.cpp x Minimac4-1.0.2/src/MarkovModel.h x Minimac4-1.0.2/src/MarkovParameters.cpp x Minimac4-1.0.2/src/MarkovParameters.h x Minimac4-1.0.2/src/MyVariables.h x Minimac4-1.0.2/src/Unique.cpp x Minimac4-1.0.2/src/Unique.h x Minimac3Executable/ x Minimac3Executable/bin/ x Minimac3Executable/bin/Minimac3 x Minimac3Executable/bin/Minimac3-omp x Minimac3Executable/test/ x Minimac3Executable/test/refPanelChrX.Non.Auto.vcf x Minimac3Executable/test/refPanel.m3vcf.gz x Minimac3Executable/test/refPanel.vcf x Minimac3Executable/test/refPanel.vcf.gz x Minimac3Executable/test/targetStudyChrX.females.vcf x Minimac3Executable/test/targetStudyChrX.males.vcf x Minimac3Executable/test/targetStudy.vcf x Minimac3Executable/test/TestRun_1.sh x Minimac3Executable/test/TestRun_2.sh x Minimac3Executable/test/TestRun_3.sh x Minimac3Executable/README.txt Installing Dependencies - Libstatgen ... Generating MakeFiles ... install.sh: line 9: cmake: command not found make: *** No targets specified and no makefile found. Stop. Binary created at /release-build/minimac4 cp: /Users/ahmetgorkemer/impuSARS/conda/Minimac4-1.0.2/release-build/minimac4: No such file or directory

I would be glad if you find solutions to these problems. I am looking forward to using the tool for imputing the genome.

Best, Ahmet

fortuno commented 1 year ago

Hi Ahmet,

Sorry for these inconveniences, we are working to solve them and make it works for you.

According to your logs, it seems likes the docker is taking some of the installation from cache (from a previous installation). We have updated the repository to make sure docker is rebuilt completely (without using cache). We have also forced a specific ingest for the base Ubuntu v20.04 to be sure you are installing same version as the one working. Would you mind to retry again installing the docker version with these changes, please?

If still not working, we would like to know what Docker version and SO you are using (for docker version you can run the command "docker --version"). That way we will try to reproduce the error in our machines.

Also, we are working on the conda fix too, but that one could be more complex since it has strong dependencies on each machine packages and libraries.

Thanks again for your interest!

ahmetgorkemer commented 1 year ago

Hi Francisco,

I have tried to install the tool in different settings. Unfortunately, all of them are stuck at the same point:

Step 44/60 : RUN chmod 755 Miniconda3.sh  ---> Running in 4bd5dc044b4d Removing intermediate container 4bd5dc044b4d  ---> 3c0e0b4935e1 Step 45/60 : RUN mkdir /root/.conda && bash Miniconda3.sh -b && rm -f Miniconda3.sh  ---> Running in 9498968abc72 PREFIX=/root/miniconda3 Unpacking payload ... Miniconda3.sh: line 412: /root/miniconda3/conda.exe: cannot execute binary file: Exec format error Miniconda3.sh: line 414: /root/miniconda3/conda.exe: cannot execute binary file: Exec format error The command '/bin/sh -c mkdir /root/.conda && bash Miniconda3.sh -b && rm -f Miniconda3.sh' returned a non-zero code: 1

I have re-installed Docker, version 20.10.21, and used macOS Monterey, Ubuntu 20.04, and Ubuntu 22.04 as operating systems. I hope we can find a solution.

Best, Ahmet On Nov 21, 2022, 4:31 AM -0800, Francisco Ortuno @.***>, wrote:

Hi Ahmet, Sorry for these inconveniences, we are working to solve them and make it works for you. According to your logs, it seems likes the docker is taking some of the installation from cache (from a previous installations). We have updated the repository to make sure docker is rebuilt completely (without using cache). We have also forced a specific ingest for the base Ubuntu v20.04 to be sure you are installing same version as the one working. Would you mind to retry again installing the docker version with these changes, please? If still not working, we would like to know what Docker version and SO you are using (for docker version you can run the command "docker --version"). That way we will try to reproduce the error in our machines. Also, we are working on the conda fix too, but that one could be more complex since it has strong dependencies on each machine packages and libraries. Thanks again for your interest! — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

fortuno commented 1 year ago

Hi Ahmet,

I have it working in Ubuntu 20.04. For the error you sent, I suspect it could be an architecture issue. Would you mind to let me know what architecture you are trying to install in? You can just run the command uname -m

Once I have your architecture I will update the Dockerfile to make it working for your architecture.

Thanks again! Francisco

ahmetgorkemer commented 1 year ago

Hi Francisco,

The architecture is “aarch64” on Ubuntu 20.04 (I am using it on UTM as a virtual OS), and “x86_64” on my Mac with an M1 processor.

Best, Ahmet

On Nov 25, 2022, 2:56 AM -0800, Francisco Ortuno @.***>, wrote:

Hi Ahmet, I have it working in Ubuntu 20.04. For the error you sent, I suspect it could be an architecture issue. Would you mind to let me know what architecture you are trying to install in? You can just run the command uname -m Once I have your architecture I will update the Dockerfile to make it working for your architecture. Thanks again! Francisco — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

fortuno commented 1 year ago

Hi Ahmet,

Yeah, it seems to be a problem with the architecure. The Dockerfile installs miniconda specifically for x86_64 but I have read there are some issue with the M1 processor.

We have updated the Dockerfile to automatically identify architecture and install miniconda accordingly. It should be able now to detect your aarch64 architecture in Ubuntu 20.04 and install miniconda for that architecture. However, since it is a virtual OS over UTM I am not sure if it will work. Could you please give it a try and let me know if it is working?

Meanwhile I would try to make it work with M1 processor but I don't have where to test it. Sorry about that!

Thanks again! Francisco