search

kausmees / GenoCAE

Convolutional autoencoder for genotype data
BSD 3-Clause "New" or "Revised" License
15 stars 10 forks source link

Docker script does not build anymore? #27

Closed richelbilderbeek closed 2 years ago

richelbilderbeek commented 2 years ago

Dear GenoCAE maintainer, hi Carl and Kristiina,

Thanks for GenoCAE as well as the Docker container script: It's great for running GenoCAE on a computer cluster :-)

This Issue is related to #26, which is probably also caused by an upstream update: the Docker file does not work anymore. The installation instructions at https://github.com/kausmees/GenoCAE#docker-installation are great! Doing the suggested command, i.e. (note I added sudo) ...

sudo docker build -t gcae/genocae:build -f docker/build.dockerfile .

... results in a failed build, with a full error log below.

I have been trying the whole day ( for example, there are 6 failed attempts here), but could not fix this.

Does the Docker build work for you? Do you have an idea how to fix the Docker file?

A temporary workaround could be to upload an existing Docker container to Docker hub. Do you happen to have one? Would be awesome!

I hope it will be easy for you to help me solve this. I am not very experiences with Docker nor Python, so I can imagine an easy fix being possible (on the other hand, the 6 Stack Overflow 'solutions' hint that the problem is there).

To reproduce, I have created a script to build the Docker container, together with a GitHub Actions script with an error log here.

I hope you can help me out here! Thanks and cheers, Richel

Full error log

Sending build context to Docker daemon  186.6MB
Step 1/15 : ARG CUDA_VERSION=11.1.1
Step 2/15 : ARG OS_VERSION=20.04
Step 3/15 : FROM nvidia/cuda:${CUDA_VERSION}-cudnn8-devel-ubuntu${OS_VERSION}
 ---> 1189781af5ec
Step 4/15 : LABEL maintainer="Dong Wang"
 ---> Using cache
 ---> 9ae2635141d3
Step 5/15 : ENV PATH="/root/miniconda3/bin:${PATH}"
 ---> Using cache
 ---> f907151c27bd
Step 6/15 : ARG PATH="/root/miniconda3/bin:${PATH}"
 ---> Using cache
 ---> 76b31f23bd5e
Step 7/15 : SHELL ["/bin/bash", "-c"]
 ---> Using cache
 ---> e52cb6a4a70e
Step 8/15 : RUN apt-get update && apt-get upgrade -y &&     apt-get install -y wget
 ---> Using cache
 ---> f779a2c5021a
Step 9/15 : RUN wget     https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh     && mkdir /root/.conda     && bash Miniconda3-latest-Linux-x86_64.sh -b     && rm -f Miniconda3-latest-Linux-x86_64.sh
 ---> Using cache
 ---> e86837b5b18e
Step 10/15 : RUN pip3 install --upgrade pip
 ---> Running in 7a97102a4336
Requirement already satisfied: pip in /root/miniconda3/lib/python3.9/site-packages (21.1.3)
Collecting pip
  Downloading pip-22.0.3-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.1.3
    Uninstalling pip-21.1.3:
      Successfully uninstalled pip-21.1.3
Successfully installed pip-22.0.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Removing intermediate container 7a97102a4336
 ---> b48bafcfa623
Step 11/15 : RUN pip3 install --upgrade setuptools
 ---> Running in ea564922654f
Requirement already satisfied: setuptools in /root/miniconda3/lib/python3.9/site-packages (52.0.0.post20210125)
Collecting setuptools
  Downloading setuptools-60.8.1-py3-none-any.whl (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 13.7 MB/s eta 0:00:00
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 52.0.0.post20210125
    Uninstalling setuptools-52.0.0.post20210125:
      Successfully uninstalled setuptools-52.0.0.post20210125
Successfully installed setuptools-60.8.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Removing intermediate container ea564922654f
 ---> 874907ea03a5
Step 12/15 : WORKDIR /workspace
 ---> Running in ca79d4cb1457
Removing intermediate container ca79d4cb1457
 ---> a96a63e990ee
Step 13/15 : ADD ./requirements.txt /workspace
 ---> ad981a4056bd
Step 14/15 : RUN pip3 install -r /workspace/requirements.txt and && rm /workspace/requirements.txt
 ---> Running in 47c7e4f5b30c
Collecting and
  Downloading and-0.1.1-py3-none-any.whl (2.0 kB)
Collecting docopt
  Downloading docopt-0.6.2.tar.gz (25 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting grpcio
  Downloading grpcio-1.43.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.1/4.1 MB 24.1 MB/s eta 0:00:00
Collecting setuptools==47.1.1
  Downloading setuptools-47.1.1-py3-none-any.whl (583 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 583.2/583.2 KB 32.3 MB/s eta 0:00:00
Collecting tensorflow>=2.2.0
  Downloading tensorflow-2.8.0-cp39-cp39-manylinux2010_x86_64.whl (497.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 497.6/497.6 MB 4.1 MB/s eta 0:00:00
Collecting numpy==1.18.4
  Downloading numpy-1.18.4.zip (5.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.4/5.4 MB 26.8 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting scikit-learn
  Downloading scikit_learn-1.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (26.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 26.4/26.4 MB 25.0 MB/s eta 0:00:00
Collecting matplotlib==3.2.1
  Downloading matplotlib-3.2.1.tar.gz (40.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.3/40.3 MB 20.9 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting seaborn
  Downloading seaborn-0.11.2-py3-none-any.whl (292 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 292.8/292.8 KB 26.8 MB/s eta 0:00:00
Collecting scipy==1.4.1
  Downloading scipy-1.4.1.tar.gz (24.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24.6/24.6 MB 24.9 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [171 lines of output]
      setup.py:418: UserWarning: Unrecognized setuptools command ('dist_info --egg-base /tmp/pip-modern-metadata-jxp98lbc'), proceeding with generating Cython sources and expanding templates
        warnings.warn("Unrecognized setuptools command ('{}'), proceeding with "
      Running from scipy source directory.
      lapack_opt_info:
      lapack_mkl_info:
      customize UnixCCompiler
        libraries mkl_rt not found in ['/root/miniconda3/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
        NOT AVAILABLE

      openblas_lapack_info:
      customize UnixCCompiler
      customize UnixCCompiler
        libraries openblas not found in ['/root/miniconda3/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
        NOT AVAILABLE

      openblas_clapack_info:
      customize UnixCCompiler
      customize UnixCCompiler
        libraries openblas,lapack not found in ['/root/miniconda3/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
        NOT AVAILABLE

      flame_info:
      customize UnixCCompiler
        libraries flame not found in ['/root/miniconda3/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
        NOT AVAILABLE

      atlas_3_10_threads_info:
      Setting PTATLAS=ATLAS
      customize UnixCCompiler
        libraries lapack_atlas not found in /root/miniconda3/lib
      customize UnixCCompiler
        libraries tatlas,tatlas not found in /root/miniconda3/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries tatlas,tatlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib64
      customize UnixCCompiler
        libraries tatlas,tatlas not found in /usr/lib64
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib
      customize UnixCCompiler
        libraries tatlas,tatlas not found in /usr/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
      customize UnixCCompiler
        libraries tatlas,tatlas not found in /usr/lib/x86_64-linux-gnu
      <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
        NOT AVAILABLE

      atlas_3_10_info:
      customize UnixCCompiler
        libraries lapack_atlas not found in /root/miniconda3/lib
      customize UnixCCompiler
        libraries satlas,satlas not found in /root/miniconda3/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries satlas,satlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib64
      customize UnixCCompiler
        libraries satlas,satlas not found in /usr/lib64
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib
      customize UnixCCompiler
        libraries satlas,satlas not found in /usr/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
      customize UnixCCompiler
        libraries satlas,satlas not found in /usr/lib/x86_64-linux-gnu
      <class 'numpy.distutils.system_info.atlas_3_10_info'>
        NOT AVAILABLE

      atlas_threads_info:
      Setting PTATLAS=ATLAS
      customize UnixCCompiler
        libraries lapack_atlas not found in /root/miniconda3/lib
      customize UnixCCompiler
        libraries ptf77blas,ptcblas,atlas not found in /root/miniconda3/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib64
      customize UnixCCompiler
        libraries ptf77blas,ptcblas,atlas not found in /usr/lib64
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib
      customize UnixCCompiler
        libraries ptf77blas,ptcblas,atlas not found in /usr/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
      customize UnixCCompiler
        libraries ptf77blas,ptcblas,atlas not found in /usr/lib/x86_64-linux-gnu
      <class 'numpy.distutils.system_info.atlas_threads_info'>
        NOT AVAILABLE

      atlas_info:
      customize UnixCCompiler
        libraries lapack_atlas not found in /root/miniconda3/lib
      customize UnixCCompiler
        libraries f77blas,cblas,atlas not found in /root/miniconda3/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries f77blas,cblas,atlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib64
      customize UnixCCompiler
        libraries f77blas,cblas,atlas not found in /usr/lib64
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib
      customize UnixCCompiler
        libraries f77blas,cblas,atlas not found in /usr/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
      customize UnixCCompiler
        libraries f77blas,cblas,atlas not found in /usr/lib/x86_64-linux-gnu
      <class 'numpy.distutils.system_info.atlas_info'>
        NOT AVAILABLE

      accelerate_info:
        NOT AVAILABLE

      lapack_info:
      customize UnixCCompiler
        libraries lapack not found in ['/root/miniconda3/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
        NOT AVAILABLE

      /tmp/pip-build-env-jf9lnjy9/overlay/lib/python3.9/site-packages/numpy/distutils/system_info.py:1712: UserWarning:
          Lapack (http://www.netlib.org/lapack/) libraries not found.
          Directories to search for the libraries can be specified in the
          numpy/distutils/site.cfg file (section [lapack]) or by setting
          the LAPACK environment variable.
        if getattr(self, '_calc_info_{}'.format(lapack))():
      lapack_src_info:
        NOT AVAILABLE

      /tmp/pip-build-env-jf9lnjy9/overlay/lib/python3.9/site-packages/numpy/distutils/system_info.py:1712: UserWarning:
          Lapack (http://www.netlib.org/lapack/) sources not found.
          Directories to search for the sources can be specified in the
          numpy/distutils/site.cfg file (section [lapack_src]) or by setting
          the LAPACK_SRC environment variable.
        if getattr(self, '_calc_info_{}'.format(lapack))():
        NOT AVAILABLE

      Traceback (most recent call last):
        File "/root/miniconda3/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/root/miniconda3/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/root/miniconda3/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
        File "/tmp/pip-build-env-jf9lnjy9/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 173, in prepare_metadata_for_build_wheel
          self.run_setup()
        File "/tmp/pip-build-env-jf9lnjy9/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 266, in run_setup
          super(_BuildMetaLegacyBackend,
        File "/tmp/pip-build-env-jf9lnjy9/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 157, in run_setup
          exec(compile(code, __file__, 'exec'), locals())
        File "setup.py", line 540, in <module>
          setup_package()
        File "setup.py", line 536, in setup_package
          setup(**metadata)
        File "/tmp/pip-build-env-jf9lnjy9/overlay/lib/python3.9/site-packages/numpy/distutils/core.py", line 137, in setup
          config = configuration()
        File "setup.py", line 435, in configuration
          raise NotFoundError(msg)
      numpy.distutils.system_info.NotFoundError: No lapack/blas resources found.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
richelbilderbeek commented 2 years ago

I just sent an email to UPPMAX support, maybe they have the resources to help us out :-)

richelbilderbeek commented 2 years ago

I got a fix from Pavlin Mitev that is confirmed to work by GHA

richelbilderbeek commented 2 years ago

Oooh, build a Docker container with

sudo docker save 73004ca590ec | gzip > docker_container.tar.gz
richelbilderbeek commented 2 years ago

I got this to work at https://github.com/richelbilderbeek/GenoCAE/commit/0dc24d1dddc3efb725d26e00d20f19c6d13f3b43 . Happily closing this Issue!