docker build fails when trying to install pyshtools

[hirasso]

Hi there, what a great project. Thank you!!

I am on an M1 MacbookPro and am trying to use Docker (version 20.10.11, build dea9396) to make the utils/multires/generate.py script work. When running docker build -t generate-panorama . as instructed, I get the following error from Docker (sorry, a lot of info):

 => ERROR [3/4] RUN pip3 install pyshtools                                                                                                                 4.0s
------                                                                                                                                                          
 > [3/4] RUN pip3 install pyshtools:                                                                                                                            
#6 0.725 Collecting pyshtools                                                                                                                                   
#6 0.853   Downloading pyshtools-4.9.1.tar.gz (37.7 MB)                                                                                                         
#6 3.981     ERROR: Command errored out with exit status 1:                                                                                                     
#6 3.981      command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-1buygh4k/pyshtools/setup.py'"'"'; __file__='"'"'/tmp/pip-install-1buygh4k/pyshtools/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-1buygh4k/pyshtools/pip-egg-info
#6 3.981          cwd: /tmp/pip-install-1buygh4k/pyshtools/
#6 3.981     Complete output (59 lines):
#6 3.981     *** pypandoc is not installed. PYPI long_description will not be formatted correctly. ***
#6 3.981     INSTALLING SHTOOLS v4.9.1
#6 3.981     get_default_fcompiler: matching types: '['gnu95', 'intel', 'lahey', 'pg', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor']'
#6 3.981     customize Gnu95FCompiler
#6 3.981     Could not locate executable gfortran
#6 3.981     Could not locate executable f95
#6 3.981     customize IntelFCompiler
#6 3.981     Could not locate executable ifort
#6 3.981     Could not locate executable ifc
#6 3.981     customize LaheyFCompiler
#6 3.981     Could not locate executable lf95
#6 3.981     customize PGroupFCompiler
#6 3.981     Could not locate executable pgfortran
#6 3.981     customize AbsoftFCompiler
#6 3.981     Could not locate executable f90
#6 3.981     Could not locate executable f77
#6 3.981     customize NAGFCompiler
#6 3.981     customize VastFCompiler
#6 3.981     customize CompaqFCompiler
#6 3.981     Could not locate executable fort
#6 3.981     customize IntelItaniumFCompiler
#6 3.981     Could not locate executable efort
#6 3.981     Could not locate executable efc
#6 3.981     customize IntelEM64TFCompiler
#6 3.981     customize GnuFCompiler
#6 3.981     Could not locate executable g77
#6 3.981     customize G95FCompiler
#6 3.981     Could not locate executable g95
#6 3.981     customize PathScaleFCompiler
#6 3.981     Could not locate executable pathf95
#6 3.981     customize NAGFORCompiler
#6 3.981     Could not locate executable nagfor
#6 3.981     searching SHTOOLS in: build/temp.linux-aarch64-3.8
#6 3.981     fftw_info:
#6 3.981     customize UnixCCompiler
#6 3.981       libraries fftw3 not found in ['/usr/local/lib', '/usr/lib']
#6 3.981       fftw3 not found
#6 3.981     customize UnixCCompiler
#6 3.981       libraries rfftw,fftw not found in ['/usr/local/lib', '/usr/lib']
#6 3.981       fftw2 not found
#6 3.981     /usr/lib/python3/dist-packages/setuptools/dist.py:473: UserWarning: Normalizing 'v4.9.1' to '4.9.1'
#6 3.981       warnings.warn(
#6 3.981     Traceback (most recent call last):
#6 3.981       File "<string>", line 1, in <module>
#6 3.981       File "/tmp/pip-install-1buygh4k/pyshtools/setup.py", line 243, in <module>
#6 3.981         setup(**metadata)
#6 3.981       File "/usr/lib/python3/dist-packages/numpy/distutils/core.py", line 137, in setup
#6 3.981         config = configuration()
#6 3.981       File "/tmp/pip-install-1buygh4k/pyshtools/setup.py", line 194, in configuration
#6 3.981         fftw_info = get_info('fftw', notfound_action=2)
#6 3.981       File "/usr/lib/python3/dist-packages/numpy/distutils/system_info.py", line 444, in get_info
#6 3.981         return cl().get_info(notfound_action)
#6 3.981       File "/usr/lib/python3/dist-packages/numpy/distutils/system_info.py", line 696, in get_info
#6 3.981         raise self.notfounderror(self.notfounderror.__doc__)
#6 3.981     numpy.distutils.system_info.FFTWNotFoundError:
#6 3.981         FFTW (http://www.fftw.org/) libraries not found.
#6 3.981         Directories to search for the libraries can be specified in the
#6 3.981         numpy/distutils/site.cfg file (section [fftw]) or by setting
#6 3.981         the FFTW environment variable.
#6 3.981     ----------------------------------------
#6 3.999 ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
------
executor failed running [/bin/sh -c pip3 install pyshtools]: exit code: 1

The Docker container is not being built.

Any Idea? Maybe the Docker GitHub would be a better place to post this issue?

[hirasso]

I was able to install the dependencies manually like so:

$ brew install python@3.9
$ brew install pillow
$ brew install numpy
$ brew install --cask hugin
$ pip3 install --user pyshtools

Additional Steps

• open and allow the Apps in /Applications/Hugin/
• make nona available: $ export PATH="/Applications/Hugin/PTBatcherGUI.app/Contents/MacOS:$PATH"

But I still would like to know how to get it to work with Docker...

[hirasso]

If I deactivate pyshtools like this:

# RUN pip3 install pyshtools

...the Docker build passes, but of course the shtHashes won't be generated.

[mpetroff]

The Dockerfile assumes that pyshtools will be installed as a binary Wheel. However, PyPI only has Wheels for x86_64, so installation on Arm requires building from source, which requires additional dependencies. Per the pyshtools documentation, adding gcc gfortran libfftw3-dev libblas-dev liblapack-dev to the end of the apt-get install line should install the dependencies required for building from source.

[hirasso]

WOW @mpetroff , that worked!!! I changed my Dockerfile as follows:

FROM ubuntu:20.04

# docker build -t generate-panorama .
# docker run -it -v $PWD:/data generate-panorama /data/image.jpg

ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \
    python3 python3-dev python3-numpy python3-pip python3-pil hugin-tools \
    gcc gfortran libfftw3-dev libblas-dev liblapack-dev \
 && rm -rf /var/lib/apt/lists/*
RUN pip3 install pyshtools --no-binary pyshtools

ADD generate.py /generate.py
ENTRYPOINT ["python3", "/generate.py"]

Thank you so much for your fast help!

[mpetroff]

I guess the ideal solution would be to check if a binary wheel exists for pyshtools for the architecture Docker is being run on. If it exists, the wheel should be installed. If not, the additional distro packages should be installed, and pyshtools should then be built from the source package. Unfortunately, there doesn't seem to be a good way to check this with pip. If pyshtools started uploading binary wheels for Arm, that would also solve this issue.

[cyango]

Hi @mpetroff , I tried @hirasso solution on my mac m1, and getting the following error:

Collecting pyshtools Downloading pyshtools-4.11.10.tar.gz (41.5 MB) Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Installing backend dependencies: started Installing backend dependencies: finished with status 'done' Preparing wheel metadata: started Preparing wheel metadata: finished with status 'error' ERROR: Command errored out with exit status 1: command: /usr/bin/python3 /tmp/tmp4ewa_eo9 prepare_metadata_for_build_wheel /tmp/tmp7uyhowwi cwd: /tmp/pip-install-k33pyqxz/pyshtools Complete output (30 lines):

• meson setup /tmp/pip-install-k33pyqxz/pyshtools /tmp/pip-install-k33pyqxz/pyshtools/.mesonpy-4jxnlo60 -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --default-library=static --native-file=/tmp/pip-install-k33pyqxz/pyshtools/.mesonpy-4jxnlo60/meson-python-native-file.ini The Meson build system Version: 1.5.1 Source dir: /tmp/pip-install-k33pyqxz/pyshtools Build dir: /tmp/pip-install-k33pyqxz/pyshtools/.mesonpy-4jxnlo60 Build type: native build Program python3 found: YES (/usr/bin/python3) WARNING: You should add the boolean check kwarg to the run_command call. It currently defaults to false, but it will default to true in future releases of meson. See also: https://github.com/mesonbuild/meson/issues/9300 Project name: shtools Project version: 4.11.10 Fortran compiler for the host machine: gfortran (gcc 9.4.0 "GNU Fortran (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0") Fortran linker for the host machine: gfortran ld.bfd 2.34 C compiler for the host machine: cc (gcc 9.4.0 "cc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0") C linker for the host machine: cc ld.bfd 2.34 Host machine cpu family: x86_64 Host machine cpu: x86_64 Did not find pkg-config by name 'pkg-config' Found pkg-config: NO Did not find CMake 'cmake' Found CMake: NO Run-time dependency openblas found: NO (tried pkgconfig and cmake) Run-time dependency openblas found: NO (tried pkgconfig and cmake) Run-time dependency blas found: NO

../meson.build:15:9: ERROR: Dependency lookup for BLAS with method 'pkgconfig' failed: Pkg-config for machine host machine not found. Giving up.

[mpetroff]

@cyango I think you ran into the same issue that is described in #1205.

I just made some updates to the Dockerfile in 9beb4184e34dc0073ab35bbebf0eba9eac1d4e86 and badf8812c39aaf515e6a8aa2f5f034331b33aa18, and with these updates, it will hopefully just work.

[cyango]

It fixed it, thanks!