Closed Rhys-L closed 10 months ago
Hey Rhys, can you try the following for a dockerfile?
FROM jupyter/scipy-notebook:2021-11-20
USER root
RUN apt update -y && apt upgrade -y && \
apt install -y software-properties-common && \
add-apt-repository ppa:ubuntugis/ppa && \
apt update -y && apt install -y \
gdal-bin \
geotiff-bin \
git \
libgdal-dev \
libgl1 \
libspatialindex-dev \
wget \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
RUN pip uninstall numpy -y
# Install Conda packages
RUN conda install -c conda-forge -y \
#python=3.7 # had to be excluded because newer version of python req by Jupyterhub \
cython \
gdal \
libspatialindex \
nbgrader \
#nbtutor \
numpy \
opencv \
pip \
pygeos \
requests \
rtree \
scipy \
wheel \
zarr \
&& conda install -c phausamann sklearn-xarray -y \
&& conda clean -a -y
RUN pip3 install --upgrade pip --no-cache-dir && pip3 install --no-cache-dir \
pip-tools \
git+https://github.com/jgrss/geowombat.git
I'm going to try to add ARM to my cond-forge build - I think the issue is it has specific C compiler needs.
It looks like the build got close but failed to from issues with rasterio and fiona. Going to keep googling through the error message, and wanted to share below in case anything jumps out.
67.28 INFO:root:gcc -pthread -B /opt/conda/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O3 -Wall -fPIC -O3 -isystem /opt/conda/include -fPIC -O3 -isystem /opt/conda/include -fPIC -I/tmp/pip-build-env-tkwl4h61/overlay/lib/python3.9/site-packages/numpy/core/include -I/opt/conda/include -I/opt/conda/include/python3.9 -c rasterio/_warp.cpp -o build/temp.linux-aarch64-cpython-39/rasterio/_warp.o -Wno-unused-parameter -Wno-unused-function -std=c++11
67.28 gcc: fatal error: cannot execute ‘cc1plus’: execvp: No such file or directory
67.28 compilation terminated.
67.28 error: command '/usr/bin/gcc' failed with exit code 1
67.28 [end of output]
67.28
67.28 note: This error originates from a subprocess, and is likely not a problem with pip.
67.28 ERROR: Failed building wheel for rasterio
67.28 Building wheel for fiona (pyproject.toml): started
69.50 Building wheel for fiona (pyproject.toml): finished with status 'error'
69.50 error: subprocess-exited-with-error
69.50
69.50 × Building wheel for fiona (pyproject.toml) did not run successfully.
69.50 │ exit code: 1
69.50 ╰─> [76 lines of output]
69.50 /tmp/pip-build-env-e4ryhszq/overlay/lib/python3.9/site-packages/setuptools/_distutils/extension.py:134: UserWarning: Unknown Extension options: 'cython_compile_time_env'
69.50 warnings.warn(msg)
69.50 INFO:root:running bdist_wheel
69.50 INFO:root:running build
69.50 INFO:root:running build_py
69.50 INFO:root:creating build
69.50 INFO:root:creating build/lib.linux-aarch64-cpython-39
69.50 INFO:root:creating build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/enums.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/vfs.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/path.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/crs.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/model.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/io.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/meta.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/drvsupport.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/session.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/collection.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/logutils.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/transform.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/compat.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/__init__.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/_show_versions.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/errors.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/env.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/inspector.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/rfc3339.py -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:creating build/lib.linux-aarch64-cpython-39/fiona/_vendor
69.50 INFO:root:copying fiona/_vendor/__init__.py -> build/lib.linux-aarch64-cpython-39/fiona/_vendor
69.50 INFO:root:creating build/lib.linux-aarch64-cpython-39/fiona/fio
69.50 INFO:root:copying fiona/fio/filter.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50 INFO:root:copying fiona/fio/options.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50 INFO:root:copying fiona/fio/bounds.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50 INFO:root:copying fiona/fio/ls.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50 INFO:root:copying fiona/fio/cat.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50 INFO:root:copying fiona/fio/collect.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50 INFO:root:copying fiona/fio/dump.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50 INFO:root:copying fiona/fio/info.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50 INFO:root:copying fiona/fio/insp.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50 INFO:root:copying fiona/fio/main.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50 INFO:root:copying fiona/fio/distrib.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50 INFO:root:copying fiona/fio/__init__.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50 INFO:root:copying fiona/fio/rm.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50 INFO:root:copying fiona/fio/calc.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50 INFO:root:copying fiona/fio/helpers.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50 INFO:root:copying fiona/fio/load.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50 INFO:root:copying fiona/fio/env.py -> build/lib.linux-aarch64-cpython-39/fiona/fio
69.50 INFO:root:creating build/lib.linux-aarch64-cpython-39/fiona/_vendor/munch
69.50 INFO:root:copying fiona/_vendor/munch/__init__.py -> build/lib.linux-aarch64-cpython-39/fiona/_vendor/munch
69.50 INFO:root:copying fiona/_vendor/munch/python3_compat.py -> build/lib.linux-aarch64-cpython-39/fiona/_vendor/munch
69.50 INFO:root:copying fiona/gdal.pxi -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/_err.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/crs.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/gdal.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/ogrext2.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/_geometry.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/_cpl.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/_csl.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/ogrext1.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/_shim.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/ogrext3.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:copying fiona/_env.pxd -> build/lib.linux-aarch64-cpython-39/fiona
69.50 INFO:root:running build_ext
69.50 INFO:root:building 'fiona._geometry' extension
69.50 INFO:root:creating build/temp.linux-aarch64-cpython-39
69.50 INFO:root:creating build/temp.linux-aarch64-cpython-39/fiona
69.50 INFO:root:gcc -pthread -B /opt/conda/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O3 -Wall -fPIC -O3 -isystem /opt/conda/include -fPIC -O3 -isystem /opt/conda/include -fPIC -I/opt/conda/include -I/opt/conda/include/python3.9 -c fiona/_geometry.c -o build/temp.linux-aarch64-cpython-39/fiona/_geometry.o
69.50 INFO:root:gcc -pthread -B /opt/conda/compiler_compat -shared -Wl,-rpath,/opt/conda/lib -Wl,-rpath-link,/opt/conda/lib -L/opt/conda/lib -Wl,-rpath,/opt/conda/lib -Wl,-rpath-link,/opt/conda/lib -L/opt/conda/lib build/temp.linux-aarch64-cpython-39/fiona/_geometry.o -L/opt/conda/lib -lgdal -o build/lib.linux-aarch64-cpython-39/fiona/_geometry.cpython-39-aarch64-linux-gnu.so
69.50 INFO:root:building 'fiona.schema' extension
69.50 INFO:root:gcc -pthread -B /opt/conda/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O3 -Wall -fPIC -O3 -isystem /opt/conda/include -fPIC -O3 -isystem /opt/conda/include -fPIC -I/opt/conda/include -I/opt/conda/include/python3.9 -c fiona/schema.c -o build/temp.linux-aarch64-cpython-39/fiona/schema.o
69.50 INFO:root:gcc -pthread -B /opt/conda/compiler_compat -shared -Wl,-rpath,/opt/conda/lib -Wl,-rpath-link,/opt/conda/lib -L/opt/conda/lib -Wl,-rpath,/opt/conda/lib -Wl,-rpath-link,/opt/conda/lib -L/opt/conda/lib build/temp.linux-aarch64-cpython-39/fiona/schema.o -L/opt/conda/lib -lgdal -o build/lib.linux-aarch64-cpython-39/fiona/schema.cpython-39-aarch64-linux-gnu.so
69.50 INFO:root:building 'fiona._transform' extension
69.50 INFO:root:gcc -pthread -B /opt/conda/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O3 -Wall -fPIC -O3 -isystem /opt/conda/include -fPIC -O3 -isystem /opt/conda/include -fPIC -I/opt/conda/include -I/opt/conda/include/python3.9 -c fiona/_transform.cpp -o build/temp.linux-aarch64-cpython-39/fiona/_transform.o -std=c++11
69.50 gcc: fatal error: cannot execute ‘cc1plus’: execvp: No such file or directory
69.50 compilation terminated.
69.50 error: command '/usr/bin/gcc' failed with exit code 1
69.50 [end of output]
69.50
69.50 note: This error originates from a subprocess, and is likely not a problem with pip.
69.50 ERROR: Failed building wheel for fiona
69.50 Successfully built geowombat
69.50 Failed to build rasterio fiona
69.50 ERROR: Could not build wheels for rasterio, fiona, which is required to install pyproject.toml-based projects
------
Dockerfile:41
--------------------
40 |
41 | >>> RUN pip3 install --upgrade pip --no-cache-dir && pip3 install --no-cache-dir \
42 | >>> pip-tools \
43 | >>> git+https://github.com/jgrss/geowombat.git
--------------------
ERROR: failed to solve: process "/bin/bash -o pipefail -c pip3 install --upgrade pip --no-cache-dir && pip3 install --no-cache-dir pip-tools git+https://github.com/jgrss/geowombat.git" did not complete successfully: exit code: 1
For dockerfile try the simple version:
FROM jupyter/scipy-notebook
USER root
RUN mamba install -c conda-forge -y geowombat
CMD ["/bin/bash"]
Got another error with that, I think because it is failing to find a version of the package. I'm currently looking into how we can use Docker compose and base image of linux with an intermediate image of a C compiler. Also reading up on this: https://learning.oreilly.com/library/view/learn-docker-in/9781617297052/OEBPS/Text/16.xhtml#heading_id_8, and open to any other docker reading/watching recommendations.
Error message:
> [2/2] RUN mamba install -c conda-forge -y geowombat:
8.332
8.332 __ __ __ __
8.332 / \ / \ / \ / \
8.332 / \/ \/ \/ \
8.332 ███████████████/ /██/ /██/ /██/ /████████████████████████
8.332 / / \ / \ / \ / \ \____
8.332 / / \_/ \_/ \_/ \ o \__,
8.332 / _/ \_____/ `
8.332 |/
8.332 ███╗ ███╗ █████╗ ███╗ ███╗██████╗ █████╗
8.332 ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗
8.332 ██╔████╔██║███████║██╔████╔██║██████╔╝███████║
8.332 ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║
8.332 ██║ ╚═╝ ██║██║ ██║██║ ╚═╝ ██║██████╔╝██║ ██║
8.332 ╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ ╚═╝╚═════╝ ╚═╝ ╚═╝
8.332
8.332 mamba (0.18.1) supported by @QuantStack
8.332
8.332 GitHub: https://github.com/mamba-org/mamba
8.332 Twitter: https://twitter.com/QuantStack
8.332
8.332 █████████████████████████████████████████████████████████████
8.332
8.332
8.332 Looking for: ['geowombat']
8.332
8.332
8.332 Pinned packages:
8.332 - python 3.9.*
8.332 - python 3.9.7
8.332
8.332
8.332 Encountered problems while solving:
8.332 - nothing provides requested geowombat
8.332
8.332 conda-forge/linux-aarch6
8.332 conda-forge/noarch
------
Dockerfile:5
--------------------
3 | USER root
4 |
5 | >>> RUN mamba install -c conda-forge -y geowombat
6 |
7 | CMD ["/bin/bash"]
--------------------
ERROR: failed to solve: process "/bin/bash -o pipefail -c mamba install -c conda-forge -y geowombat" did not complete successfully: exit code: 1
That is very strange. Geowombat is definitely available through conda-forge, and I was able to run that docker file. How are you building it? Could u be limiting Internet access somehow???
I cloned the repo and I'm inside the top level directory and running the following command:
docker image build -t geowombat:arm .
Are there any different flags I should use here? I don't think it's a connectivity issue from inside the image or from my home internet.
I'll let you know how the conda-forge build goes. Till then use apporto.
@Rhys-L @jgrss I just pushed the osx arm build to the conda feedstock. Hopefully within a few hours it should be good to go!
Hi @jgrss! I'm a student of Dr. Mann's @mmann1123 and Dr. Engstrom's, hoping to work with and build on Geowombat in some projects this semester. I've been having issues installing and running the package on an ARM-based mac. I have tried 3 different approaches, returning different errors. Do you have any recommendations or guidance on the best way to install? If you have any ideas on the docker image issue described below, I'm happy to push an ARM-version to a registry, or help debug more with any other approaches.
I have attempted to build from source locally using the following in a python 3.9.6 venv using the following commands
python setup.py build && python setup.py install
, but I run into some clang issues with some of the cython generated code, returning the following error message: `src/geowombat/moving/_moving.c:721:10: fatal error: 'omp.h' file not foundinclude
1 warning and 1 error generated. error: command '/usr/bin/clang' failed with exit code 1`
I also cloned the code and have been working on natively building a docker image on mac. I modified the Docker file to use the following base image and env, but keep running into gdal version issues inside the docker image:
FROM arm64v8/ubuntu:20.04 WORKDIR /source
I also modified env.yml and setup.cfg to contain
gdal>=3.1.0
. I also tried further modifying the docker file with variants of this, but haven't been able to successfully build the image:RUN pip install GDAL>=3.1.* --no-binary=gdal
Full error: `12.61 Installing build dependencies: started
15.60 Installing build dependencies: finished with status 'done'
15.60 Getting requirements to build wheel: started
15.73 Getting requirements to build wheel: finished with status 'error'
15.73 error: subprocess-exited-with-error
15.73
15.73 × Getting requirements to build wheel did not run successfully.
15.73 │ exit code: 1
15.73 ╰─> [3 lines of output]
15.73:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg\_resources.html
15.73 INFO:root:GDAL API version obtained from gdal-config: 3.0.4
15.73 ERROR: GDAL >= 3.1 is required for rasterio. Please upgrade GDAL.
15.73 [end of output]`
ENV PLATFORM linux/arm64`