search

developmentseed / geolambda

Create and deploy Geospatial AWS Lambda functions
MIT License
303 stars 85 forks source link

Installation instruction fails on GDAL installation. #81

Closed saheelBreezo closed 4 years ago

saheelBreezo commented 4 years ago

Summary of the bug/issue.

Python GeoLambda Readme Installation fails while installing GDAL. The GDAL version it is trying to install is 3.0.1

Steps to reproduce the bug.

Following the installation instructions from the README. VERSION=1.2.0 docker build . --build-arg VERSION=${VERSION} --build-arg PYVERSION=3.6.8 -t geo-lambda:latest

Buggy behaviour and/or error message

Sending build context to Docker daemon  15.87kB
Step 1/10 : ARG VERSION=1.2.0
Step 2/10 : FROM developmentseed/geolambda:${VERSION}
 ---> 7e5ae746c387
Step 3/10 : LABEL maintainer="Development Seed <info@developmentseed.org>"
 ---> Using cache
 ---> d561fa6c0519
Step 4/10 : LABEL authors="Matthew Hanson  <matt.a.hanson@gmail.com>"
 ---> Using cache
 ---> c140aea93a85
Step 5/10 : ARG PYVERSION=3.7.4
 ---> Using cache
 ---> e1feebc7d86d
Step 6/10 : ENV     PYENV_ROOT=/root/.pyenv     PATH=/root/.pyenv/shims:/root/.pyenv/bin:$PATH
 ---> Using cache
 ---> 97846c82a747
Step 7/10 : RUN     curl https://pyenv.run | bash;     CONFIGURE_OPTS="--with-openssl=${PREFIX}/openssl"         LD_RUN_PATH="${PREFIX}/openssl/lib"         pyenv install ${PYVERSION};     pyenv global ${PYVERSION};     pip install --upgrade pip
 ---> Using cache
 ---> 1fa92623f65f
Step 8/10 : COPY requirements*.txt ./
 ---> cc3f7cd7092e
Step 9/10 : RUN     pip install -r requirements-pre.txt;     pip install -r requirements.txt
 ---> Running in 7476483364fa
Collecting numpy
  Downloading https://files.pythonhosted.org/packages/d2/ab/43e678759326f728de861edbef34b8e2ad1b1490505f20e0d1f0716c3bf4/numpy-1.17.4-cp36-cp36m-manylinux1_x86_64.whl (20.0MB)
Installing collected packages: numpy
Successfully installed numpy-1.17.4
Collecting GDAL==3.0.1
  Downloading https://files.pythonhosted.org/packages/26/c6/f8846ba726b37f9a1099acb5bff43c309fb913577674c1ba491f6fc60598/GDAL-3.0.1.tar.gz (576kB)
Collecting rasterio==1.1.0
  Downloading https://files.pythonhosted.org/packages/1a/6d/ee8b15e0d45dd7e9e7f2559e37f91915a19155cf8403948ff2082903ee30/rasterio-1.1.0.tar.gz (2.1MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Collecting shapely==1.6.4.post2
  Downloading https://files.pythonhosted.org/packages/38/b6/b53f19062afd49bb5abd049aeed36f13bf8d57ef8f3fa07a5203531a0252/Shapely-1.6.4.post2-cp36-cp36m-manylinux1_x86_64.whl (1.5MB)
Collecting pyproj==2.4.0
  Downloading https://files.pythonhosted.org/packages/7a/b1/ab67ad924770e1c1432fa0953a665b8ea193b60c7494457b69da052d6e83/pyproj-2.4.0-cp36-cp36m-manylinux1_x86_64.whl (10.1MB)
Collecting cligj>=0.5
  Downloading https://files.pythonhosted.org/packages/e4/be/30a58b4b0733850280d01f8bd132591b4668ed5c7046761098d665ac2174/cligj-0.5.0-py3-none-any.whl
Requirement already satisfied: numpy in /root/.pyenv/versions/3.6.8/lib/python3.6/site-packages (from rasterio==1.1.0->-r requirements.txt (line 2)) (1.17.4)
Collecting snuggs>=1.4.1
  Downloading https://files.pythonhosted.org/packages/cc/0e/d27d6e806d6c0d1a2cfdc5d1f088e42339a0a54a09c3343f7f81ec8947ea/snuggs-1.4.7-py3-none-any.whl
Collecting attrs
  Downloading https://files.pythonhosted.org/packages/a2/db/4313ab3be961f7a763066401fb77f7748373b6094076ae2bda2806988af6/attrs-19.3.0-py2.py3-none-any.whl
Collecting click-plugins
  Downloading https://files.pythonhosted.org/packages/e9/da/824b92d9942f4e472702488857914bdd50f73021efea15b4cad9aca8ecef/click_plugins-1.1.1-py2.py3-none-any.whl
Collecting click<8,>=4.0
  Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)
Collecting affine
  Downloading https://files.pythonhosted.org/packages/ac/a6/1a39a1ede71210e3ddaf623982b06ecfc5c5c03741ae659073159184cd3e/affine-2.3.0-py2.py3-none-any.whl
Collecting pyparsing>=2.1.6
  Downloading https://files.pythonhosted.org/packages/c0/0c/fc2e007d9a992d997f04a80125b0f183da7fb554f1de701bbb70a8e7d479/pyparsing-2.4.5-py2.py3-none-any.whl (67kB)
Building wheels for collected packages: rasterio
  Building wheel for rasterio (PEP 517): started
  Building wheel for rasterio (PEP 517): still running...
  Building wheel for rasterio (PEP 517): finished with status 'done'
  Created wheel for rasterio: filename=rasterio-1.1.0-cp36-cp36m-linux_x86_64.whl size=4943674 sha256=21999fcf1f9bd1f1f0d6d4c82925a52b9f4630539de92409f813320c08f83e83
  Stored in directory: /root/.cache/pip/wheels/6c/44/28/18fca8582827f0a8594284474ad863e806ceb28d9fbcc948c4
Successfully built rasterio
Installing collected packages: GDAL, click, cligj, pyparsing, snuggs, attrs, click-plugins, affine, rasterio, shapely, pyproj
    Running setup.py install for GDAL: started
    Running setup.py install for GDAL: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /root/.pyenv/versions/3.6.8/bin/python3.6 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-j2lowkfe/GDAL/setup.py'"'"'; __file__='"'"'/tmp/pip-install-j2lowkfe/GDAL/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-w4f0b8w0/install-record.txt --single-version-externally-managed --compile
         cwd: /tmp/pip-install-j2lowkfe/GDAL/
    Complete output (50 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    copying gdal.py -> build/lib.linux-x86_64-3.6
    copying ogr.py -> build/lib.linux-x86_64-3.6
    copying osr.py -> build/lib.linux-x86_64-3.6
    copying gdalconst.py -> build/lib.linux-x86_64-3.6
    copying gdalnumeric.py -> build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/osgeo
    copying osgeo/ogr.py -> build/lib.linux-x86_64-3.6/osgeo
    copying osgeo/__init__.py -> build/lib.linux-x86_64-3.6/osgeo
    copying osgeo/gnm.py -> build/lib.linux-x86_64-3.6/osgeo
    copying osgeo/gdal.py -> build/lib.linux-x86_64-3.6/osgeo
    copying osgeo/osr.py -> build/lib.linux-x86_64-3.6/osgeo
    copying osgeo/gdal_array.py -> build/lib.linux-x86_64-3.6/osgeo
    copying osgeo/gdalnumeric.py -> build/lib.linux-x86_64-3.6/osgeo
    copying osgeo/gdalconst.py -> build/lib.linux-x86_64-3.6/osgeo
    Fixing build/lib.linux-x86_64-3.6/gdal.py build/lib.linux-x86_64-3.6/ogr.py build/lib.linux-x86_64-3.6/osr.py build/lib.linux-x86_64-3.6/gdalconst.py build/lib.linux-x86_64-3.6/gdalnumeric.py build/lib.linux-x86_64-3.6/osgeo/ogr.py build/lib.linux-x86_64-3.6/osgeo/__init__.py build/lib.linux-x86_64-3.6/osgeo/gnm.py build/lib.linux-x86_64-3.6/osgeo/gdal.py build/lib.linux-x86_64-3.6/osgeo/osr.py build/lib.linux-x86_64-3.6/osgeo/gdal_array.py build/lib.linux-x86_64-3.6/osgeo/gdalnumeric.py build/lib.linux-x86_64-3.6/osgeo/gdalconst.py
    Skipping optional fixer: ws_comma
    Fixing build/lib.linux-x86_64-3.6/gdal.py build/lib.linux-x86_64-3.6/ogr.py build/lib.linux-x86_64-3.6/osr.py build/lib.linux-x86_64-3.6/gdalconst.py build/lib.linux-x86_64-3.6/gdalnumeric.py build/lib.linux-x86_64-3.6/osgeo/ogr.py build/lib.linux-x86_64-3.6/osgeo/__init__.py build/lib.linux-x86_64-3.6/osgeo/gnm.py build/lib.linux-x86_64-3.6/osgeo/gdal.py build/lib.linux-x86_64-3.6/osgeo/osr.py build/lib.linux-x86_64-3.6/osgeo/gdal_array.py build/lib.linux-x86_64-3.6/osgeo/gdalnumeric.py build/lib.linux-x86_64-3.6/osgeo/gdalconst.py
    Skipping optional fixer: ws_comma
    running build_ext
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I../../port -I../../gcore -I../../alg -I../../ogr/ -I../../ogr/ogrsf_frmts -I../../gnm -I../../apps -I/root/.pyenv/versions/3.6.8/include/python3.6m -I/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/numpy/core/include -I/usr/local/include -c gdal_python_cxx11_test.cpp -o gdal_python_cxx11_test.o
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I../../port -I../../gcore -I../../alg -I../../ogr/ -I../../ogr/ogrsf_frmts -I../../gnm -I../../apps -I/root/.pyenv/versions/3.6.8/include/python3.6m -I/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/numpy/core/include -I/usr/local/include -c gdal_python_cxx11_test.cpp -o gdal_python_cxx11_test.o -std=c++11
    building 'osgeo._gdal' extension
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/extensions
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I../../port -I../../gcore -I../../alg -I../../ogr/ -I../../ogr/ogrsf_frmts -I../../gnm -I../../apps -I/root/.pyenv/versions/3.6.8/include/python3.6m -I/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/numpy/core/include -I/usr/local/include -c extensions/gdal_wrap.cpp -o build/temp.linux-x86_64-3.6/extensions/gdal_wrap.o -std=c++11 -I/usr/local/include
    extensions/gdal_wrap.cpp: In function ‘OSRSpatialReferenceShadow* GDALDatasetShadow_GetSpatialRef(GDALDatasetShadow*)’:
    extensions/gdal_wrap.cpp:4672:54: error: ‘GDALGetSpatialRef’ was not declared in this scope
         OGRSpatialReferenceH ref = GDALGetSpatialRef(self);
                                                          ^
    extensions/gdal_wrap.cpp: In function ‘void GDALDatasetShadow_SetSpatialRef(GDALDatasetShadow*, OSRSpatialReferenceShadow*)’:
    extensions/gdal_wrap.cpp:4681:57: error: ‘GDALSetSpatialRef’ was not declared in this scope
          GDALSetSpatialRef( self, (OGRSpatialReferenceH)srs );
                                                             ^
    extensions/gdal_wrap.cpp: In function ‘OSRSpatialReferenceShadow* GDALDatasetShadow_GetGCPSpatialRef(GDALDatasetShadow*)’:
    extensions/gdal_wrap.cpp:4722:57: error: ‘GDALGetGCPSpatialRef’ was not declared in this scope
         OGRSpatialReferenceH ref = GDALGetGCPSpatialRef(self);
                                                             ^
    extensions/gdal_wrap.cpp: In function ‘CPLErr GDALDatasetShadow_SetGCPs2(GDALDatasetShadow*, int, const GDAL_GCP*, OSRSpatialReferenceShadow*)’:
    extensions/gdal_wrap.cpp:4735:73: error: ‘GDALSetGCPs2’ was not declared in this scope
         return GDALSetGCPs2( self, nGCPs, pGCPs, (OGRSpatialReferenceH)hSRS );
                                                                             ^
    extensions/gdal_wrap.cpp:4736:3: warning: control reaches end of non-void function [-Wreturn-type]
       }
       ^
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /root/.pyenv/versions/3.6.8/bin/python3.6 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-j2lowkfe/GDAL/setup.py'"'"'; __file__='"'"'/tmp/pip-install-j2lowkfe/GDAL/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-w4f0b8w0/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.
The command '/bin/sh -c pip install -r requirements-pre.txt;     pip install -r requirements.txt' returned a non-zero code: 1

Environment Information

matthewhanson commented 4 years ago

Hi @saheelBreezo ,

There seems to be an error in the README - you should set the VERSION=2.0.0, rather than 1.2.0.

It's trying to use the 1.2.0 version of the GeoLambda base layer (which is GDAL 2.4.1) but then trying to install GDAL bindings for GDAL 3.0.1.

Let me know if that works!

saheelBreezo commented 4 years ago

Hey @matthewhanson , Thanks a lot it worked. Also noticed that the Dockerfile also has the default set to 1.2.0

Successfully installed GDAL-3.0.1 affine-2.3.0 attrs-19.3.0 click-7.0 click-plugins-1.1.1 cligj-0.5.0 pyparsing-2.4.5 pyproj-2.4.0 rasterio-1.1.0 shapely-1.6.4.post2 snuggs-1.4.7
Removing intermediate container 724568cf36d1
 ---> f900455310f6
Step 10/10 : COPY bin/* /usr/local/bin/
 ---> a55d2cf7c811
Successfully built a55d2cf7c811
Successfully tagged geo-lambda:latest
matthewhanson commented 4 years ago

Thanks @saheelBreezo for pointing this out, I've fixed this on develop with https://github.com/developmentseed/geolambda/commit/deaab242ceb8587783722826714b31aea146e800

I'll publish a new version when there's some changes to package versions, or #79 is resolved.