search

heroku / heroku-geo-buildpack

Classic buildpack that installs the Geo/GIS libraries GDAL, GEOS and PROJ.
38 stars 28 forks source link

Pip installation of GDAL fails with GCC undeclared identifer errors #20

Closed edmorley closed 2 years ago

edmorley commented 3 years ago

STR:

$ mkdir -p testapp-geo && cd $_ && git init && h create
$ h buildpacks:add https://github.com/heroku/heroku-geo-buildpack
$ h buildpacks:add heroku/python
$ echo 'GDAL' > requirements.txt
$ git add -A && git commit -m '.' && git push heroku main

This uses buildpack default Python (currently 3.6.12), default stack (currently Heroku-20), and latest GDAL (currently 3.2.1).

Expected

The pip install of the [GDAL]() package succeeds.

Actual

The pip install fails during the GCC compilation step, with 12 thousand lines of GCC output, the first/last parts of which are below. Example error:

remote:          extensions/gdalconst_wrap.c: In function ‘PyInit__gdalconst’:
remote:          extensions/gdalconst_wrap.c:3879:71: error: ‘GDAL_OF_MULTIDIM_RASTER’ undeclared (first use in this function)
remote:           3879 |   SWIG_Python_SetConstant(d, "OF_MULTIDIM_RASTER",SWIG_From_int((int)(GDAL_OF_MULTIDIM_RASTER)));
remote:                |                                                                       ^~~~~~~~~~~~~~~~~~~~~~~

The GDAL package has had a few releases recently: https://pypi.org/project/GDAL/#history https://github.com/OSGeo/gdal/blob/v3.2.1/gdal/NEWS

However trying older releases (eg GDAL==3.1.3) didn't help.

Build log:

$ mkdir -p testapp-geo && cd $_ && git init && h create
$ h buildpacks:add https://github.com/heroku/heroku-geo-buildpack
$ h buildpacks:add heroku/python
$ echo 'GDAL' > requirements.txt
$ git add -A && git commit -m '.' && git push heroku main
...
remote: -----> Building on the Heroku-20 stack
remote: -----> Geo Packages (GDAL/GEOS/PROJ) app detected
remote: -----> Installing GDAL-2.4.0
remote: -----> Installing GEOS-3.7.2
remote: -----> Installing PROJ-5.2.0
remote: -----> Python app detected
remote: -----> Installing python-3.6.12
remote: -----> Installing pip 20.1.1, setuptools 47.1.1 and wheel 0.34.2
remote: -----> Installing SQLite3
remote: -----> Installing requirements with pip
remote:        Collecting GDAL
remote:          Downloading GDAL-3.2.1.tar.gz (604 kB)
remote:        Building wheels for collected packages: GDAL
remote:          Building wheel for GDAL (setup.py): started
remote:          Building wheel for GDAL (setup.py): finished with status 'error'
remote:          ERROR: Command errored out with exit status 1:
remote:           command: /app/.heroku/python/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_ae9v2xt/GDAL/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_ae9v2xt/GDAL/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-rmcrkk4o
remote:               cwd: /tmp/pip-install-_ae9v2xt/GDAL/
remote:          Complete output (6667 lines):
remote:          WARNING: numpy not available!  Array support will not be enabled
remote:          running bdist_wheel
remote:          running build
remote:          running build_py
remote:          creating build
remote:          creating build/lib.linux-x86_64-3.6
remote:          creating build/lib.linux-x86_64-3.6/osgeo
remote:          copying osgeo/ogr.py -> build/lib.linux-x86_64-3.6/osgeo
remote:          copying osgeo/__init__.py -> build/lib.linux-x86_64-3.6/osgeo
remote:          copying osgeo/gnm.py -> build/lib.linux-x86_64-3.6/osgeo
remote:          copying osgeo/gdal_array.py -> build/lib.linux-x86_64-3.6/osgeo
remote:          copying osgeo/gdalnumeric.py -> build/lib.linux-x86_64-3.6/osgeo
remote:          copying osgeo/osr.py -> build/lib.linux-x86_64-3.6/osgeo
remote:          copying osgeo/gdal.py -> build/lib.linux-x86_64-3.6/osgeo
remote:          copying osgeo/gdalconst.py -> build/lib.linux-x86_64-3.6/osgeo
remote:          creating build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/__init__.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/gdalchksum.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/gdal_pansharpen.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/gdalident.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/epsg_tr.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/gdal_polygonize.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/esri2wkt.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/gdal2xyz.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/gcps2wld.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/gdal_fillnodata.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/rgb2pct.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/gcps2vec.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/gdal_proximity.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/gdalimport.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/gdal_auth.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/gdalmove.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/gdal_edit.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/pct2rgb.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/gdalcompare.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/mkgraticule.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/gdal_retile.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/gdal_merge.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/ogrmerge.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/gdal_calc.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          copying osgeo/utils/gdal_sieve.py -> build/lib.linux-x86_64-3.6/osgeo/utils
remote:          /app/.heroku/python/lib/python3.6/site-packages/setuptools/lib2to3_ex.py:44: SetuptoolsDeprecationWarning: 2to3 support is deprecated. If the project still requires Python 2 support, please migrate to a single-codebase solution or employ an independent conversion process.
remote:            SetuptoolsDeprecationWarning)
remote:          Fixing build/lib.linux-x86_64-3.6/osgeo/ogr.py build/lib.linux-x86_64-3.6/osgeo/__init__.py build/lib.linux-x86_64-3.6/osgeo/gnm.py build/lib.linux-x86_64-3.6/osgeo/gdal_array.py build/lib.linux-x86_64-3.6/osgeo/gdalnumeric.py build/lib.linux-x86_64-3.6/osgeo/osr.py build/lib.linux-x86_64-3.6/osgeo/gdal.py build/lib.linux-x86_64-3.6/osgeo/gdalconst.py build/lib.linux-x86_64-3.6/osgeo/utils/__init__.py build/lib.linux-x86_64-3.6/osgeo/utils/gdalchksum.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal_pansharpen.py build/lib.linux-x86_64-3.6/osgeo/utils/gdalident.py build/lib.linux-x86_64-3.6/osgeo/utils/epsg_tr.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal_polygonize.py build/lib.linux-x86_64-3.6/osgeo/utils/esri2wkt.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal2xyz.py build/lib.linux-x86_64-3.6/osgeo/utils/gcps2wld.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal_fillnodata.py build/lib.linux-x86_64-3.6/osgeo/utils/rgb2pct.py build/lib.linux-x86_64-3.6/osgeo/utils/gcps2vec.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal_proximity.py build/lib.linux-x86_64-3.6/osgeo/utils/gdalimport.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal_auth.py build/lib.linux-x86_64-3.6/osgeo/utils/gdalmove.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal_edit.py build/lib.linux-x86_64-3.6/osgeo/utils/pct2rgb.py build/lib.linux-x86_64-3.6/osgeo/utils/gdalcompare.py build/lib.linux-x86_64-3.6/osgeo/utils/mkgraticule.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal_retile.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal_merge.py build/lib.linux-x86_64-3.6/osgeo/utils/ogrmerge.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal_calc.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal_sieve.py
remote:          Skipping optional fixer: ws_comma
remote:          Fixing build/lib.linux-x86_64-3.6/osgeo/ogr.py build/lib.linux-x86_64-3.6/osgeo/__init__.py build/lib.linux-x86_64-3.6/osgeo/gnm.py build/lib.linux-x86_64-3.6/osgeo/gdal_array.py build/lib.linux-x86_64-3.6/osgeo/gdalnumeric.py build/lib.linux-x86_64-3.6/osgeo/osr.py build/lib.linux-x86_64-3.6/osgeo/gdal.py build/lib.linux-x86_64-3.6/osgeo/gdalconst.py build/lib.linux-x86_64-3.6/osgeo/utils/__init__.py build/lib.linux-x86_64-3.6/osgeo/utils/gdalchksum.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal_pansharpen.py build/lib.linux-x86_64-3.6/osgeo/utils/gdalident.py build/lib.linux-x86_64-3.6/osgeo/utils/epsg_tr.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal_polygonize.py build/lib.linux-x86_64-3.6/osgeo/utils/esri2wkt.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal2xyz.py build/lib.linux-x86_64-3.6/osgeo/utils/gcps2wld.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal_fillnodata.py build/lib.linux-x86_64-3.6/osgeo/utils/rgb2pct.py build/lib.linux-x86_64-3.6/osgeo/utils/gcps2vec.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal_proximity.py build/lib.linux-x86_64-3.6/osgeo/utils/gdalimport.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal_auth.py build/lib.linux-x86_64-3.6/osgeo/utils/gdalmove.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal_edit.py build/lib.linux-x86_64-3.6/osgeo/utils/pct2rgb.py build/lib.linux-x86_64-3.6/osgeo/utils/gdalcompare.py build/lib.linux-x86_64-3.6/osgeo/utils/mkgraticule.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal_retile.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal_merge.py build/lib.linux-x86_64-3.6/osgeo/utils/ogrmerge.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal_calc.py build/lib.linux-x86_64-3.6/osgeo/utils/gdal_sieve.py
remote:          Skipping optional fixer: ws_comma
remote:          running build_ext
remote:          gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I../../port -I../../gcore -I../../alg -I../../ogr/ -I../../ogr/ogrsf_frmts -I../../gnm -I../../apps -I/app/.heroku/python/include/python3.6m -I. -I/tmp/tmp.s8g7jc8xnf/include -c gdal_python_cxx11_test.cpp -o gdal_python_cxx11_test.o
remote:          building 'osgeo._gdal' extension
remote:          creating build/temp.linux-x86_64-3.6
remote:          building 'osgeo._gdalconst' extension
remote:          creating build/temp.linux-x86_64-3.6/extensions
remote:          creating build/temp.linux-x86_64-3.6/extensions
remote:          gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I../../port -I../../gcore -I../../alg -I../../ogr/ -I../../ogr/ogrsf_frmts -I../../gnm -I../../apps -I/app/.heroku/python/include/python3.6m -I. -I/tmp/tmp.s8g7jc8xnf/include -c extensions/gdal_wrap.cpp -o build/temp.linux-x86_64-3.6/extensions/gdal_wrap.o -I/tmp/tmp.s8g7jc8xnf/include
remote:          gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I../../port -I../../gcore -I../../alg -I../../ogr/ -I../../ogr/ogrsf_frmts -I../../gnm -I../../apps -I/app/.heroku/python/include/python3.6m -I. -I/tmp/tmp.s8g7jc8xnf/include -c extensions/gdalconst_wrap.c -o build/temp.linux-x86_64-3.6/extensions/gdalconst_wrap.o -I/tmp/tmp.s8g7jc8xnf/include
remote:          extensions/gdalconst_wrap.c: In function ‘PyInit__gdalconst’:
remote:          extensions/gdalconst_wrap.c:3879:71: error: ‘GDAL_OF_MULTIDIM_RASTER’ undeclared (first use in this function)
remote:           3879 |   SWIG_Python_SetConstant(d, "OF_MULTIDIM_RASTER",SWIG_From_int((int)(GDAL_OF_MULTIDIM_RASTER)));
remote:                |                                                                       ^~~~~~~~~~~~~~~~~~~~~~~
remote:          extensions/gdalconst_wrap.c:3879:71: note: each undeclared identifier is reported only once for each function it appears in
remote:          extensions/gdalconst_wrap.c:3903:68: error: ‘GDAL_DCAP_UNIQUE_FIELDS’ undeclared (first use in this function); did you mean ‘GDAL_DCAP_DEFAULT_FIELDS’?
remote:           3903 |   SWIG_Python_SetConstant(d, "DCAP_UNIQUE_FIELDS",SWIG_FromCharPtr(GDAL_DCAP_UNIQUE_FIELDS));
remote:                |                                                                    ^~~~~~~~~~~~~~~~~~~~~~~
remote:                |                                                                    GDAL_DCAP_DEFAULT_FIELDS
...
<several thousand lines removed>
...
remote:            extensions/gdal_wrap.cpp:44258:100: error: ‘GVOT_MIN_TARGET_HEIGHT_FROM_GROUND’ was not declared in this scope
remote:            44258 |   SWIG_Python_SetConstant(d, "GVOT_MIN_TARGET_HEIGHT_FROM_GROUND",SWIG_From_int(static_cast< int >(GVOT_MIN_TARGET_HEIGHT_FROM_GROUND)));
remote:                  |                                                                                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
remote:            extensions/gdal_wrap.cpp: At global scope:
remote:            extensions/gdal_wrap.cpp:5542:15: warning: ‘MDArrayReadWriteCheckArguments’ defined but not used [-Wunused-variable]
remote:             5542 | static CPLErr MDArrayReadWriteCheckArguments(GDALMDArrayHS* array,
remote:                  |               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
remote:            extensions/gdal_wrap.cpp:5519:13: warning: ‘CheckNumericDataType’ defined but not used [-Wunused-variable]
remote:             5519 | static bool CheckNumericDataType(GDALExtendedDataTypeHS* dt)
remote:                  |             ^~~~~~~~~~~~~~~~~~~~
remote:            g++ -pthread -shared build/temp.linux-x86_64-3.6/extensions/gnm_wrap.o -L../../.libs -L../../ -L/tmp/tmp.s8g7jc8xnf/lib -lgdal -o build/lib.linux-x86_64-3.6/osgeo/_gnm.cpython-36m-x86_64-linux-gnu.so
remote:            error: command 'gcc' failed with exit status 1
remote:            ----------------------------------------
remote:        ERROR: Command errored out with exit status 1: /app/.heroku/python/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_ae9v2xt/GDAL/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_ae9v2xt/GDAL/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-3fwdlt_2/install-record.txt --single-version-externally-managed --compile --install-headers /app/.heroku/python/include/python3.6m/GDAL Check the logs for full command output.
remote:  !     Push rejected, failed to compile Python app.
edmorley commented 3 years ago

The build succeeds if I use GDAL < v3, eg GDAL==2.4.4.

So I'm presuming this is a combination of:

  1. This buildpack defaults to GDAL libs v2, which aren't compatible with GDAL Python bindings 3+ (would be good to mention the incompatibility in the README)
  2. GDAL Python bindings 3+ don't check the version of GDAL libs before building, and so display thousands of errors, instead of a useful error message