Auto-detection of library locations not working

[edmorley]

With the legacy Python buildpack BUILD_WITH_GEO_LIBRARIES feature, the GDAL library that was vendored was auto-detectable by Django, without the need to set GDAL_LIBRARY_PATH in the Django settings file.

Django's auto-detection is here: https://github.com/django/django/blob/f63f3cdf0969c23fd0c05de0f4a2a1df0cd5112e/django/contrib/gis/gdal/libgdal.py#L13-L47

But in short it does the equivalent of:

from ctypes.util import find_library

find_library('gdal')

Trying this using the legacy feature works:

$ mkdir testapp-geo-legacy && cd $_ && git init && h create --stack heroku-18
$ h buildpacks:add https://github.com/heroku/heroku-buildpack-python#v184
$ h config:set BUILD_WITH_GEO_LIBRARIES=1
$ echo 'python-2.7.16' > runtime.txt
$ touch requirements.txt
$ git add -A; git commit -m '.' && git push heroku main
...
$ h run -- python -c 'from ctypes.util import find_library; print(find_library("gdal"))'
libgdal.so.20

However using this buildpack it doesn't:

$ mkdir testapp-geo-buildpack && cd $_ && git init && h create --stack heroku-18
$ h buildpacks:add https://github.com/heroku/heroku-geo-buildpack
$ h buildpacks:add heroku/python
$ echo 'python-2.7.16' > runtime.txt
$ touch requirements.txt
$ git add -A; git commit -m '.' && git push heroku main
...
$ h run -- python -c 'from ctypes.util import find_library; print(find_library("gdal"))'
None

Interestingly, if I change the Python version from 2.7 to 3.6, it does work.

# <continued from testapp-geo-buildpack example above>
$ echo 'python-3.6.12' > runtime.txt
$ git add -A; git commit -m '.' && git push heroku main
...
$ h run -- python -c 'from ctypes.util import find_library; print(find_library("gdal"))'
libgdal.so.20

Looking at https://docs.python.org/3/library/ctypes.html#finding-shared-libraries I see:

On Linux, find_library() tries to run external programs (/sbin/ldconfig, gcc, objdump and ld) to find the library file. It returns the filename of the library file.

Changed in version 3.6: On Linux, the value of the environment variable LD_LIBRARY_PATH is used when searching for libraries, if a library cannot be found by any other means.

[edmorley]

Ah with this change it then works on Python 2.7:

$ h run bash
Running bash on ⬢ dry-beyond-57098... up, run.8608 (Free)
~ $ python -c 'from ctypes.util import find_library; print(find_library("gdal"))'
None
~ $ echo $LIBRARY_PATH
/app/.heroku/vendor/lib:/app/.heroku/python/lib:
~ $ export LIBRARY_PATH="/app/.heroku/vendor/lib:/app/.heroku/python/lib:/app/.heroku-geo-buildpack/vendor/lib:"
~ $ python -c 'from ctypes.util import find_library; print(find_library("gdal"))'
libgdal.so.20

Looks like adding .heroku-geo-buildpack/vendor/lib to LIBRARY_PATH (and not just LD_LIBRARY_PATH) will fix this.

(Checking the legacy feature, it adds to both too.)