heroku / heroku-buildpack-python

Heroku's buildpack for Python applications.
https://www.heroku.com/python
MIT License
974 stars 1.84k forks source link

Django does find GEOS libraries on Heroku-16 #398

Closed codingjoe closed 6 years ago

codingjoe commented 7 years ago

I'm getting an ImportError from Django when switching to the heroku-16 stack.

File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/gis/geos/libgeos.py", line 57, in load_geos
'", "'.join(lib_names)
ImportError: Could not find the GEOS library (tried "geos_c", "GEOS"). Try setting GEOS_LIBRARY_PATH in your settings.

After some digging, I don't see the difference. The libraries are installed and the LD_LIBRARY_PATH is still the same as it used to be on cedar-14.

I could fix the issue for me by setting GEOS_LIBRARY_PATH explicitly in the post_compile hook.

bin/post_compile

#!/usr/bin/env bash

set -e

export GEOS_LIBRARY_PATH="$BUILD_DIR/.heroku/vendor/lib"
kennethreitz commented 7 years ago

@codingjoe did setting GOES_LIBRARY_PATH fix the issue for you?

codingjoe commented 7 years ago

@kennethreitz nope, when connecting to a run dyno yes. But post_compile is than only available during build, not during runtime. I'm investigating...

codingjoe commented 7 years ago

Scratch that, it did not work. Hm... strange....

edmorley commented 7 years ago

Reading the Django source, I believe GOES_LIBRARY_PATH needs to be set in Django settings rather than directly in the environment (though likely worth setting that via os.environ / the django-environ package): https://github.com/django/django/blob/1.11.2/django/contrib/gis/geos/libgeos.py

Strange that it's still not working out of the box however.

Fwiw I tried comparing the directory structure of the cedar-14 and heroku-16 geos archives (eg https://lang-python.s3.amazonaws.com/heroku-16/libraries/vendor/geos.tar.gz), and they are identical - so the difference isn't say an inadvertent change in location.

kennethreitz commented 7 years ago

I believe the GEOS libraries aren't functional on heroku-16, even though they're identical copies to the working ones on cedar-14. The binary needs to be rebuilt for heroku-16, but i'm having trouble doing so. Might take another approach, possibly (adding it to the build stack image).

codingjoe commented 7 years ago

@kennethreitz yes, that would be cool. I'd love to switch to heroku-16

kennethreitz commented 7 years ago

I think I figured out a way to run the build. Executing now. Will post an update if it succeeds.

maiksprenger commented 7 years ago

Hi @kennethreitz, did you have a chance to investigate this further? If I'm not mistaken, this is an issue for all GeoDjango installations on Heroku. I'm happy to have a look - could you share your progress so far?

kennethreitz commented 7 years ago

This should be fixed shortly, as soon as the build is complete!

kennethreitz commented 7 years ago

Build complete!

kennethreitz commented 7 years ago

✨🍰✨

maiksprenger commented 7 years ago

Good stuff, thanks Kenneth!

harelw commented 6 years ago

To get GeoDjango working I also had to set the following Config Variables:

GDAL_LIBRARY_PATH=/app/.heroku/vendor/lib/libgdal.so
GEOS_LIBRARY_PATH=/app/.heroku/vendor/lib/libgeos_c.so

Shouldn't the buildpack take care of this when BUILD_WITH_GEO_LIBRARIES=1 is set?

kennethreitz commented 6 years ago

@harelw on which stack?

harelw commented 6 years ago

heroku-16

kennethreitz commented 6 years ago

I'll have to experiment with that myself. If true, the buildpack can definitely do that for you.

codingjoe commented 6 years ago

@kennethreitz I just checked, it seems the binaries on your bucked have not be updated. This is odd, since I updated all packages in my last 3 pull-requests.

root@8bc20bf16f82:/app# ls -lah /app/.heroku/vendor/lib/
total 85M
drwx------ 4 44182 44182 4.0K Sep 30 14:37 .
drwxr-xr-x 6 root  root  4.0K Sep 30 14:37 ..
drwxr-xr-x 2  7527  7527 4.0K Sep  8 19:26 gdalplugins
-rwxr-xr-x 1  7527  7527 1.1K Sep  8 19:26 libgdal.la
lrwxrwxrwx 1  7527  7527   17 Sep  8 19:26 libgdal.so -> libgdal.so.1.18.5
lrwxrwxrwx 1  7527  7527   17 Sep  8 19:26 libgdal.so.1 -> libgdal.so.1.18.5
-rwxr-xr-x 1  7527  7527  62M Sep  8 19:26 libgdal.so.1.18.5
-rwxr-xr-x 1 28988 28988  20M Sep  8 19:17 libgeos-3.4.3.so
-rwxr-xr-x 1 28988 28988  941 Sep  8 19:17 libgeos.la
lrwxrwxrwx 1 28988 28988   16 Sep  8 19:17 libgeos.so -> libgeos-3.4.3.so
-rwxr-xr-x 1 28988 28988  980 Sep  8 19:17 libgeos_c.la
lrwxrwxrwx 1 28988 28988   18 Sep  8 19:17 libgeos_c.so -> libgeos_c.so.1.8.3
lrwxrwxrwx 1 28988 28988   18 Sep  8 19:17 libgeos_c.so.1 -> libgeos_c.so.1.8.3
-rwxr-xr-x 1 28988 28988 1.1M Sep  8 19:17 libgeos_c.so.1.8.3
-rw-r--r-- 1 root  root  311K Sep 30 14:37 libjasper.so
-rw-r--r-- 1 root  root  311K Sep 30 14:37 libjasper.so.1
-rw-r--r-- 1 root  root  311K Sep 30 14:37 libjasper.so.1.0.0
-rwxr-xr-x 1 44182 44182  919 Sep  8 19:15 libproj.la
lrwxrwxrwx 1 44182 44182   16 Sep  8 19:15 libproj.so -> libproj.so.0.7.0
lrwxrwxrwx 1 44182 44182   16 Sep  8 19:15 libproj.so.0 -> libproj.so.0.7.0
-rwxr-xr-x 1 44182 44182 1.3M Sep  8 19:15 libproj.so.0.7.0
drwx------ 2 44182 44182 4.0K Sep  8 19:15 pkgconfig

I ran the build pack in a docker container to debug it, but you can also just download the binaries from your bucket:

curl -O https://lang-python.s3.amazonaws.com/heroku-16/libraries/vendor/gdal.tar.gz
codingjoe commented 6 years ago

Ok, when I set the pathes manually in my Django application it works:

GDAL_LIBRARY_PATH = "/app/.heroku/vendor/lib/libgdal.so"
GEOS_LIBRARY_PATH = "/app/.heroku/vendor/lib/libgeos_c.so"

I will try what happens if I set the LD_LIBRARY_PATH to /app/.heroku/vendor/lib

codingjoe commented 6 years ago

Ok, I opened a pull-request. It should fix the issue for the latest Django releases. The problem with the wrong binaries still remains. I guess you guys should double check your deployment pipeline for those binaries.

harelw commented 6 years ago

Still seems to fail if I remove GDAL_LIBRARY_PATH=/app/.heroku/vendor/lib/libgdal.so from config vars:

Traceback (most recent call last): File "manage.py", line 23, in <module> execute_from_command_line(sys.argv) File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line utility.execute() File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 338, in execute django.setup() File "/app/.heroku/python/lib/python3.6/site-packages/django/__init__.py", line 27, in setup apps.populate(settings.INSTALLED_APPS) File "/app/.heroku/python/lib/python3.6/site-packages/django/apps/registry.py", line 85, in populate app_config = AppConfig.create(entry) File "/app/.heroku/python/lib/python3.6/site-packages/django/apps/config.py", line 94, in create module = import_module(entry) File "/app/.heroku/python/lib/python3.6/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 978, in _gcd_import File "<frozen importlib._bootstrap>", line 961, in _find_and_load File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 655, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 678, in exec_module File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed File "/app/.heroku/python/lib/python3.6/site-packages/mapwidgets/__init__.py", line 4, in <module> from .widgets import GooglePointFieldWidget, GooglePointFieldInlineWidget, \ File "/app/.heroku/python/lib/python3.6/site-packages/mapwidgets/widgets.py", line 4, in <module> from django.contrib.gis.forms import BaseGeometryWidget File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/gis/forms/__init__.py", line 3, in <module> from .fields import ( # NOQA File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/gis/forms/fields.py", line 4, in <module> from django.contrib.gis.geos import GEOSException, GEOSGeometry File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/gis/geos/__init__.py", line 5, in <module> from .collections import ( # NOQA File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/gis/geos/collections.py", line 11, in <module> from django.contrib.gis.geos.geometry import GEOSGeometry, LinearGeometryMixin File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/gis/geos/geometry.py", line 11, in <module> from django.contrib.gis import gdal File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/gis/gdal/__init__.py", line 28, in <module> from django.contrib.gis.gdal.datasource import DataSource File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/gis/gdal/datasource.py", line 39, in <module> from django.contrib.gis.gdal.driver import Driver File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/gis/gdal/driver.py", line 5, in <module> from django.contrib.gis.gdal.prototypes import ds as vcapi, raster as rcapi File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/gis/gdal/prototypes/ds.py", line 9, in <module> from django.contrib.gis.gdal.libgdal import GDAL_VERSION, lgdal File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/gis/gdal/libgdal.py", line 45, in <module> % '", "'.join(lib_names) django.core.exceptions.ImproperlyConfigured: Could not find the GDAL library (tried "gdal", "GDAL", "gdal2.1.0", "gdal2.0.0", "gdal1.11.0", "gdal1.10.0", "gdal1.9.0"). Is GDAL installed? If it is, try setting GDAL_LIBRARY_PATH in your settings.

codingjoe commented 6 years ago

Yes, same here, I set it manually too.

codingjoe commented 6 years ago

Wait @harelw actually this should have fixed it: #478

codingjoe commented 6 years ago

Haven't tried it yet, since it was just merged ;)

harelw commented 6 years ago

Yea I mentioned it because I am pretty sure I tried it after the merge.

goinnn commented 6 years ago

I fixed it today with this changes:

changes

codingjoe commented 5 years ago

So I further drilled own into the issue and found that the Django problem is here: https://github.com/heroku/stack-images/issues/118