[Migrated] Getting GeoDjango Working

jneves commented 3 years ago

Originally from: https://github.com/Miserlou/Zappa/issues/985 by bahoo

Update September 2019: This issue + comment originally laid out a proposal to make project directory a configurable setting in Zappa, as a way of getting GeoDjango to work more seamlessly in Zappa across different projects. I've found a better solution to my original problem, laid out in a comment below, but I'm going to leave this issue as-is in case it offers guidance for someone else with a similar issue. Mods / maintainer folks, if you've got better solutions for next steps / what to do with this ticket, I'm all ears. 👍

A suggestion: make the project directory a configurable option within a stage.

Context

Following davepretty's comment on Slack for getting GeoDjango working, I hardcoded the rpath in one of my libraries I needed.

To make this approach more re-usable for future GeoDjango / Zappa projects, I thought that having the option for a consistent path for the project directory would be helpful. Then, future projects can use the same library files, specify the same path (something generic like /tmp/code/ in my case) and Bob's your uncle.

Currently, Zappa uses /tmp/{0!s}'.format(self.settings.PROJECT_NAME) which a good and fine default for most cases, but being able to use the same directory across different projects would be more portable.

Possible Fix

I've made this change in a branch, here: https://github.com/bahoo/Zappa/commits/specify_project_directory

Your Environment

Zappa version used: 0.42.0
Operating System and Python version: macOS Sierra, Python 2.7.12

The output of pip freeze:

appdirs==1.4.3
argcomplete==1.8.2
base58==0.2.4
boto3==1.4.4
botocore==1.5.40
click==6.7
dj-database-url==0.4.2
Django==1.10.6
django-bootstrap3==8.2.1
django-dotenv==1.4.1
docutils==0.13.1
durationpy==0.4
EasyProcess==0.2.3
future==0.16.0
futures==3.1.1
googlemaps==2.4.6
gunicorn==19.7.1
hjson==2.0.7
jmespath==0.9.3
kappa==0.6.0
lambda-packages==0.16.1
packaging==16.8
placebo==0.8.1
psycopg2==2.7.1
pyparsing==2.2.0
python-dateutil==2.6.0
python-slugify==1.2.4
pyunpack==0.1.2
PyYAML==3.12
requests==2.11.1
s3transfer==0.1.10
six==1.10.0
toml==0.9.2
tqdm==4.14.0
troposphere==1.9.4
Unidecode==0.4.21
Werkzeug==0.12
whitenoise==3.3.0
wsgi-request-logger==0.4.6
zappa==0.42.0

Your zappa_settings.py:

{
"production": {
    "aws_region": "us-west-2", 
    "django_settings": "culverstack.settings_zappa", 
    "profile_name": "default",
    "project_directory": "/tmp/code",
    "s3_bucket": "culverstack",
    "slim_handler": true,
    "remote_env": "[ ... snipped ... ]"
}
}

Alex-Mackie commented 3 years ago

Update for 2021. The neatest, lightest way I could find was to use these layers (these are minimalist and conveniently supplied in all aws regions which is not the case for geolambda).

Concretely:

zappa_settings.json

        "layers": ["arn:aws:lambda:eu-west-2:524387336408:layer:gdal32:4"],
       ...
        "environment_variables": {
            "GDAL_LIBRARY_PATH": "/opt/lib/libgdal.so.3.2.1",
            "GEOS_LIBRARY_PATH": "/opt/lib/libgeos_c.so.1.16.2",
            "GDAL_DATA": "/var/task/share/gdal" 
        },

settings.py:

    GDAL_LIBRARY_PATH = os.environ['GDAL_LIBRARY_PATH']
    GEOS_LIBRARY_PATH = os.environ['GEOS_LIBRARY_PATH']

To find where the library paths to put in zappa_settings.json actually are:

    # zappa invoke dev "import subprocess; print(subprocess.getoutput(['find / -name *libgdal*']))" --raw
    # zappa invoke dev "import subprocess; print(subprocess.getoutput(['find / -name *libgeos*']))" --raw

Django==3.2
zappa==0.52.0

pfcodes commented 1 year ago

Update for 2021. The neatest, lightest way I could find was to use these layers (these are minimalist and conveniently supplied in all aws regions which is not the case for geolambda).

Concretely:

zappa_settings.json
        "layers": ["arn:aws:lambda:eu-west-2:524387336408:layer:gdal32:4"],
       ...
        "environment_variables": {
            "GDAL_LIBRARY_PATH": "/opt/lib/libgdal.so.3.2.1",
            "GEOS_LIBRARY_PATH": "/opt/lib/libgeos_c.so.1.16.2",
            "GDAL_DATA": "/var/task/share/gdal" 
        },
settings.py:
    GDAL_LIBRARY_PATH = os.environ['GDAL_LIBRARY_PATH']
    GEOS_LIBRARY_PATH = os.environ['GEOS_LIBRARY_PATH']
To find where the library paths to put in zappa_settings.json actually are:
    # zappa invoke dev "import subprocess; print(subprocess.getoutput(['find / -name *libgdal*']))" --raw
    # zappa invoke dev "import subprocess; print(subprocess.getoutput(['find / -name *libgeos*']))" --raw
Django==3.2
zappa==0.52.0

thanks for this, only difference for me was that I had to set GDAL_DATA to /opt/share/gdal

Alex-Mackie commented 1 year ago

Update for 2023 - IMO the best way to deploy GeoDjango with Zappa is via a custom Docker image. This knocks the socks off previous solutions. See #967 and https://ianwhitestone.work/zappa-serverless-docker/.

For example (there are lots of ways to do this but this worked for me):

Dockerfile

FROM python:3.8-slim-buster

RUN apt-get update && \
  apt-get install -y \
  g++ \
  make \
  cmake \
  unzip \
  libcurl4-openssl-dev \
  binutils \
  libproj-dev  \
  gdal-bin

ARG FUNCTION_DIR="/var/task/"
RUN mkdir -p ${FUNCTION_DIR}
WORKDIR ${FUNCTION_DIR}

RUN pip install --target ${FUNCTION_DIR} awslambdaric

ENTRYPOINT ["/usr/local/bin/python", "-m", "awslambdaric"]

COPY ./ ${FUNCTION_DIR}

RUN pip install poetry
RUN POETRY_VIRTUALENVS_CREATE=false poetry install --no-root

RUN ZAPPA_HANDLER_PATH=$( \
    python -c "from zappa import handler; print (handler.__file__)" \
    ) \
    && echo $ZAPPA_HANDLER_PATH \
    && cp $ZAPPA_HANDLER_PATH ${FUNCTION_DIR}

CMD ["handler.lambda_handler"]

settings.py (in this exact case)

GDAL_LIBRARY_PATH = '/usr/lib/libgdal.so.20.5.0'
GEOS_LIBRARY_PATH = '/usr/lib/x86_64-linux-gnu/libgeos_c.so.1'

github-actions[bot] commented 5 months ago

Hi there! Unfortunately, this Issue has not seen any activity for at least 90 days. If the Issue is still relevant to the latest version of Zappa, please comment within the next 10 days if you wish to keep it open. Otherwise, it will be automatically closed.

github-actions[bot] commented 5 months ago

Hi there! Unfortunately, this Issue was automatically closed as it had not seen any activity in at least 100 days. If the Issue is still relevant to the latest version of Zappa, please open a new Issue.

zappa / Zappa