azavea / pfb-network-connectivity

PFB Bicycle Network Connectivity
Other
40 stars 10 forks source link

Update Tilegarden node to 12.x #810

Closed flibbertigibbet closed 3 years ago

flibbertigibbet commented 3 years ago

Overview

Updates the Tilegarden node version to 12 locally, and updates all node dependencies.

This PR is intended to test what happens in the AWS environment when the update gets deployed to staging; if it works fine, this will close #801. I expect however that there will be AWS Lambda environment-specific errors with node-mapnik.

Notes

I've manually updated the node runtime on AWS for the staging environment to 12 in preparation for merging this. I couldn't find anything in the deployment directory to set the runtime, so it appears to be a manual configuration. The Tilegarden manual deployment instructions look to be very involved, so I think the easiest way to test how this behaves in the AWS environment would be to let CI deploy to staging once this PR is merged in.

Given this issue with node-mapnik, I expect that there will be node-mapnik errors on AWS that do not occur locally. I think it may be helpful to confirm those errors exist before proceeding to implement a custom runtime (as described in the linked issue).

Testing Instructions

Checklist

Connects #801

KlaasH commented 3 years ago

I pushed a test branch which Jenkins built and deployed to staging (branch here, CI build here). As expected, all is not well in Mapnik-land.

The logs are full of

Mapnik LOG> 2020-10-19 21:40:56: Problem loading plugin library: /var/task/node_modules/mapnik/lib/binding/lib/mapnik/input/postgis.input (dlopen failed - plugin likely has an unsatisfied dependency or incompatible ABI)

and

2020-10-19T21:40:56.610Z b8f31252-d77c-4cc8-8ee0-2aeeb97c3f93 ERROR [Error: Could not create datasource for type: 'postgis' (searched for datasource plugins in '/var/task/node_modules/mapnik/lib/binding/lib/mapnik/input') encountered during parsing of layer 'neighborhood_waysTF' in Layer]

flibbertigibbet commented 3 years ago

I just pushed a commit to here and the test branch that adds the node PostgreSQL package pg as a dependency, in the hopes that will pick up some missing binary, as this comment seems to suggest is the issue.

KlaasH commented 3 years ago

It looks like it's crashing the same way.

Since the build takes like 40 minutes to fully cycle, a tighter trial-and-error loop would be helpful. I believe this will work (from the default directory, /vagrant/, within the VM) to manually deploy just the Tilegarden service on staging:

export ENVIRONMENT="staging"
export PFB_SETTINGS_BUCKET="${ENVIRONMENT}-pfb-config-us-east-1"
export GIT_COMMIT=$(git rev-parse --short HEAD)
aws s3 cp "s3://${PFB_SETTINGS_BUCKET}/tilegarden/.env" "./src/tilegarden/.env"
aws s3 cp "s3://${PFB_SETTINGS_BUCKET}/tilegarden/claudia.json" "./src/tilegarden/claudia/claudia.json"

docker-compose \
    -f docker-compose.yml \
    -f docker-compose.test.yml \
    run --rm --entrypoint yarn tilegarden deploy
flibbertigibbet commented 3 years ago

I'm going to close this PR but leave the branch. There appears to be some package/binary missing from the base image in the more recent Node Lambda environments that was present previously. It's not immediately clear to me how to fix the Lambda environment issues, particularly given the use of Claudia for building and deploying the Lambda.