502 Bad Gateway error in AWS production deploy

MikeTheCanuck commented 6 years ago

The base service URL for this project's API in AWS is http://service.civicpdx.org/local-elections/

Currently requests to this URL return 502 Bad Gateway. This is a result of a few factors:

Hack Oregon APIs are published as a load-balanced service, thus we use ALB (Application Load Balancer from AWS) to front-end all requests and route them to a health replica of the service's container. ALB itself is returning the 502 error, because it is unable to get a valid 200 response from either replica of the container as currently published.
All API services are hosted at the same FQDN (service.civicpdx.org) at separate routes. While in local development context the route is not necessary to locate the APIs, once the container is published to AWS they are necessary to disambiguate requests and route them to the correct container service e.g. the request for /local-elections/ is routed to a separate container pair than the requests for /transportation-systems/
When the routed request reaches the container the gunicorn lightweight web server (that hosts the Django code) needs to know where to route each request within the application - a request for /transportation-systems/ is answered by separate Django code from /transportation-systems/biketown/ for example. Thus the external route /local-elections/ must be explicitly specified in the urls.py configurations of the Django project, and then specify each of the endpoint routes separately in each of the "Django app" urls.py configurations.

To tie it all together, this is exactly what is happening here:

this project's Docker container is being successfully uploaded to ECR (the AWS Elastic Container Registry) and successfully pushed to ECS (the AWS EC2 Container Service)
ECS starts the container fine, and it becomes available (internally within AWS) for ALB to perform the tests we have configured, to check whether the container is healthy (before making it available to the world to answer external requests)
ALB sends a request - to that specific, new container - at the base route we have configured - /local-elections/ - and awaits any response with HTTP code 200 - 299
gunicorn receives that request for /local-elections/ and maps it to any of the URL routes that are published in urls.py, but currently doesn't find that route
gunicorn returns a 404 Not Found response to ALB for that request, which is exactly what we see in the CloudWatch logs that record the output from gunicorn: 10.180.21.252 [02/Jun/2018:16:22:06 +0000] GET /local-elections/ HTTP/1.1 404 90 - ELB-HealthChecker/2.0 16.020645
since ALB doesn't yet have a working container to respond successfully to that request, it returns a 502 Bad Gateway error to the original request (Why is it a 502 error and not 404 Not Found or something else a little more intuitive? Well, we just don't know, but that's what we've observed)

grantholly commented 6 years ago

MikeTheCanuck commented 6 years ago

Yep, looking good - thanks for reminding me, I've harvested these notes over to https://github.com/hackoregon/civic-devops/issues/168

hackoregon / elections-2018-backend