Remove deprecated ECS services

[MikeTheCanuck]

Now that we have a stable cluster of the 2018 services, and they have survived 2018 Demo Day (yay!), we can weed out the legacy services that were put in place last year, that are no longer needed in the current infrastructure.

Candidate services to remove - those whose role is unneeded or unclear - include:

• hacko-integration-EndPoints
• hacko-integration-CivicLabService

[MikeTheCanuck]

EndPoints service

• the Task Definition in ECS is labelled endpoint-service
• the service.yaml defines a Path of /endpoints*
• the Container Definition in the [Task Definition page] defines an Image value of 845828040396.dkr.ecr.us-west-2.amazonaws.com/integration/endpoint-service:latest and a Hard memory limit of 100 MiB

Further digging

• there is a Travis repo called endpoint-service-catalog whose ECS_SERVICE_NAME env var has a value of hacko-integration-EndPoints-G1ITM6CT7DDP-Service-1328677GKHL27
• the ECS service above has a different name - hacko-integration-EndPoints-1VSWR1EK7HGKE-Service-1LSYANQCAYMGJ
  • the naming difference is probably a result of the recent complete redeploy of the ECS cluster
• this indicates that the corresponding GitHub repo is https://github.com/hackoregon/endpoint-service-catalog
• by experimenting with turning off one of the services at a time, I confirmed that the "endpoint service" is what specifically answers to http://service.civicpdx.org/endpoints/, and is responsible for nothing else (has no bearing on the other routes to the API, such as http://service.civicpdx.org/budget)

Conclusion

Per issue #185, if we deem it worthwhile to publish a directory of APIs, we could continue to host the API directory (http://service.civicpdx.org/endpoints/) as a single-page NGINX site derived from the endpoint-service-catalog repo.

However we would have to do some repair work - probably to the Load Balancer, to the Travis/ECR/ECS deployment pipeline, and certainly to the content of the index.html (if not also to some of the container configuration).

If we abandon this container, then we will deprecate or delete the endpoint-service-catalog repo as well.

[nam20485]

Do we have a NGINX server? I've been wondering about this.

Currently we serve the Django service from Django's builtin UWSGI server, but this is not recommended for production use. Recommendation is to use the builtin UWSGI server for serving the dynamic content, with a NGINX server as reverse-proxy upfront, handing connections to the UWSGI server for response, and having the NGINX server handle all the static assets itself. As far as I know this reduces the load on, and performs better, under load.

Both Django and NGINX have functionality to enable this recommended production setup.

I also wonder if the lack of the NGINX server may have anything to do with our Django-DB persistent idle connection issues.

At any rate our setup is not a recommended setup for production use, as described by Django itself.

I know the answer will probably be not enough resources and bandwidth (human kind I mean) to accomplish, but is there any desire to move our nonstandard production setup to a recommended method? I've done it before on my own projects.

Maybe something we could tackle before or during next season?

On Sat, Jun 23, 2018, 12:09 PM Mike Lonergan notifications@github.com wrote:

EndPoints service

• the Task Definition in ECS is labelled endpoint-service
• the service.yaml https://github.com/hackoregon/hackoregon-aws-infrastructure/blob/master/services/endpoint-service/service.yaml defines a Path of /endpoints*
• the Container Definition in the [Task Definition page] defines an Image value of 845828040396.dkr.ecr.us-west-2.amazonaws.com/integration/endpoint-service:latest and a Hard memory limit of 100 MiB

Further digging

• there is a Travis repo called endpoint-service-catalog whose ECS_SERVICE_NAME env var has a value of hacko-integration-EndPoints-G1ITM6CT7DDP-Service-1328677GKHL27
• the ECS service above has a different name - hacko-integration-EndPoints-1VSWR1EK7HGKE-Service-1LSYANQCAYMGJ
  • the naming difference is probably a result of the recent complete redeploy of the cluster
• this indicates that the corresponding GitHub repo is https://github.com/hackoregon/endpoint-service-catalog

Conclusion Per issue #185 https://github.com/hackoregon/civic-devops/issues/185, if we deem it worthwhile to publish a directory of APIs, we could continue to host that directory as a single-page NGINX site derived from the endpoint-service-catalog repo - but we would have to do some repair work - probably to the Load Balancer, to the Travis/ECR/ECS deployment pipeline, and certainly to the content of the index.html (if not also to some of the container configuration).

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hackoregon/civic-devops/issues/183#issuecomment-399701731, or mute the thread https://github.com/notifications/unsubscribe-auth/ABvqFPKasYQNrkxAnVgXvifHCLqVrHwvks5t_pJbgaJpZM4U04QP .

[MikeTheCanuck]

Civic Lab service

• the Task Definition in ECS is labelled civic-lab-service
• the Container Definition in the Task Definition page defines an Image value of 845828040396.dkr.ecr.us-west-2.amazonaws.com/integration/civic-lab-service:latest and a Hard memory limit of 50 MiB
• the service.yaml defines a Path of /

Further digging

• there is a Travis repo called civic-platform, but it leads nowhere and doesn't currently have a .travis.yml so it's hard to imagine that's what's running in the civic-lab-service ECS container.
• there are no other Travis repos that jump out here
• the ECR (ECS Container Repository) repo for this service has only four images published to it, all from 2017-03-18
• of all the hints at "civic-lab-service" in any of the repo's code under the HackO organization, none of them tell us where the service's source code lives: https://github.com/search?q=org%3Ahackoregon+civic-lab-service&type=Code
• by experimenting with turning off one of the services at a time, I confirmed that the "civic lab service" is what specifically answers to http://service.civicpdx.org/, and is responsible for nothing else (has no bearing on any of the routes off this domain name, such as http://service.civicpdx.org/endpoints/ or http://service.civicpdx.org/budget)

Questions

1. Which repo holds the code running in the civic-lab-service containers?
2. Which URL leads to the "/" Path that this is supposed to listen on?
3. Was this an abandoned idea, that was instead deployed as the civicpdx.org site that lives in S3 (see #186)?

If we can't find any trace of this service's source code, then it's time to rip it out of all the infrastructure and just forget it ever happened.

Edit: Answers

1. The "source" to the page answering at http://service.civicpdx.org/ is in the hacko-aws-infrastructure repo as mentioned here. But there's definitely a container answering the requests for http://service.civicpdx.org/, so there must be some way that the container was built and deployed in the first place.
2. Not sure how exactly the requests for http://service.civicpdx.org/ get routed to this container and not the other containers that are being load-balanced via the same load balancer.
3. Best I can piece together, the "Civic Lab" was a similar idea to the Civic Platform is for this year - it would be a brand for the platform as it was envisioned in 2017, which at the time was very API-centric, so it stands to reason that service.civicpdx.org would be the centerpiece of the whole "civic lab".

[MikeTheCanuck]

Now that I've figured out how to get the endpoint-service-catalog's Docker container to listen on http://service.civicpdx.org/ (https://github.com/hackoregon/hackoregon-aws-infrastructure/pulls 45-50), the EndPoints service is sticking around. And the CivicLabService has been eliminated from the CloudFormation templates.