[SPIKE] As tech lead, I want to decouple backend apps

[andrew-jameson]

Description: When deploying our app to cloudgov, we run the gunicorn_start.sh script as the manifest's command, which is run to start up the application. Currently, gunicorn_start.sh runs the following co-hosted operations

• deploy local redis
• django makemigrations
• django migrate
• seed script populate_stts
• django collectstatic
• start worker processes celery and beat
• start application with gunicorn_start

Ideally, application startup should only consist of gunicorn start; all application dependencies would be handled by other steps in our pipeline. In some cases, such as when running python manage.py migrate where the python code is needed in order to execute the step, this may be impractical. Investigate which of the gunicorn_start.sh steps can be decoupled from application startup. Prioritize:

• speed and reliability in deploying the application
• reliability and recoverability of pipeline steps
• decoupling of infrastructure and application code

Acceptance Criteria:

• [x] Identify steps that can be decoupled from application start. priority order
  • deploy redis to an external service
  • remove makemigrations call
  • call migrate,populate_stts, and collectstatic from other pipeline steps.
• [ ] Testing Checklist has been run and all tests pass
• [ ] README is updated, if necessary

Tasks:

• [x] Investigate connecting to redis cf service
• [x] Attempt proof of concept of not installing redis in buildpack
• [x] Investigate using a python-enabled task in circleci to run migrate, populate_stts, and collectstatic
• [x] Investigate if hosting celery as a separate service/application is possible (a separately hosted repo with a REST api might be the only possibility).
• [ ] Run Testing Checklist and confirm all tests pass

Notes:

• makemigrations
  • https://docs.djangoproject.com/en/4.1/topics/migrations/#workflow
  • Migration files should be generated locally, committed to a git repository, checked out on the server, then applied to the production database. Running makemigrations on the production server creates the potential for untracked database changes.
• collectstatic
  • https://docs.djangoproject.com/en/4.1/ref/contrib/staticfiles/
  • We should consider running collectstatic elsewhere and serving static files via a new nginx location
  • https://dev.to/koladev/serving-djangos-static-files-with-nginx-3nc4

Supporting Documentation:

Open Questions:

[jtimpe]

• migrate and populate_stts from circle ci
  • The cloud.gov-prescribed way to handle this issue for migrations is to deploy a one-off app based on the local branch, then run the task (migrations, populate, etc), and finally tear down the app - https://cloud.gov/docs/management/one-off-tasks/#deploy-an-app-that-performs-a-task
  • We could potentially use cf run-task as shown in the example from the docs (linked in the above article) but would need to ensure the application containing the new code has been deployed, which could pose problems if the app can't start up without the database migrations having taken effect.
  • A compromise could be to deploy the new code in a new application without starting it, run migrations/database operations through independent run-task commands in circleci, then start the new app and finally roll over traffic. That would allow each step to be independently controlled, but might be more overhead than the out-of-the-box --rolling deploy
• Deploy celery as a separate cloud.gov application (separate manifest)
  • independently scalable
  • requires separate redis service - draft pr https://github.com/raft-tech/TANF-app/pull/2590

[jtimpe]

@Smithh-Co i think this can be closed as all the follow-on tickets have been discussed and prioritized.