CI/CD Workflow v2

[atolopko-czi]

Goals/Requirements:

• Simplify multiple GHA workflow design into a single workflow to:
  • Ease engineering understanding of the worklow
  • Have a single view of deployment graph in GHA
  • Eliminate complexities and edge case behaviors that can occur with having multiple workflows invoke one another and having to check statuses of upstream workflows
• GHA builds should not become backlogged. As much as possible, later build/deploy jobs should preempt earlier ones. The latest build/deploy should be given priority for RDev and staging environments.
• E2E tests should only run if unit tests pass (i.e. fail fast)
• Runs unit tests only once when building image
• Docker images built once for a given commit and used for deploys to all environments
• No E2E tests run on prod, only real smoke tests (a limited set of spot checks to ensure prod is available post-deployment). Datadog Synthetics may be deemed sufficient (see https://docs.datadoghq.com/continuous_testing/cicd_integrations/github_actions), after a review and/or with changes.
• Eliminate staging and prod branches, replace with tags (this eliminates git merge issues if/when a target branch gets out of sync; tagging is equivalent to just recreating a target branch each time).
• Ensure race conditions are avoided that might cause the last deployed commit to be incorrect (i.e. not the latest commit)
• Ensure that a staging or prod deployment will only occur if the unit tests and E2E tests for the commit being deployed have all passed (there should be no possibility of passing tests from another commit allowing a failing HEAD commit to be deployed).
• RDev deployments should occur as fast as possible for Draft PRs; they should not wait on unit tests or for previous RDev deployments GHA jobs to complete running E2E tests.
• RDev deployments should only occur if source code changes (i.e. changes that affect the behavior of the deployed system).

TODO: Write proposal to explain the proposed new workflow, as depicted in this diagram

This epic captures the work that was originally detailed in the following issues: https://github.com/chanzuckerberg/single-cell-data-portal/issues/6143 https://github.com/chanzuckerberg/single-cell-data-portal/issues/6047 https://github.com/chanzuckerberg/single-cell-data-portal/issues/5895 https://github.com/chanzuckerberg/single-cell-data-portal/issues/5939 https://github.com/chanzuckerberg/single-cell-data-portal/issues/5896

There is likely additional detail in these issues that should be reviewed and incorporated into a forthcoming tech spec for this work.

[Bento007]

We will likely want to make use of GHA reusable workflows