Closed cisaacstern closed 1 year ago
A bit more brainstorming before opening a PR:
deployment_status
event looks like the best option for the GitHub Workflow trigger. This way, when Heroku updates the Review App's deployment_status.state
to success
, we know we have a functioning instance to call this test on.deployment_status
event alone, then we are potentially also deploying a Dataflow job at each one of these Review App builds.build-review-app
label to the PR, which would mirror the PR-label-based developer UI for https://github.com/pangeo-forge/pangeo-forge-runner/pull/52.So brainstorming a rough work plan for the PR that adds this test:
deployment_status
trigger when the Review App is successfully built/run
recipe test comment on an open PR in the pforgetest
org. The latter is easy enough, and probably preferable for sake of end-to-end completeness.pforgetest
, and then develop some logic to dynamically assign an un-used one of these apps to the Heroku Review App on creation.
Recent experiences with failed dataflow job deployment #220 have demonstrated the need for integration testing. Last week I added a Dataflow integration test to
pangeo-forge-runner
https://github.com/pangeo-forge/pangeo-forge-runner/pull/52. This gives us a baseline confidence that a given release ofpangeo-forge-runner
works in the test environment used there. It's possible that our application container environment may introduce unique client/dataflow incompatibilities and/or that the way we invokepangeo-forge-runner
(the literal command we use, but also the process context: subprocess, network call to external service, etc.) may introduce problems. In fact, as discussed in #220, it seems that we are facing some type/combination of these issues currently.A dataflow integration test here could:
pangeo-forge-runner
directly, but within the app container. This would test the client/dataflow incompatibility point mentioned above, but it would not get at the way our application actually invokespangeo-forge-runner
in production. The brainstorm I started in #223 reflects this idea. I've come to feel that this is not sufficiently realistic, and that a better approach would be to...pangeo-forge-runner
indirectly, by sending a payload to a running instance of the application which triggers the actual code path used in production to deploy jobs to dataflow. This approach is a bit more involved to setup, but ultimately is the more confidence-inspiring and realistic test, as it gets much closer to replicating the production scenario.Assuming we do want to pursue the latter, more realistic, option, I can envision two ways of getting an instance of the application started to test:
docker-compose
to stand up a production instance within GitHub Actions, similar to what we already do in the docker tests: https://github.com/pangeo-forge/pangeo-forge-orchestrator/blob/main/.github/workflows/test-docker.yaml. In fact, such an approach could simply build off of that same workflow. The advantage to this approach is that we control the setup of the stack. The disadvantage is that it's less reflective of the actual production environment.I'll start a PR to address this.