Met with @AaronPlave @duranb @mattdailis and @skovati about this yesterday.
Background
The Aerie-UI repo contains a number of E2E tests which are run on every PR, and also run locally by the frontend devs during development to ensure nothing has broken due to frontend changes. This process uses:
and the Aerie backend docker containers that are published to our image repository
Problem
There have been a few instances lately where changes to the backend Aerie repo unintentionally caused the UI's E2E tests to break. This can happen for a few reasons: Sometimes it's due to an actual bug in the backend code, sometimes an unintentional or intentional breaking change to the model interface requires an update to the Banananation model to continue working, and sometimes the Banananation model just needs to be recompiled and the new JAR needs to be committed.
The problem is that this can happen unexpectedly, and it's not always clear to the frontend devs why the test is breaking - often they assume it's due to a frontend problem and end up losing time trying to debug when it should be solved by a mission model update/recompile.
Proposed Solution
Several solutions were discussed which could give more exhaustive coverage here, but we decided that the best first step would be a "canary in the coal mine" which would at least give us some advance warning of breaking changes, and make it clear when the breakage was happening due to backend changes vs. frontend. The proposed implementation is:
Set up a Github action to run nightly, using the schedule event, which behaves similarly to a cronjob
In this nightly action, run the UI E2E tests using the develop branch of aerie-ui, (+ the checked-in JAR and published Aerie backend, per usual)
Have some way for this action to alert us if the tests fail
This should give us some early indication that breaking changes exist and need to be resolved, and would help the UI team understand which failures are due to their code changes vs. other things outside of their control.
Met with @AaronPlave @duranb @mattdailis and @skovati about this yesterday.
Background
The Aerie-UI repo contains a number of E2E tests which are run on every PR, and also run locally by the frontend devs during development to ensure nothing has broken due to frontend changes. This process uses:
Problem
There have been a few instances lately where changes to the backend Aerie repo unintentionally caused the UI's E2E tests to break. This can happen for a few reasons: Sometimes it's due to an actual bug in the backend code, sometimes an unintentional or intentional breaking change to the model interface requires an update to the Banananation model to continue working, and sometimes the Banananation model just needs to be recompiled and the new JAR needs to be committed.
The problem is that this can happen unexpectedly, and it's not always clear to the frontend devs why the test is breaking - often they assume it's due to a frontend problem and end up losing time trying to debug when it should be solved by a mission model update/recompile.
Proposed Solution
Several solutions were discussed which could give more exhaustive coverage here, but we decided that the best first step would be a "canary in the coal mine" which would at least give us some advance warning of breaking changes, and make it clear when the breakage was happening due to backend changes vs. frontend. The proposed implementation is:
scheduleevent, which behaves similarly to a cronjobdevelopbranch ofaerie-ui, (+ the checked-in JAR and published Aerie backend, per usual)This should give us some early indication that breaking changes exist and need to be resolved, and would help the UI team understand which failures are due to their code changes vs. other things outside of their control.