Use web app end to end tests before releasing new versions

svenseeberg commented 1 year ago

Motivation

We quite often release new versions of the CMS without testing if the app still works with the new CMS version. The app team already has a end to end test setup for the web app, which we could use to test compatibility of the API.

Proposed Solution

When preparing a release (pull request from develop branch to main branch), trigger a re-run of the last web app end to end test. The app team can parametrize the domain for the test setup so that it can test against any publicly running CMS.

Three things need to happen:

The relevant content from integreat-test.tuerantuer.org/testumgebung-e2e/ needs to be imported into the test data / fixtures
The CI pipeline for main branch pull requests would have to deploy the CMS to a Hetzner VM
The CI pipeline for main branch pull requests would have to re-run the web app end to end tests with the correct domain parameter and wait for the test result

Alternatives

Alternatively, we could only use the app API client code and do some integration tests with it. That would be much more efficient, but not be a full end to end test.

Additional Context

There are multiple alternatives:

This would have to be accompanied by https://github.com/digitalfabrik/integreat-app/issues/2108
We could reduce the effort by simply triggering a deployment of the latest develop commit on integreat-test.tuerantuer.org. For that we do not need to change a lot, only create a working web hook. Another advantage is that the app team would not have to change anything.

Design Requirements

timobrembeck commented 1 year ago

The relevant content from integreat-test.tuerantuer.org/testumgebung-e2e/ needs to be imported into the test data / fixtures

I don't think that's necessary, the ci tests with the test fixtures are independent from the database deployment on the server which is used for the end to end tests? Sure, we could use the same test data for consistency reasons, but it's not required for the task I think?

The CI pipeline for main branch pull requests would have to deploy the CMS to a Hetzner VM

Why Hetzner VM? Why isn't it sufficient to use https://cms-test.integreat-app.de/ which is automatically updated to the develop branch anyway?

svenseeberg commented 1 year ago

The relevant content from integreat-test.tuerantuer.org/testumgebung-e2e/ needs to be imported into the test data / fixtures

I don't think that's necessary, the ci tests with the test fixtures are independent from the database deployment on the server which is used for the end to end tests? Sure, we could use the same test data for consistency reasons, but it's not required for the task I think?

Indeed, the idea here was to have consistent test data across front end and back end. But I agree, this is no strictly necessary.

The CI pipeline for main branch pull requests would have to deploy the CMS to a Hetzner VM

Why Hetzner VM? Why isn't it sufficient to use https://cms-test.integreat-app.de/ which is automatically updated to the develop branch anyway?

Agreed. Would save us a lot of effort.

To sum things up: We should do it as you suggested. This is definitely the least effort and we can iterate later if required.

svenseeberg commented 7 months ago

The ticket on the app side got closed: https://github.com/digitalfabrik/integreat-app/issues/2108#issuecomment-1680041797

digitalfabrik / integreat-cms