27 Github action tests fails out of 53 test for the sprint 33 snapshots

Informatievlaanderen / VSDS-LDES-E2E-testing

End-to-end tests and tooling needed for testing LDES components build as part of VSDS

European Union Public License 1.2

5 stars 2 forks source link

27 Github action tests fails out of 53 test for the sprint 33 snapshots #364

Closed xdxxxdx closed 1 year ago

xdxxxdx commented 1 year ago

Validation tags:

Scenario: Run the Github action against the provided tag: https://github.com/xdxxxdx/VSDS-LDES-E2E-testing/actions/runs/5900026842

Current result: 27/53 failed.

Expected result: All tests should pass. Or there is an issue in the test setup. Or there is an issue in the VSDS solution.

rorlic commented 1 year ago

The tests have been written with some knowledge on how things work internally (e.g. fragmentation happens during ingest). The LDES server and other LDES components sometimes change their inner workings (e.g. fragmentation now happens after ingest) which breaks the tests sometimes. In addition, some fragmentations have been changed (e.g. pagination is now done by default).

As we are also working on onboardings so the E2E tests are not always updated immediately.

I have disabled 4 tests and fixed the server basic and fragmentation tests. There are still 12 out of the 49 tests that fail (mostly due to some config issues). I will try to fix these soon.

rorlic commented 1 year ago

We have fixed all tests (currently 51). These tests all work when run locally. Occasionally on github action a test may fail. This is most likely due to timing. We will harden the tests to handle these failures.

Please close this issue.

xdxxxdx commented 1 year ago

https://github.com/xdxxxdx/VSDS-LDES-E2E-testing/actions/runs/6039492027

49/51 passes the test. E2E up-to-date with sprint 34 tags. 3c2cdc6132c5d667447e9511160c430405295317