Open jmini opened 2 months ago
LeJeanbono
commented
1 month ago Even if this would allow to update GitLab, I fear to face the same issue: it takes time to start GitLab inside a container and the test might continue to be flaky.
With TC you can wait for readyness: https://java.testcontainers.org/features/startup_and_waits/
jmini
commented
1 month ago Yes, I think this is working in the branch pr_935 where I continued the work started in https://github.com/gitlab4j/gitlab4j-api/pull/935
I see two problems with test containers:
Right now the running the tests is very quick. Even on GitLab CI the job takes now less than a minute now that I made the switch to use gradle.
The only advantage would be that it is probably easier than the solution with capture and replay.
At the end what we want in this project:
I think a "capture and replay" mechanism would really help with this. In an other context I worked on a prototype here: https://github.com/jmini/quarkus-experiments/tree/main/mockserver-record-replay
I now would explore if the concept presented in that project would work in a bigger project with so many API calls.
Current setup:
The current setup is based on the fact that maven is starting a very old image of GitLab (version
12.9.2-ce.0) and is then executing some integration tests against this instance of GitLab.Since I am maintaining this library I do not see any value in those tests.
They are slow and very flaky. Contributors try to extend existing integration tests and sometimes those are failing because the GitLab version is outdated.
Testcontainers attempt:
There is an issue https://github.com/gitlab4j/gitlab4j-api/issues/925 and a pull request https://github.com/gitlab4j/gitlab4j-api/pull/935 where some work was made to replace the current setup based on maven with something based on Testcontainers.
This is almost finished but I never took the time to finish it.
Even if this would allow to update GitLab, I fear to face the same issue: it takes time to start GitLab inside a container and the test might continue to be flaky.
Decision:
New solution:
This should be based on a mock server (I am currently working with https://www.mock-server.com), in order to have fast and reproducible HTTP responses during our integration tests.
The maintenance of the stubs (the responses corresponding to given http requests) is a real challenge over time. I do not want to write them manually and I would prefer capturing them at development time, by running the tests locally against a real GitLab instance (probably started locally, with or without docker). This way we would always have a script available to renew or extend the mocked responses.
But in the CI we would have very fast and reproducible HTTP responses. Running the test suite would not require any docker or any gitlab installation.
This issue is open to collect feedback.