Elasticsearch Stack client tests

[picandocodigo]

Given this project, we've talked about doing the same for the Elasticsearch Stack clients: write our own YAML tests instead of depending on the Elasticsearch ones.

This would give us several advantages:

• The "Elasticsearch YAML tests" test Elasticsearch, the server. This means there's a lot of edge cases and functionality that needs to be tested, that is not necessarily relevant to clients. It could be argued that testing all of these ensures the clients are 100% compatible with the API and it is a very robust set of tests. But we end up skipping tests a lot due to different configurations and such, so we're not 100% testing anyway. We can have more control, even add tests that are not yet implemented on the server when an API is experimental and such, if we write our own client tests.
• As long as we implement teardown on each test, there's no need to have a clean-up phase. This is constantly being updated and in a lot of cases has been the cause for the clients build braking. It would save a lot of execution time: deleting only necessary artifacts created for the test, not having to get lists of artifacts and delete one by one, and I'm sure we'll see our integration tests build take less time. We would also run less tests, but the cleanup phase is run after every single test on the current architecture. It would also save developer time, not having to update the clean-up phase every time something changes, tests should have their own clean-up.
• Initially the Elasticsearch Stack tests will be a superset of the Serverless tests, although these could eventually diverge. I think the process moving forward should be:
  • Leave the current folder tests as is, and use it for the "common" tests, tests that will run both in Stack and Serverless.
  • Write Elasticsearch Stack only tests in a new directory stack.
  • When we find a test that's exclusive to serverless, create it in a new directory serverless.
• If not implemented already, we might want to add catch, headers and potentially other features from the previous test set. The current tests are already using do, match, set, gt, gte, lt, lte, is_true, setup and teardown.
• We have ~ 480 APIs in Stack. At the time of writing this issue, there are ~ 200 APIs being tested in this repo, so we're not that far...

[ezimuel]

@picandocodigo thanks for putting all these new YAML tests together, this iwll be a game changer for us!

I think this proposal makes sense for many reasons:

• I never think that the Elasticsearch YAML tests are needed from a client perspective. These are blackbox tests for Elasticsearch with many specific tests about the internal server behaviour. We don't need it for testing clients. We used this tests in the past because we didn't have other options :smile:
• I really like the usage of teardown instead of having a clean-up phase at the end of each tests. This will drastically improve the speed of testing and also the consistency, since the tests will become more isolated, without tedious cascading effects that we had in the past. :+1:
• I like the idea of having a common set of tests in the main folder and specific directories for stack and serverless but I think we should also tag the versions for the stack release (i.e. as we have today). That said, potentially we'll need to create a separate repository for the stack tests, even because the serverless test will have a different versioning system (i.e. API version). I'm not sure about this, I think we need to discuss more.

[ezimuel]

I think we need also to tag or release the tests with the stack number version (e.g. 8.12.1).