Integration test framework for failed 7.x-to-8.x upgrades

[williamrandolph]

We need a rather convoluted new integration test case for the 8.x upgrade. See: https://github.com/elastic/elasticsearch/issues/81326

Our Gradle tests must be able to:

1. Start up a 6.x node and create indices, then shut it down
2. Start up a 7.17 node and leave at least one 6.x index unmigrated, then shut it down
3. Start up an 8.x node and wait for a process failure
4. Run a rollback command line utility to restore the data directory to a 7.x format
5. Start up a 7.17 node and check cluster health

We do not need a matrix test; it should be sufficient to use 6.last and 7.last in these tests.

This issue is for the Gradle work that will coordinate the node startup and shutdown. Once this is complete, we can follow up with Java classes that handle creating and checking the test data.

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[mark-vieira]

@williamrandolph does https://github.com/elastic/elasticsearch/issues/81862 change this at all? With the requirement to upgrade through 7.last does that change this scenario in any way or will it still exist, and thus necessitate this testing.

[rdnm]

As @williamrandolph is now out, I'll try answer this one. I believe this is no longer required as we are taking a different approach to this which no longer requires the recovery utility, at least in the short term. cc @grcevski @DaveCTurner in case I have this wrong.

Step 4 above is certainly not required. As there should be no destructive action taken as we bail out early, the user would re-install 7.x. I don't know if we have tests which do that currently.

[DaveCTurner]

I believe this is no longer required

I think we will require this tool although we think it's not critical to deliver it in 8.0 - at least, we are ok taking that risk.

[williamrandolph]

I'll remove the 8.0 label from the issue. I also think we want to have this kind of test in the long run.