Add an API (?) for restoring an entire cluster from a snapshot

[joegallo]

Description

Add an API or other special mode/operation for restoring an entire cluster from a snapshot.

Motivation

The reason to have this is twofold. First, the instructions for doing this operation, while documented, are somewhat long (link), and they seem to keep accreting more steps.

More than that, however, the nature of instructions like that have a way of turning feature releases into breaking changes.

To wit: perhaps I have carefully captured those instructions into my homegrown Elasticsearch management tool. Then Elasticsearch releases a new version, let's call it 8.13, and that version of Elasticsearch has a new frobnicate feature. That feature also has to be disabled (POST frobnicate/_stop) and later re-enabled (POST frobnicate/_start) and our 'Restore an entire cluster' documentation has been updated to match. The problem is that now my homegrown Elasticsearch management tool no longer works for restoring an entire cluster -- a feature release in those instructions became a breaking change in my tool.

Some potential implementation approaches / notes

My first thought is that this is fundamentally just a new destructive option that's added to the _restore process, perhaps something like this:

POST _snapshot/my_repository/my_snapshot_2/_restore?wait_for_completion=true
{
  "indices": "*",
  "include_global_state": true,
  "destructively_restore_entire_cluster": true
}

That sort of implies that this operation is valid to run at more or less any time against more or less any cluster, that it's just something somebody could wish to do whenever. And maybe that's a bad idea, which brings us to the next option...

In conversation with @DaveCTurner he proposed that perhaps a different view of this feature is that it's more like bootstrapping a cluster, which we could loosely model in a way like cluster.initial_master_nodes (maybe cluster.initial_snapshot_restore?). That is, from a blank slate with a bunch of machines but with no cluster, drop some configuration into elasticsearch.yml across all the machines that indicates that this cluster is not being started empty, rather it should bootstrap itself to the point where it can execute a snapshot restore (but probably cannot execute arbitrary write operations), then execute the snapshot restore, and only once the restore has finished would it actually finish booting and open itself up to arbitrary write operations. After that snapshot restore is completed on first boot, then the cluster.initial_snapshot_restore setting (or whatever we settle on) would be ignored forever.

I'm sure there are many tricky bits there that I don't know enough to see, but one thing that springs to mind is that some parts of this would be living in elasticsearch.yml while other parts would need to be banged into place post boot with API calls (at the very least there's the matter of where the snapshot repository configuration comes from..., but of course we could also imagine a world in which this is a special elasticsearch.yml configuration option that also allows specifying the entire snapshot repository configuration inline or something like that).

Unrelated to the previous idea, in conversation with @dakrone he mentioned that it would be especially nice if this were a persistent task that could survive master elections / restarts -- there's not much point in adding a cool API (or operation) like this if we end up punting on the error handling to the point where the user still needs to do a bunch of status checking and retrying themselves externally.

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[hrishikeshdkakkad]

Can I help in building this? I am a first-time contributor. Would need some guidance from Engineers working on the codebase - but I am really good at Java, I can explore, triangulate and add the enhancement.

Thanks