[Feature Request] Package and export all cluster configuration, including plugins

[Jon-AtAWS]

Is your feature request related to a problem? Please describe.

Create an API that packages all configuration necessary to recreate the cluster, either for mirroring the cluster to a different location, or for HA/DR. At present, the _snapshot API exports a portion of the config (and data) needed. However, much of the plugin configuration is missing, necessitating its recreation via a large number of individual, plugin APIs.

This feature would support:

• HA/DR
• Creating identical clusters in the cloud and in my own data center.
• Migrating a cluster from my data center to the cloud or the reverse.
• Cloning a cluster for Dev/Test/QA

Possibly this is an extension to the existing Saved Objects import/export. It should include config for all of the plugins installed.

Describe the solution you'd like OpenSearch should provide import/export for at least the following configuration. It must preserve the object relationships.

[Exists] Index patterns [Exists] Dashboards dashboards and visualizations Dashboards tenants Alerting Notifications Streaming Anomaly Detection Security Index State Management Transforms Rollups Reports [Maybe] ML Commons [Maybe] Observability [Maybe] Dynamic _cluster/settings, not in the yml. [Probably not] Cross-cluster setup [Speculative] cluster infrastructure (this would be helpful, but probably isn't possible) [not considering] data

Consider updating the Saved Objects panel in Dashboards to read an exported package, and allow the user to selectively apply configuration.

Describe alternatives you've considered At present, you can accomplish most desired import/export tasks with various per-plugin APIs. A more packaged solution will make it much easier.

Another alternative would be expanding the snapshot API to include the above [I read up on include_global_state: true, but couldn't figure out what it actually includes]. There would need to be an improvement to the snapshot API, and could be a Dashboards UI to simplify selective import to various components.

[stevef-5gn-com-au]

Any movement on this? It's effectively impossible to automatically back up our Opensearch cluster.

[dblock]

@stevef-5gn-com-au I don't see anyone working on this. Maybe @sumobrian or @setiah can share some existing strategies, but otherwise would love for someone (you?) to take it up!

[stevef-5gn-com-au]

Thanks @dblock. I did have a quick look at this, but unfortunately I don't have the knowledge of all the plugins and API's etc needed. This really needs to come from Open Search itself, and shouldn't need to rely on backing up each and every plugin individually.

And even if I did get all of the GET API's working to pull the data, reimporting it is a whole another story, because the format of the data will not be correct.

I would love a button (or an API, or a config when can be scheduled) which will export all of the OS config, and allow it to be imported into another cluster.