Editing the found-snapshots repository fails

[andreidan]

Kibana version: 8.4.1

Elasticsearch version: 8.4.1

Server OS version:

Browser version:

Browser OS version:

Original install method (e.g. download page, yum, from source, etc.): cloud staging

Describe the bug: Attempting to edit the found-snapshots repository yields:

[request body]: types that failed validation: - [request body.0.settings.location]: expected value of type [string] but got [undefined] - [request body.1.settings.url]: expected value of type [string] but got [undefined] - [request body.2.settings]: types that failed validation: - [request body.settings.0.location]: expected value of type [string] but got [undefined] - [request body.settings.1.url]: expected value of type [string] but got [undefined] - [request body.settings.2.bucket]: expected value of type [string] but got [undefined] - [request body.settings.3.uri]: expected value of type [string] but got [undefined] - [request body.settings.4.containerName]: definition for this key is missing - [request body.settings.5.bucket]: expected value of type [string] but got [undefined] - [request body.settings.6.delegateType]: expected value of type [string] but got [undefined] - [request body.3.settings.bucket]: expected value of type [string] but got [undefined] - [request body.4.settings.uri]: expected value of type [string] but got [undefined] - [request body.5.settings.containerName]: definition for this key is missing - [request body.6.settings.bucket]: expected value of type [string] but got [undefined]

Steps to reproduce: Stack Management -> Repositories -> Edit the found-snapshots repository but not make any changes -> Save

Expected behavior: Editing the repository goes through successfully

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:

[elasticmachine]

Pinging @elastic/platform-deployment-management (Team:Deployment Management)

[alisonelizabeth]

It looks like the error is due to the S3 schema validation failing (definition). The existing found-snapshots repository includes the following settings, which the validation does not account for:

• aws_account
• use_for_peer_recovery

I don't see these settings in the ES docs, so it's unclear to me if they were recently introduced or not.

[kingherc]

If I can help a bit, I cannot find aws_account in ES source code (it's probably introduced somewhere else?), but can refer @fcofdez for the introduction of use_for_peer_recovery in a PR (https://github.com/elastic/elasticsearch/pull/76237) a year ago to comment for the ES docs.

[fcofdez]

use_for_peer_recovery is a repository setting flag that lets Elasticsearch use that repository during peer recoveries in order to reduce data transfer costs. It's intended to be used only by cloud, and it is documented under indices.recovery.use_snapshots.

[alisonelizabeth]

Thanks @fcofdez and @kingherc! I will reach out to the ES distributed team to see about the aws_account setting.

It's intended to be used only by cloud, and it is documented under indices.recovery.use_snapshots.

Should it also be documented under the repository docs?

[fcofdez]

Should it also be documented under the repository docs?

That repository setting is meant to be used only in Elastic's cloud, so I'm not sure if we should document it there.

[alisonelizabeth]

I confirmed aws_account is also a cloud-only repository setting used for GCP and AWS

[rseldner]

This can be encountered outside of cloud, in any custom repo, because it is currently possible to add non-secure client settings into a repo's config. Though I do see this is being deprecated.

From ES 8.5 S3 Repo Doc:: https://www.elastic.co/guide/en/elasticsearch/reference/8.5/repository-s3.html#repository-s3

The option of defining client settings in the repository settings as documented below is considered deprecated, and will be removed in a future version.

In addition to the above settings, you may also specify all non-secure client settings in the repository settings. In this case, the client settings found in the repository settings will be merged with those of the named client used by the repository.

So users can run into this unless they remove the additional client settings from their repo, and move those settings over to the keystore and/or elasticsearch.yml

To reproduce...

1. create a snapshot repo via API and include any non-secure client settings

   PUT  _snapshot/test?verify=false
   {
   "type": "s3",
   "settings": {
     "bucket": "test-bucket",
     "client": "test-client",
     "base_path": "test-path",
     "path_style_access": true
   }
   }

2. Edit and save the repository in Kibana.