Closed andreidan closed 1 year ago
Pinging @elastic/platform-deployment-management (Team:Deployment Management)
It looks like the error is due to the S3 schema validation failing (definition). The existing found-snapshots
repository includes the following settings, which the validation does not account for:
aws_account
use_for_peer_recovery
I don't see these settings in the ES docs, so it's unclear to me if they were recently introduced or not.
If I can help a bit, I cannot find aws_account
in ES source code (it's probably introduced somewhere else?), but can refer @fcofdez for the introduction of use_for_peer_recovery
in a PR (https://github.com/elastic/elasticsearch/pull/76237) a year ago to comment for the ES docs.
use_for_peer_recovery
is a repository setting flag that lets Elasticsearch use that repository during peer recoveries in order to reduce data transfer costs. It's intended to be used only by cloud, and it is documented under indices.recovery.use_snapshots
.
Thanks @fcofdez and @kingherc! I will reach out to the ES distributed team to see about the aws_account
setting.
It's intended to be used only by cloud, and it is documented under indices.recovery.use_snapshots.
Should it also be documented under the repository docs?
Should it also be documented under the repository docs?
That repository setting is meant to be used only in Elastic's cloud, so I'm not sure if we should document it there.
I confirmed aws_account
is also a cloud-only repository setting used for GCP and AWS
This can be encountered outside of cloud, in any custom repo, because it is currently possible to add non-secure client settings into a repo's config. Though I do see this is being deprecated.
From ES 8.5 S3 Repo Doc:: https://www.elastic.co/guide/en/elasticsearch/reference/8.5/repository-s3.html#repository-s3
The option of defining client settings in the repository settings as documented below is considered deprecated, and will be removed in a future version.
In addition to the above settings, you may also specify all non-secure client settings in the repository settings. In this case, the client settings found in the repository settings will be merged with those of the named client used by the repository.
So users can run into this unless they remove the additional client settings from their repo, and move those settings over to the keystore
and/or elasticsearch.yml
To reproduce...
PUT _snapshot/test?verify=false
{
"type": "s3",
"settings": {
"bucket": "test-bucket",
"client": "test-client",
"base_path": "test-path",
"path_style_access": true
}
}
Kibana version: 8.4.1
Elasticsearch version: 8.4.1
Server OS version:
Browser version:
Browser OS version:
Original install method (e.g. download page, yum, from source, etc.): cloud staging
Describe the bug: Attempting to edit the
found-snapshots
repository yields:Steps to reproduce: Stack Management -> Repositories -> Edit the
found-snapshots
repository but not make any changes -> SaveExpected behavior: Editing the repository goes through successfully
Screenshots (if relevant):
Errors in browser console (if relevant):
Provide logs and/or server output (if relevant):
Any additional context: