Open DaveCTurner opened 7 months ago
Pinging @elastic/es-distributed (Team:Distributed)
I'm running into this at work, where we're required to use IMDSv2. I'm glad I found the recent post by strophy; I've been pulling my hair out on this as well. I'd also like this info be added to the documentation (that only v1 is currently supported), to save others from the frustration, at least while v2 support is being added.
We are also interested in IMDSv2 support for the repository-s3 plugin.
Any updates on when repository-s3 will support IMDSv2 ? Is there a timeline for this?. Does anyone know of a technical representative who we can contact?
We also need it here. Any updates?
I spend some time trying to understand the reason why backup (and any other operations related to S3) fail in one of our clusters. Unfortunately, we are using a old version of ES (6.3) and repository-s3 plugin (6.3.2). The response is quit confuse:
{
"error": {
"root_cause": [
{
"type": "amazon_service_exception",
"reason": "Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)"
}
],
"type": "amazon_service_exception",
"reason": "Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)"
},
"status": 500
}
The log is also limited
[WARN ][r.suppressed ] path: /_cat/snapshots/my_repo, params: {s=id, v=, repository=my_repo}
com.amazonaws.AmazonServiceException: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)
at com.amazonaws.internal.EC2CredentialsUtils.handleErrorResponse(EC2CredentialsUtils.java:156) ~[?:?]
Basically, before try to access the S3 bucket, elasticsearch instance try to get their our profile using metadata API. The response is an 401, since IMDSv2 required a valid token before access the API data. This 401 make the elasticsearch operation fail and return the errors listed above. The only solution for make it work is disable IMDSv2 requirement for now.
curl -i 169.254.169.254/latest/meta-data/iam/security-credentials/
HTTP/1.1 401 Unauthorized
I hope that this can be useful for anyone facing this "issue".
I (@davecturner) have hidden this comment because it (a) relates to an extremely old version of ES and (b) suggests some very risky actions to modify the Elasticsearch installation. We strongly recommend not running versions of ES which have passed EOL, and definitely discourage the other risky actions mentioned here.
I (@davecturner) have hidden this comment because it (a) relates to an extremely old version of ES and (b) suggests some very risky actions to modify the Elasticsearch installation. We strongly recommend not running versions of ES which have passed EOL, and definitely discourage the other risky actions mentioned here.
Today we support IMDSv2 in the
discovery-ec2
plugin (see https://github.com/elastic/elasticsearch/pull/84410) butrepository-s3
still only supports IMDSv1. Should we add IMDSv2 support torepository-s3
?