Open joyfulwang opened 6 months ago
@joyfulwang Have you mapped the EMR job's IAM role to a internal user within ElasticSearch/OpenSearch according to the documentation here: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac.html#fgac-access-control
Hi, my team is adding SigV4 signing to all of the read/write requests that our resources send to Elasticsearch. We've successfully added signing to requests from our backend Java service and to a Lambda function. We're now trying to add signing to our EMR Spark jobs, which are using emr-6.6.0, Spark 3.x, Scala 2.12, and opensearch-hadoop (Maven-org-opensearch-client_opensearch-spark-30_2_12). The Elasticsearch cluster is version 7.10
After reading the opensearch-hadoop User Guide and the Configuration Options for Maven-org-opensearch-client_opensearch-spark, I updated our OpenSearch config to the following
After enabling the SigV4 signing config, I tested if the Spark job could read index names from a cluster that has fine-grained access control enabled and got "Unauthorized" as the response. Here's what I've tried for troubleshooting:
Any ideas for troubleshooting?