Metricbeat is unable to connect to AWS OpenSearch service. ERROR 401 Unauthorized

[nagachinni]

Metricbeat running on EC2 is failing with error: unable to connect to AWS OpenSearch service. ERROR 401 Unauthorized My openSearch domain enabled fine-grain access control(FGAC) with IAM role as master user. I have the access policy attached to EC2 IAM role already and mapped the opensearch role "all_access" so it should reach the domain endpoint. However, this is not working.

To Reproduce Create AWS OpenSearch domain and enable FGAC. Create an EC2 instance and attach the IAM policy to allow access to opensearch domain created above.

"Effect": "Allow",
"Action": [
    "es:*"
 ],
 "Resource": [
     "<domainarn>"
]

Update opensearch domain access policy as below.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": [
        "es:*"
      ],
      "Resource": "<domain_arn>"
    }
  ]
}

Install metricbeat-oss7.12.1 on EC2 instance and configure to push metrics to opensearch domain endpoint. You'll receive error as stated above when running metricbeat.

Expected behavior Metricbeat should be able to push metrics to opensearch endpoint.

Additional Details Host/Environment (please complete the following information):

OS: Windows

[botelastic[bot]]

This issue doesn't have a Team:<team> label.

[nagachinni]

Does metricbeat support AWS Sigv4 authentication? If so can someone please help how to enable this in the config?

[nagachinni]

Hello Guys, if someone could help me with this would be great.!!

[g-bohncke]

https://github.com/awslabs/aws-sigv4-proxy

A Temp way around the issue