[Question] Using IRSA for Opensearch Authentication Failed

[vumdao]

I use opensearch and use IAM role as service account for authorzing

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: monitoring
  name: event-exporter
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/EksCluster-eksblueprint-YSE1WGEEZD3K

Config

apiVersion: v1
kind: ConfigMap
metadata:
  name: event-exporter-cfg
  namespace: monitoring
data:
  config.yaml: |
    logLevel: error
    logFormat: json
    route:
      routes:
        - match:
            - receiver: "dump"
    receivers:
    - name: "dump"
      opensearch:
        hosts:
          - https://dev-es.cloudopz.co
        index: kube-events
        indexFormat: "kube-events-{2006-01-02}"
        useEventID: true
        tls: # optional, advanced options for tls
          insecureSkipVerify: false # optional, if set to true, the tls cert won't be verified

Error

{"level":"error","time":"2022-07-29T16:26:19Z","caller":"/app/pkg/sinks/opensearch.go:139","message":"Indexing failed: Unauthorized"}
{"level":"error","time":"2022-07-29T16:26:19Z","caller":"/app/pkg/sinks/opensearch.go:139","message":"Indexing failed: Unauthorized"}
{"level":"error","time":"2022-07-29T16:26:19Z","caller":"/app/pkg/sinks/opensearch.go:139","message":"Indexing failed: Unauthorized"}
{"level":"error","time":"2022-07-29T16:26:19Z","caller":"/app/pkg/sinks/opensearch.go:139","message":"Indexing failed: Unauthorized"}
{"level":"error","time":"2022-07-29T16:26:19Z","caller":"/app/pkg/sinks/opensearch.go:139","message":"Indexing failed: Unauthorized"}
I0729 16:26:21.160973       1 request.go:665] Waited for 1.197094422s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/authentication.k8s.io/v1?timeout=32s
{"level":"error","time":"2022-07-29T16:26:29Z","caller":"/app/pkg/sinks/opensearch.go:139","message":"Indexing failed: Unauthorized"}
{"level":"error","time":"2022-07-29T16:28:39Z","caller":"/app/pkg/sinks/opensearch.go:139","message":"Indexing failed: Unauthorized"}
{"level":"error","time":"2022-07-29T16:28:39Z","caller":"/app/pkg/sinks/opensearch.go:139","message":"Indexing failed: Unauthorized"}
I0729 16:28:41.320471       1 request.go:665] Waited for 1.190263149s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/events.k8s.io/v1?timeout=32s
{"level":"error","time":"2022-07-29T16:28:50Z","caller":"/app/pkg/sinks/opensearch.go:139","message":"Indexing failed: Unauthorized"}

[mustafaakin]

Currently, the support for the request signer is not automatic. We need to add following code which should be trivial to add:

https://github.com/opensearch-project/opensearch-go/blob/main/USER_GUIDE.md#how-to-use-iams-as-authentication-method

[vumdao]

@mustafaakin I just thought this tool is like aws-for-fluent-bit which I just need to specify the IAM role ARN as the IRSA (IAM role for serviceAccount).

Without this support, we must use user/password which is treated as anonymous user from opensearch and from that the access policy must allow as bellow

 {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": [
        "es:ESHttp*
      ],
      "Resource": "arn:aws:es:ap-south-1:*:domain/dev-opensearch/*"
 }

[paulbraham-ds]

Just wondering if this is on the roadmap at all? We have the same use case and would prefer not to use username/password if possible.

[paulbraham-ds]

Just in case anyone else comes across this. I was able to use the sigv4 proxy admission controller to inject a sidecar that was capable of handling the Amazon request signing. It works perfectly with IRSA. https://github.com/aws-observability/aws-sigv4-proxy-admission-controller

[arvind-v]

Thanks for the suggestion to use sigv4 proxy, but I am running into issues getting sigv4 to work with Amazon OpenSearch Serverless (AOSS). I used the underlying aws-sigv4-proxy instead of aws-sigv4-proxy-admission-controller, because aws-sigv4-proxy provides more granular configuration. An IRSA is setup and grants the pod access to write to OpenSearch Serverless.

Here is my helm config.yaml:

config:
  route:
    routes:
      - match:
          - receiver: opensearch-receiver
  receivers:
    - name: "opensearch-receiver"
      opensearch:
        index: my-kubernetes-events
        hosts:
          - http://localhost:9000

sidecars:
  - name: aws-sigv4-proxy-sidecar
    image: public.ecr.aws/aws-observability/aws-sigv4-proxy:latest
    imagePullPolicy: IfNotPresent
    ports:
      - containerPort: 9000    
    args:
      - --host
      - aaa
      - --port
      - :9000
      - --region
      - bbbb
      - --name
      - aoss
      - -v

serviceAccount:
  name: events-aoss
  create: true
  automountServiceAccountToken: true  
  annotations:
    "eks.amazonaws.com/role-arn": cccc

Proxy logs show that requests are getting to the OpenSearch host (only GET methods, which is strange):

time="2024-09-16T03:48:56Z" level=debug msg="signed request" region=us-west-2 service=aoss
...
time="2024-09-16T03:48:56Z" level=error msg="error proxying request" message= request="GET https://myaosshost.us-west-2.aoss.amazonaws.com/" status_code=404

Here is a snippet of logs from the exporter pod:

{"level":"debug","sink":"opensearch-receiver","event":"Created pod: events-to-opensearch-kubernetes-event-exporter-5bdb56b6b-77g5x","time":"2024-09-16T03:48:56Z","message":"sending event to sink"}
{"level":"debug","error":"cannot retrieve information from OpenSearch","sink":"opensearch-receiver","event":"Created pod: events-to-opensearch-kubernetes-event-exporter-5bdb56b6b-77g5x","time":"2024-09-16T03:48:56Z","message":"Cannot send event"}
{"level":"debug","sink":"opensearch-receiver","event":"No matching pods found","time":"2024-09-16T03:48:56Z","message":"sending event to sink"}
{"level":"debug","error":"cannot retrieve information from OpenSearch","sink":"opensearch-receiver","event":"No matching pods found","time":"2024-09-16T03:48:56Z","message":"Cannot send event"}
{"level":"debug","sink":"opensearch-receiver","event":"Scaled up replica set events-to-opensearch-kubernetes-event-exporter-5bdb56b6b to 1","time":"2024-09-16T03:48:56Z","message":"sending event to sink"}
{"level":"debug","error":"cannot retrieve information from OpenSearch","sink":"opensearch-receiver","event":"Scaled up replica set events-to-opensearch-kubernetes-event-exporter-5bdb56b6b to 1","time":"2024-09-16T03:48:56Z","message":"Cannot send event"}