[Question] Using IRSA for Opensearch Authentication Failed

[vumdao]

I use opensearch and use IAM role as service account for authorzing

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: monitoring
  name: event-exporter
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/EksCluster-eksblueprint-YSE1WGEEZD3K

Config

apiVersion: v1
kind: ConfigMap
metadata:
  name: event-exporter-cfg
  namespace: monitoring
data:
  config.yaml: |
    logLevel: error
    logFormat: json
    route:
      routes:
        - match:
            - receiver: "dump"
    receivers:
    - name: "dump"
      opensearch:
        hosts:
          - https://dev-es.cloudopz.co
        index: kube-events
        indexFormat: "kube-events-{2006-01-02}"
        useEventID: true
        tls: # optional, advanced options for tls
          insecureSkipVerify: false # optional, if set to true, the tls cert won't be verified

Error

{"level":"error","time":"2022-07-29T16:26:19Z","caller":"/app/pkg/sinks/opensearch.go:139","message":"Indexing failed: Unauthorized"}
{"level":"error","time":"2022-07-29T16:26:19Z","caller":"/app/pkg/sinks/opensearch.go:139","message":"Indexing failed: Unauthorized"}
{"level":"error","time":"2022-07-29T16:26:19Z","caller":"/app/pkg/sinks/opensearch.go:139","message":"Indexing failed: Unauthorized"}
{"level":"error","time":"2022-07-29T16:26:19Z","caller":"/app/pkg/sinks/opensearch.go:139","message":"Indexing failed: Unauthorized"}
{"level":"error","time":"2022-07-29T16:26:19Z","caller":"/app/pkg/sinks/opensearch.go:139","message":"Indexing failed: Unauthorized"}
I0729 16:26:21.160973       1 request.go:665] Waited for 1.197094422s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/authentication.k8s.io/v1?timeout=32s
{"level":"error","time":"2022-07-29T16:26:29Z","caller":"/app/pkg/sinks/opensearch.go:139","message":"Indexing failed: Unauthorized"}
{"level":"error","time":"2022-07-29T16:28:39Z","caller":"/app/pkg/sinks/opensearch.go:139","message":"Indexing failed: Unauthorized"}
{"level":"error","time":"2022-07-29T16:28:39Z","caller":"/app/pkg/sinks/opensearch.go:139","message":"Indexing failed: Unauthorized"}
I0729 16:28:41.320471       1 request.go:665] Waited for 1.190263149s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/events.k8s.io/v1?timeout=32s
{"level":"error","time":"2022-07-29T16:28:50Z","caller":"/app/pkg/sinks/opensearch.go:139","message":"Indexing failed: Unauthorized"}

[mustafaakin]

Currently, the support for the request signer is not automatic. We need to add following code which should be trivial to add:

https://github.com/opensearch-project/opensearch-go/blob/main/USER_GUIDE.md#how-to-use-iams-as-authentication-method

[vumdao]

@mustafaakin I just thought this tool is like aws-for-fluent-bit which I just need to specify the IAM role ARN as the IRSA (IAM role for serviceAccount).

Without this support, we must use user/password which is treated as anonymous user from opensearch and from that the access policy must allow as bellow

 {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": [
        "es:ESHttp*
      ],
      "Resource": "arn:aws:es:ap-south-1:*:domain/dev-opensearch/*"
 }

[paulbraham-ds]

Just wondering if this is on the roadmap at all? We have the same use case and would prefer not to use username/password if possible.

[paulbraham-ds]

Just in case anyone else comes across this. I was able to use the sigv4 proxy admission controller to inject a sidecar that was capable of handling the Amazon request signing. It works perfectly with IRSA. https://github.com/aws-observability/aws-sigv4-proxy-admission-controller