Error with opensearch sink - basic authentication fails if EKS and AWS OS cluster are in different AWS Region

[siimaus]

Openseacrh basic authentication fails with:

 ERR Indexing failed: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [shep-prod-01-kube-events-writer] for REST request [/shep-prod-01-kube-events/_doc/6a6e1d7c-4a46-4b54-b4bb-dd9a340e6de0]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}}],"type":"security_exception","reason":"unable to authenticate user [shep-prod-01-kube-events-writer] for REST request [/shep-prod-01-kube-events/_doc/6a6e1d7c-4a46-4b54-b4bb-dd9a340e6de0]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}},"status":401}

Config for events exporter:

leaderElection: {}
logFormat: pretty
logLevel: debug
receivers:
- file:
    layout: {}
    name: stdout
    path: /dev/stdout
  name: dumpa
- name: secondary
  opensearch:
    deDot: true
    hosts:
    - https://vpc-redacted.eu-central-1.es.amazonaws.com
    index: shep-prod-01-kube-events
    password: ir*redacted*(Y5%P9#
    tls:
      insecureSkipVerify: false
    useEventID: true
    username: shep-prod-01-kube-events-writer
route:
  routes:
  - match:    
    - receiver: secondary

kubernetes-event-exporter: binary have no builtin version reporting parameter but both

• docker.io/bitnami/kubernetes-event-exporter:1.7.0-debian-12-r11
• docker.io/bitnami/kubernetes-event-exporter:1.7.0-debian-12-r13 were tried

Exactly same credentials work with curl:

export BASEURL=$(yq '.receivers[2].opensearch.hosts[0]' data/config.yaml); 
export INDEX=$(yq '.receivers[2].opensearch.index' data/config.yaml); 
export URL="$BASEURL/$INDEX/_doc/6a6e1d7c-4a46-4b54-b4bb-dd9a340e6de0"
export AUTH=$(yq '.receivers[2].opensearch | ("\(.username):\(.password)")' data/config.yaml)
curl -i "$URL" -u "$AUTH" -X POST -d '{ "message": "debug2", "@timestamp":"2024-09-20T10:00:00Z"}' -H "Content-Type: application/json"

results:

HTTP/2 200 
date: Fri, 20 Sep 2024 11:06:23 GMT
content-type: application/json; charset=UTF-8
content-length: 210
access-control-allow-origin: *

{"_index":"shep-prod-01-kube-events-2024.09.20-00001","_id":"6a6e1d7c-4a46-4b54-b4bb-dd9a340e6de0","_version":2,"result":"updated","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":63,"_primary_term":1}

Have no idea why.

[siimaus]

similar setup with same config works when Opensearch and EKS cluster are in same region..

Failing instance is located in eu-west-2 and opensearch aws managed cluster in eu-central-1

Perhaps it is related to AWS signing somehow?

[siimaus]

To be sure, I checked and indeed, if Opensearch cluster and EKS cluster are in same region, logs are sent without error.