[BUG] Opensearch does not refresh AWS Web Identity Token

Describe the bug

2.13 opensearch version ( 2.19.0 opensearch helm chart ) Opensearch does not refresh AWS Web Identity Token, so repository-s3 fails to take a snapshot after some time - when AWS Web Identity Token expires. The only way to refresh it and to make snapshot working is to restart all master and data nodes.

Related component

Plugins

To Reproduce

config for master nodes (similar for data):

opensearch:
  plugins:
    enabled: true
    installList:
      …
      - ‘repository-s3’
  extraEnvs:
    …
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: ‘/usr/share/opensearch/config/irsa-token’
    - name: AWS_ROLE_ARN
      value: 'arn:aws:iam::account_id:role/role_name'
  extraVolumeMounts:
    - mountPath: /usr/share/opensearch/config/irsa-token
      name: aws-iam-token
      readOnly: true
      subPath: token
  rbac:
    create: true
    serviceAccountName: opensearch-master
    serviceAccountAnnotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::account_id:role/role_name
  config:
    s3.client.default.region: us-west-2
    s3.client.default.endpoint: s3.us-west-2.amazonaws.com
    s3.client.default.identity_token_file: /usr/share/opensearch/config/irsa-token
  keystore:
    - secretName: opensearch-keystore
  extraObjects:
    - apiVersion: v1
      kind: Secret
      metadata:
        name: opensearch-keystore
      type: Opaque
      data:
        s3.client.default.role_arn: #role_arn
        s3.client.default.role_session_name: #session_name

Expected behavior

_PUT 'https://master_node/snapshot/repository_name/snapshot_id?wait_for_completion=true_ should work

Additional Details

Plugins repository-s3

Screenshots “Token expired: current date/time 1730190842 must be before the expiration date/time 1729863829 (Service: Sts, Status Code: 400, Request ID…”

Host/Environment (please complete the following information):

Version [2.13]

Additional context Official doc is missing option for S3 repository plugin using AWS Web Identity Token for Helm Chart deployments. So we did config similar to 7982.

opensearch-project / OpenSearch