AccessDenied: User: arn:aws:sts::XXXXX:assumed-role/experimental-nodes/i-07c0b8a749aca2fce is not authorized to perform

[krimeshshah]

Describe the bug I have deployed loki 3.0 in simple-scalable deployment mode with s3 object storage and tsdb-shipper as index store. All the components write, read and backed with other pods are deployed and running fine. But write components is unable to write the logs coming from promatil to s3 bucket. I have condifured s3 bucket as per the document. Below is the error logs in loki-write pod

level=error ts=2024-06-12T15:29:42.120346177Z caller=flush.go:152 component=ingester org_id=exp msg="failed to flush" err="failed to flush chunks: store put chunk: AccessDenied: User: arn:aws:sts::355291130670:assumed-role/shoot--lp--experimental-nodes/i-07c0b8a749aca2fce is not authorized to perform: kms:GenerateDataKey on this resource because the resource does not exist in this Region, no resource-based policies allow access, or a resource-based policy explicitly denies access\n\tstatus code: 403, request id: 4FYDTGK963QK58Z0, host id: C7BSrGUr9eFPHUvtQD/FdCU+QNMUUmXI9YyiVK0PBi6+eHdD5pHtTRmMVLz0yB/WiY++zQ/AYRB2YbatL0APn6Zk8KCllUtT0ElM5jLDr7o=, num_chunks: 1, labels: {app=\"promtail\", container=\"install-oneagent\", filename=\"/var/log/pods/promtail_promtail-jkxbf_80fc9e90-e689-417e-8c63-52223cba1b08/install-oneagent/0.log\", instance=\"promtail\", job=\"promtail/promtail\", namespace=\"promtail\", node_name=\"ip-10-250-13-185.eu-central-1.compute.internal\", pod=\"promtail-jkxbf\", service_name=\"promtail\", stream=\"stdout\"}"

To Reproduce Steps to reproduce the behavior:

1. Started Loki (SHA or version) - loki 3.0
2. Started Promtail (SHA or version) to tail 'promtail 2.9
3. Query: {} term

Expected behavior loki write should be able to flush and write the logs to s3 bucket

Environment:

• Infrastructure: kubernetes
• Deployment tool: helm

Screenshots, Promtail config, or terminal output If applicable, add any output to help explain your problem.

[krimeshshah]

And here is my configuration with loki s3 storage

`sealedSecrets: enabled: true encrypted_s3accessid: encrypted_s3secretkey:

storageRequest: size: 20Gi

loki: loki: persistence: enabled: true size: 10Gi existingClaim: loki-pv-claim extraArgs:

• '-config.expand-env=true' env:
• name: S3_ACCESS_KEY_id valueFrom: secretKeyRef: key: loki_accesskeyid name: loki-iam-secret

• name: S3_SECRET_ACCESS_KEY valueFrom: secretKeyRef: key: loki_secretkey name: loki-iam-secret storage: bucketNames: chunks: loki-plg-preprod ruler: loki-plg-preprod admin: loki-plg-preprod type: s3 s3: s3: s3 bucketName: loki-plg-preprod region: eu-central-1 secretAccessKey: ${S3_ACCESS_KEY_id} accessKeyId: ${S3_SECRET_ACCESS_KEY} s3ForcePathStyle: false insecure: false schemaConfig: configs:

  • from: 2024-06-12 store: tsdb object_store: s3 schema: v13 index: prefix: lokiindex period: 24h storage_config:

    tsdb_shipper:

    active_index_directory: /data/tsdb-index

    cache_location: /data/tsdb-cache

    cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space

    aws: s3: s3://eu-central-1 bucketnames: loki-plg-preprod access_key_id: ${S3_ACCESS_KEY_id} secret_access_key: ${S3_SECRET_ACCESS_KEY} ingester: chunk_encoding: snappy tracing: enabled: true querier:

    Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing

    max_concurrent: 4

  deploymentMode: SimpleScalable

  backend: replicas: 1 extraArgs:

• '-config.expand-env=true' read: replicas: 1 extraArgs:
• '-config.expand-env=true' write: replicas: 2 extraArgs:
• '-config.expand-env=true'
  env:
• name: S3_ACCESS_KEY_id valueFrom: secretKeyRef: key: loki_accesskeyid name: loki-iam-secret

• name: S3_SECRET_ACCESS_KEY valueFrom: secretKeyRef: key: loki_secretkey name: loki-iam-secret

  singleBinary: replicas: 0 chunksCache: enabled: false

  ingester: replicas: 0 querier: replicas: 0 queryFrontend: replicas: 0 queryScheduler: replicas: 0 distributor: replicas: 0 compactor: replicas: 0 indexGateway: replicas: 0 bloomCompactor: replicas: 0 bloomGateway: replicas: 0``

[JStickler]

Questions have a better chance of being answered if you ask them on the community forums.