AWS S3 storage: failed to CAS cluster seed key / failed to flush user

[EmptyLungs]

I'm trying to install Loki with Helm:

resource "helm_release" "loki" {
  repository = "https://grafana.github.io/helm-charts"
  chart      = "loki-stack"
  version    = "2.8.7"

  namespace = kubernetes_namespace.loki.metadata[0].name
  name      = "loki"

  values = [
    "${file("./12-loki-chart-values.yaml")}"
  ]
  set {
    name = "loki.serviceAccount.name"
    value = "loki"
  }
  set {
    name = "loki.serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = "${aws_iam_role.loki_oidc.arn}"
  }
}

I've tested this role and SA - works fine with aws-cli pod in a same namespace.

values.yaml and aws role

``` loki: auth_enabled: false config: schema_config: configs: - from: 2022-12-07 store: boltdb-shipper object_store: s3 schema: v11 index: prefix: loki_index_ period: 24h compactor: working_directory: /tmp/loki/compactor shared_store: s3 common: ring: kvstore: store: inmemory path_prefix: /tmp/loki storage: s3: bucketnames: logging-storage-loki region: ap-northeast-1 access_key_id: null secret_access_key: null storage_config: aws: s3: s3://ap-northeast-1/logging-storage-loki s3forcepathstyle: true bucketnames: logging-storage-loki region: ap-northeast-1 insecure: false sse_encryption: false boltdb_shipper: shared_store: s3 cache_ttl: 24h prometheus: enabled: false test: enabled: false grafana: enabled: true sidecar: datasources: enabled: true image: tag: latest ``` ``` resource "aws_iam_policy" "loki_policy" { name = "AWSEKSLokiPolicy" description = "Allow single bucket for EKS Loki" policy = jsonencode({ Statement = [{ Action = [ "s3:ListAllMyBuckets", "s3:GetBucketLocation", "s3:DeleteObject", "s3:GetObjectAttributes", "s3:PutObject", "s3:ListBucket", "s3:GetObject" ] Effect = "Allow" Resource = aws_s3_bucket.loki_logs.arn }] Version = "2012-10-17" }) } ```

Loki itself seems to work fine, but it doesn't sync with the s3 bucket with such logs from Loki statefulset:

level=info ts=2022-12-08T06:31:39.214477811Z caller=reporter.go:125 msg="failed to CAS cluster seed key" err="AccessDenied: Access Denied\n\tstatus code: 403, request id: 17AWM76ZH08F0147, host id: g68IiVtz8NB+3heBflAJiz9Lj1qxBtVqar7yHCrh6MxOfZVHqreU78pSxXm1kASH2sIlPXDTh4A="

level=error ts=2022-12-08T06:32:06.422391507Z caller=flush.go:146 org_id=fake msg="failed to flush user" err="store put chunk: AccessDenied: Access Denied\n\tstatus code: 403, request id: FV2RAXT88XW29SSZZ, host id: 3MsfR8dX389HrIeaHCVRpIbCvEVWLvHdXF68dRH/x92iK9By9dNxj++MuH2ywOGSg4xFgvCV01g="

[EmptyLungs]

The problem was with aws policy :disappointed:

[mossad-zika]

@EmptyLungs why not be more precisely?

[FotiadisM]

@EmptyLungs facing the exact same issue, could be more specific, what was the policy error?

[tompscanlan]

For future stumblers: S3 permissions: https://grafana.com/docs/loki/latest/operations/storage/#s3