Mimir issues when setting up S3 backend

[bzlom]

When trying to setup Mimir 2.0.7 chart with S3 backend I'm getting the following errors when spinning up mimir-ingester-*, mimir-compactor-*, mimir-ruler-*, mimir-querier-* pods in k8s:

level=info ts=2022-04-22T12:04:28.378379533Z caller=main.go:193 msg="Starting application" version="(version=2.0.0, branch=HEAD, revision=9fd2da5)"
level=info ts=2022-04-22T12:04:28.380548071Z caller=server.go:285 http=[::]:8080 grpc=[::]:9095 msg="server listening on addresses"
level=error ts=2022-04-22T12:04:28.382124024Z caller=log.go:60 msg="error running application" err="no s3 endpoint in config file\ngithub.com/thanos-io/thanos/pkg/objstore/s3.validate\n\t/__w/mimir/mimir/vendor/github.com/thanos-io/thanos/pkg/objstore/s3/s3.go:330\ngithub.com/thanos-io/thanos/pkg/objstore/s3.NewBucketWithConfig\n\t/__w/mimir/mimir/vendor/github.com/thanos-io/thanos/pkg/objstore/s3/s3.go:224\ngithub.com/grafana/mimir/pkg/storage/bucket/s3.NewBucketClient\n\t/__w/mimir/mimir/pkg/storage/bucket/s3/bucket_client.go:22\ngithub.com/grafana/mimir/pkg/storage/bucket.NewClient\n\t/__w/mimir/mimir/pkg/storage/bucket/client.go:111\ngithub.com/grafana/mimir/pkg/ingester.newIngester\n\t/__w/mimir/mimir/pkg/ingester/ingester.go:246\ngithub.com/grafana/mimir/pkg/ingester.New\n\t/__w/mimir/mimir/pkg/ingester/ingester.go:275\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).initIngesterService\n\t/__w/mimir/mimir/pkg/mimir/modules.go:442\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:120\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:426\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:195\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\nfailed to create the bucket client\ngithub.com/grafana/mimir/pkg/ingester.newIngester\n\t/__w/mimir/mimir/pkg/ingester/ingester.go:248\ngithub.com/grafana/mimir/pkg/ingester.New\n\t/__w/mimir/mimir/pkg/ingester/ingester.go:275\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).initIngesterService\n\t/__w/mimir/mimir/pkg/mimir/modules.go:442\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:120\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:426\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:195\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\nerror initialising module: ingester-service\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:122\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:426\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:195\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581"

These are the values I'm passing, I'm also relying on AWS RBAC access which is why the podAnnotations: field below:

minio:
  enabled: false

blocks_storage:
  backend: s3
  s3:
    endpoint: s3.eu-west-1.amazonaws.com
    region: eu-west-1
    bucket_name: xxx-mimir-blocks
    insecure: true

compactor:
  podAnnotations: {
            "iam.amazonaws.com/role": "xxxxxx",
  }

distributor:
  podAnnotations: {
            "iam.amazonaws.com/role": "xxxxxx",
  }

ingester:
  replicas: 1
  podAnnotations: {
            "iam.amazonaws.com/role": "xxxxx",
  }

ruler_storage:
  backend: s3
  s3:
    endpoint: s3.eu-west-1.amazonaws.com
    region: eu-west-1
    bucket_name: xxx-mimir-ruler
    insecure: true

ruler:
  podAnnotations: {
            "iam.amazonaws.com/role": "xxxxxx",
  }

alertmanager_storage:
  backend: s3
  s3:
    endpoint: s3.eu-west-1.amazonaws.com
    region: eu-west-1
    bucket_name: xxx-mimir-alertmanager
    insecure: true

alertmanager:
  podAnnotations: {
            "iam.amazonaws.com/role": "xxxxxxx"
  }

server:
  http_listen_port: 9009
  log_level: error

store_gateway:
  replicas: 1
  sharding_ring:
    kvstore:
      store: memberlist

Can you advise on what am I missing in this configuration.

[ghost]

It seems like you are about at the same place I was. See the closed issue: https://github.com/grafana/helm-charts/issues/1222 for details. My example is not for AWS but it should give you some pointers.

[bzlom]

I did precreate the S3 bucket beforehand. Doesn't really solve my issue unfortunatelly.

[cgill27]

If you are using EKS, then in your podAnnotations use this: eks.amazonaws.com/role-arn Make sure you have IRSA role configured in IAM/EKS

[HeshamMeneisi]

You only need the annotation on the SA when using EKS.

  serviceAccount:
    create: true
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::xyz:role/mimir

This with a typical TF IRSA module works just fine.

[dpreston-wss]

Currently having this issue attempting to deploy on prem via argo. Was there anything obvious that was missed?