sidecar: err="check exists: stat s3 object: Access Denied."

[whipermr5]

Thanos, Prometheus and Golang version used: Thanos 0.28.1, Prometheus 2.39.1, Golang 1.18.7

Object Storage Provider: AWS S3 (ping @bwplotka)

What happened: Started seeing warning logs err="check exists: stat s3 object: Access Denied." uploaded=0; blocks not being uploaded to S3

What you expected to happen: Blocks can be uploaded to S3 successfully

Anything else we need to know: Working fine in Thanos 0.27.0. The issue seems to affect others too - see https://github.com/thanos-io/thanos/issues/3677#issuecomment-1305798381. Creating a new issue as that one was for a previous version and already closed.

[flashmann]

facing the same (or very similar) issue being not able to connect to S3 bucket starting from thanos version 0.28.1 (tested also 0.29.0), last one which works correctly is 0.28.0, using IAM on AWS EKS Error in the log: err="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied"

[rbudiharso]

Experiencing this also, I already make sure all permission for accessing S3 is present but still access denied. Any pointer would be appreciated.

[maso7]

After some troubleshooting we found out possible root cause of the problem: https://github.com/minio/minio-go/commit/fe4dc656657288125addc6b3be2f629376881075

We experience problem only with cross account bucket access on AWS. Working scenarios for us:

• Use Assume role on EKS cluster.
• Setting "aws_sdk_auth: true" on clusters where is not possible to use assume role.

Tested with version 0.30.1

[GNSunny]

I can see this issue with 0.30.2 only working version is 0.28.0

gave all the necessary permissions to s3 also followed this commit https://github.com/thanos-io/thanos/issues/5996#issuecomment-1366556738 at https://github.com/thanos-io/thanos/issues/5996 but none of the options makes the access denied error disappear except this version 0.28.0

level=info caller=factory.go:52 msg="loading bucket configuration" level=info caller=inmemory.go:174 msg="created in-memory index cache" maxItemSizeBytes=131072000 maxSizeBytes=262144000 maxItems=maxInt level=info caller=options.go:26 protocol=gRPC msg="disabled TLS, key and cert must be set to enable" level=info caller=store.go:469 msg="starting store node" level=info caller=store.go:382 msg="initializing bucket store" level=info caller=intrumentation.go:75 msg="changing probe status" status=healthy level=info caller=http.go:73 service=http/server component=store msg="listening for requests and metrics" address=0.0.0.0:10902 level=info caller=tls_config.go:232 service=http/server component=store msg="Listening on" address=[::]:10902 level=info caller=tls_config.go:235 service=http/server component=store msg="TLS is disabled level=warn caller=intrumentation.go:67 msg="changing probe status" status=not-ready reason="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied" level=info caller=http.go:91 service=http/server component=store msg="internal server is shutting down" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied" level=info caller=intrumentation.go:56 msg="changing probe status" status=ready level=info caller=http.go:110 service=http/server component=store msg="internal server is shutdown gracefully" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied" level=info caller=intrumentation.go:81 msg="changing probe status" status=not-healthy reason="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied" level=info caller=grpc.go:131 service=gRPC/server component=store msg="listening for serving gRPC" address=0.0.0.0:10901 level=warn caller=intrumentation.go:67 msg="changing probe status" status=not-ready reason="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied" level=info caller=grpc.go:138 service=gRPC/server component=store msg="internal server is shutting down" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied" level=info caller=grpc.go:151 service=gRPC/server component=store msg="gracefully stopping internal server" level=info caller=grpc.go:164 service=gRPC/server component=store msg="internal server is shutdown gracefully" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied"

much appreciated if there is any update on this.

[carlosrmendes]

I'm getting the same error on store-gateway v0.31.0, in EKS, using IAM role in service account.

[amohamedhey]

I'm getting the same error on store-gateway v0.31.0, in Kops, using IAM role in service account.

[roelywoely]

I'm facing the same issue. I can confirm that 0.28.0 is the latest working version. Version 0.28.1 and above are broken.

[maso7]

Try to extend you objstore config with "aws_sdk_auth: true". It's working for us with any version after 0.28.0 we had.

Example: --objstore.config={type: S3, config: {bucket:<bucket name>, region: eu-west-1, aws_sdk_auth: true, endpoint: s3.eu-west-1.amazonaws.com, put_user_metadata: {\"X-Amz-Acl\": \"bucket-owner-full-control\"}}}

[Pluies]

Not exactly Thanos itself, but still potentially useful – if you're hitting this issue after upgrading Bitnami's Thanos chart to v0.13, this might be because of the removal of the existingServiceAccount option. Service accounts must now be defined on a per-service basis.

[guliziskender]

still having the issue wit version 0.28.0 and 0.28.1