search

thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
https://thanos.io
Apache License 2.0
12.73k stars 2.04k forks source link

AWS S3 objectstorage is not working for ap-south-2 region inspite of endpoint and region is mentioned and getting default to dualstack endpoint of us-east-1 #7427

Open svvsmarutii opened 2 weeks ago

svvsmarutii commented 2 weeks ago

Thanos, Prometheus and Golang version used: 0.30.2

Object Storage Provider: AWS S3

What happened: Thanos sidecar, compact, query, storegateway components are trying to reach the S3 bucket in us-east-1 region inspite ap-south-2 region and s3 endpoint is set.

What you expected to happen: Thanos components should reach S3 bucket to the region(ap-south-2) as mentioned in the thanos helm chart

How to reproduce it (as minimally and precisely as possible): Deploy the thanos helm chart with below objectstorage.yaml configuration objstoresecret: type: s3 prefix: thanos config: bucket: "dev-dr-thanos-metrics" region: ap-south-2 endpoint: s3.ap-south-2.amazonaws.com

Full logs to relevant components: image

Anything else we need to know: Able to see the s3 endpoint of ap-south-2 is supported by thanos https://docs.aws.amazon.com/general/latest/gr/s3.html

harry671003 commented 2 weeks ago

I think this issue was raised in: https://github.com/thanos-io/thanos/issues/6541 You should probably use a more newer version of Thanos for the minio update to be included.

Sahana-Nagaraja commented 2 weeks ago

Hi @harry671003, I am a colleague of Maruthi. We have upgraded Thanos to 0.35.1 and now we see that Thanos is pointing to right region. But, it is trying to access dualstack endpoint of S3 bucket which is not available yet. We did not manage to find a parameter to set it to use standalone S3 endpoint in Thanos helm chart. Can you please help us here if you have a handy solution?

douglascamata commented 2 weeks ago

Sounds like this was fixed in the objstore lib (see https://github.com/thanos-io/objstore/pull/112) by allowing the dualstack endpoint to be disabled. So we only need to bump the dependency here in Thanos.

svvsmarutii commented 2 weeks ago

Hi @douglascamata Thanks for update. Is it possible to provide an ETA for this change to be available in thanos ? Also, is there a quick workaround that we could do to leverage this option ?

harry671003 commented 1 week ago

https://github.com/thanos-io/thanos/pull/7462 - Update objstore

harry671003 commented 1 week ago

My PR to update the objstore version in Thanos got merged. @svvsmarutii - could you try testing with the latest image of Thanos? As douglascamata pointed out, setting disable_dualstack: true in the config should fix this problem.

douglascamata commented 1 week ago

@svvsmarutii we don't provide ETAs for changes, as it depends on the "free work" of maintainers that might be busy with their jobs. But, as the PR was already merged, it should be going out in the next Thanos release. If you would like to have a bugfix release, please reach out in #thanos-dev in the CNCF slack.