search

thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
https://thanos.io
Apache License 2.0
12.94k stars 2.08k forks source link

thanos sidecar no upload #5193

Open tnrms007 opened 2 years ago

tnrms007 commented 2 years ago

hi all

Is it possible to run two thanos sidecars on prometheus server and stack them on two object storage?? I've tried running it with different port numbers, but only one is being uploaded.

matej-g commented 2 years ago

Hey @tnrms007, could you maybe explain what are you trying to achieve with running two sidecars?

tnrms007 commented 2 years ago

Hey @tnrms007, could you maybe explain what are you trying to achieve with running two sidecars?

hi @matej-g, I want to store data from Prometheus server in different cloud object storage. ex) aws s3 , ibm object storage

matej-g commented 2 years ago

I'm not sure if having two sidecars is a common way to achieve this, but have you had a look at Thanos tools bucket utility, which has the command to replicate data to another object store? Would that help with you use case?

tnrms007 commented 2 years ago

I'm not sure if having two sidecars is a common way to achieve this, but have you had a look at Thanos tools bucket utility, which has the command to replicate data to another object store? Would that help with you use case?

@matej-g, I want to send prometheus realtime data (2h-blockfile) to two different object storages. This is a different function from bucket replica.

yeya24 commented 2 years ago

I want to send prometheus realtime data (2h-blockfile) to two different object storages. This is a different function from bucket replica.

Actually bucket replicate is also real-time in this case. Blocks are just uploaded once every 2 hours while bucket replicate will run every 5m (or other configurable time) which is sufficient in this case.

Is it possible to run two thanos sidecars on prometheus server and stack them on two object storage?? I've tried running it with different port numbers, but only one is being uploaded.

Again to answer this question, I guess this doesn't work out of box. Because two sidecars will use the same shipper.json file. That file is basically tracking which block has been uploaded to the object store. If one sidecar uploads the block and updates this file, then another sidecar will stop uploading it. Or if they start uploading at the same time, then it might work. But this way is hacky and we recommend using bucket replicate for such cases.

stale[bot] commented 1 year ago

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.