search

thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
https://thanos.io
Apache License 2.0
13.1k stars 2.1k forks source link

Support random delay before upload blocks on sidecar #7184

Open Wing924 opened 8 months ago

Wing924 commented 8 months ago

Is your proposal related to a problem?

Currently, all sidecars upload blocks at the same time. It make huge bandwidth usage to object storage.

It will be a problem when you run sidecars in on-premises and use public cloud object storage because the dedicated line has limited bandwidth.

Describe the solution you'd like

I propose add some random delay before uploading blocks.

Let's dd the --shipper.upload-jitter flag. When sidecar upload blocks, it sleep random time between 0 and jitter.

Describe alternatives you've considered

I tried to manage QoS, but it's very hard to config.

yeya24 commented 8 months ago

I think it is ok to add some jitter when uploading blocks. The question is, do we add jitter to the shipper itself? I understand we want to jitter the caller loop. For example, in sidecar we will add it here.

https://github.com/thanos-io/thanos/blob/main/cmd/thanos/sidecar.go#L365

Vanshikav123 commented 8 months ago

Hello ! @yeya24 can i work on this issue?