search

thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
https://thanos.io
Apache License 2.0
12.91k stars 2.07k forks source link

compact: Redesign compaction planning process for cost efficiency and determinism. #3405

Open bwplotka opened 3 years ago

bwplotka commented 3 years ago

Related discussion: https://matrix.to/#/!WaUKIfoqfiyWQhenET:matrix.org/$16044993991172247rFDCZ:matrix.org?via=matrix.org&via=hoffie.info

Problems of the Current Algorithm

  1. The current algorithm is not very efficient for the cost (e.g network bandwidth). For example, if you have blocks laid out as 2h 2h 2h 2h 2h 2h 2h 2h 2h 2h 2h 2h 2h (301 of those in total), our current planning with comp range e.g [2h, 8h, 2d, 25d] will pick the oldest 4x2h to compact 8h and so on, then same for 2d only to at then do a single block of 25d. This means 88 compactions instead of just SINGLE compaction 300x2h blocks into single 25d. Vertical compactions are totally excluded as separate compactions so it's even worse.

  2. It's currently (without locking mechanism) hard to run multiple compactors on a single stream. It would be nice to apply some determinism to allow such compactor sharding across a single stream. This is however lower priority goal.

This is related to https://github.com/thanos-io/thanos/pull/3390

bwplotka commented 3 years ago

Potentially blocker as bigger compactions would require more resources: https://github.com/thanos-io/thanos/issues/3406

stale[bot] commented 3 years ago

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

yeya24 commented 3 years ago

Still valid

stale[bot] commented 3 years ago

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

bwplotka commented 3 years ago

Very valid

stale[bot] commented 3 years ago

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

stale[bot] commented 3 years ago

Closing for now as promised, let us know if you need this to be reopened! 🤗

stale[bot] commented 2 years ago

Closing for now as promised, let us know if you need this to be reopened! 🤗

yeya24 commented 2 years ago

Still valid

stale[bot] commented 2 years ago

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

vjabrayilov commented 2 years ago

Hey, dear @yeya24 and @bwplotka. The proposal that I attached to #3406, tries to solve this issue as well. https://drive.google.com/file/d/1oZd3ENSZ7v2hONNf4pDL3yXr_0eHcwsV/view?usp=sharing

Looking forward your feedback. Best.

bwplotka commented 2 years ago

Sweet, thanks, adding to my Todo list.

stale[bot] commented 2 years ago

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

Vanshikav123 commented 5 months ago

Hello ! @bwplotka @yeya24 is this issue still relevant?

calestyo commented 5 months ago

Sure it is, I stumbled just few days ago over the same problem, which is why I had reported #7198.

PS: Since #7198 is closed now as a duplicate of this issue here, people my also want to read the symptoms/etc. I've had reported there,... should anyone ever work on this issue.