search

thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
https://thanos.io
Apache License 2.0
12.74k stars 2.04k forks source link

Document rough guidelines on sizing store gateway #4802

Open snuggie12 opened 2 years ago

snuggie12 commented 2 years ago

Is your proposal related to a problem?

I'm getting OOMs on my storegateways when trying to make queries of a certain size in grafana. I'm not looking for some magical switch to make it start working but some docs on how to tune the store gateways and which variables should be taken into consideration when doing that sizing. For example, I see this comment https://github.com/thanos-io/thanos/issues/1750#issue-523912143 which starts to list how many objects in the store and total size of bucket. Are there other factors to look at there?

Additionally it'd be great to know when to consider sharding and performance impacts of choosing in-memory vs memcached for caching.

Describe the solution you'd like

See above

Describe alternatives you've considered

I've tried to find other examples but don't find much that discusses scale AFAIK.

Additional context

stale[bot] commented 2 years ago

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

snuggie12 commented 2 years ago

Despite it being more than two weeks it looks like this is still open.

I still think there's a benefit to a scaling document. It'd be nice to understand when we should shard or memory settings.

stale[bot] commented 2 years ago

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

snuggie12 commented 2 years ago

Bumping this. I still think some general guidance on how to size things would be helpful.

stale[bot] commented 1 year ago

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

milesarmstrong commented 1 year ago

Bump, I'd also find this useful!

stale[bot] commented 1 year ago

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

george-angel commented 1 year ago

Remove stale

B0go commented 1 year ago

I also find this important to have as Thanos store have different flags and configuration scenarios we can rely on to have a reliable deployment