search

thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
https://thanos.io
Apache License 2.0
13.04k stars 2.09k forks source link

query-frontend: Cache Intermittent queries (e.g inside Instant Queries) #2569

Closed bwplotka closed 3 years ago

bwplotka commented 4 years ago

It's very common for rulers to query super long time ranges like *_over_time(metric[1w]). The problem is that it's just an instant query so totally not supported by the existing cortex query frontend.

Let's make sure we can contribute something that is capable of doing so, potentially while doing https://github.com/cortexproject/cortex/issues/2178 and https://github.com/thanos-io/thanos/issues/2454

ivan-kiselev commented 4 years ago

we really struggle with that one here. Our users throw heavy aggregation queries on my_lovely_hight_cardinality_metric[28d] into ruler config and that basically unbearable, some of the queries take up to 10-15 minutes to return the value, for comparison - we have ~1.2k rules (both recording + alerting) and most of them evaluate within standard global evaluation interval of the 20s, though for metric[28d] we had to define separate eval intervals ~10 minutes and you can imagine the quality of data is not certainly best there.

It was a huge pain to realize that cortex-frontend only works for range queries :)

bwplotka commented 4 years ago

Yup, definitely high priority (:

On Tue, 2 Jun 2020 at 15:53, Ivan Kiselev notifications@github.com wrote:

we really struggle with that one here. Our users throw heavy aggregation queries on my_lovely_hight_cardinality_metric[28d] into ruler config and that basically unbearable, some of the queries take up to 10-15 minutes to return the value, for comparison - we have ~1.2k rules (both recording + alerting) and most of them evaluate within standard global evaluation interval of the 20s, though for metric[28d] we had to define separate eval intervals ~10 minutes and you can imagine the quality of data is not certainly best there.

It was a huge pain to realize that cortex-frontend only works for range queries :)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/thanos-io/thanos/issues/2569#issuecomment-637595065, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABVA3O3WZFWUIYHBIMQSY2DRUUG6XANCNFSM4M2TNDEA .

stale[bot] commented 4 years ago

Hello 👋 Looks like there was no activity on this issue for last 30 days. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

stale[bot] commented 4 years ago

Closing for now as promised, let us know if you need this to be reopened! 🤗

stale[bot] commented 4 years ago

Hello 👋 Looks like there was no activity on this issue for last 30 days. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

stale[bot] commented 4 years ago

Closing for now as promised, let us know if you need this to be reopened! 🤗

stale[bot] commented 3 years ago

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

bwplotka commented 3 years ago

Update: We have clearer idea how to do it, question is rather how important this is it and when to do it. Help wanted we can guide someone (:

On Fri, 16 Oct 2020 at 12:52, stale[bot] notifications@github.com wrote:

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command https://probot.github.io/apps/reminders/ if you wish to be reminded at some point in future.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/thanos-io/thanos/issues/2569#issuecomment-709974078, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABVA3OZ5HIMBRTHQJHTF4H3SLAQV3ANCNFSM4M2TNDEA .

yeya24 commented 3 years ago

I am interested in taking this if no other volunteers. Do we already have a plan for how to do this?

kakkoyun commented 3 years ago

@yeya24 I don't think anyone has started working on this. So go ahead.

yeya24 commented 3 years ago

It is not clear to me how to achieve this feature. Do you have anything in mind?

bwplotka commented 3 years ago

I had some PoC done, let me push it.

Kind Regards, Bartek PÅ‚otka (@bwplotka)

On Mon, 19 Oct 2020 at 15:15, Ben Ye notifications@github.com wrote:

It is not clear to me how to achieve this feature. Do you have anything in mind?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/thanos-io/thanos/issues/2569#issuecomment-712148537, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABVA3O7MFZ4YYTCRU2HM34DSLQ3YFANCNFSM4M2TNDEA .

stale[bot] commented 3 years ago

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

stale[bot] commented 3 years ago

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

stale[bot] commented 3 years ago

Closing for now as promised, let us know if you need this to be reopened! 🤗

yeya24 commented 3 years ago

Reopen as I am still interested in this one.

stale[bot] commented 3 years ago

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

stale[bot] commented 3 years ago

Closing for now as promised, let us know if you need this to be reopened! 🤗

than-pet commented 4 months ago

Caching instant queries on frontend would be very beneficial for thanos-ruler. Instant queries with large time windows are not scaling well on ruler.