[Proposal] Generic Time Series Engine in Pinot

[ankitsultana]

Hi folks, we are excited to share our proposal for supporting a Generic Time Series Engine in Pinot: https://docs.google.com/document/d/1SBDDf71QZINYUjAbRSWguNMfbrWRGfdcF1JPi8SJZlM/edit

You can find a quick overview of the proposal below.

Context

At Uber, we have seen an increase in Observability use-cases for Pinot (for both alerting and monitoring). Pinot is commonly used for High-Cardinality Metrics use-cases at Uber, since our in-house system for Observability (M3), isn't optimized for them.

One of the feedbacks from the customers has been that SQL tends to get quite complex for time-series analysis, and it not only hampers productivity, but is also lacking certain features commonly available in Observability systems (esp. Gapfill).

Earlier this year, we internally built out native support in Pinot for M3QL, Uber's open-source query language. The results were quite promising: we were able to support all of M3QL's features natively in Pinot.

We hence kick-started this effort to bring these improvements to OSS Pinot.

Language Agnostic

Our proposal is such that one will be able to implement plugins that can re-use the Generic Time Series Engine to support any time-series query language (PromQL, M3QL, etc.). This will allow each stakeholder to develop their own language plugins.

For more please refer to the doc.

cc: @kishoreg

[siddharthteotia]

@ankitsultana - can you open the design doc for comment access please ?

[ankitsultana]

@siddharthteotia done

[gortiz]

First of all, this is a great initiative. I've been working in Devo Inc, a company that is focused on the timeseries market (trying to compete with Splunk) and there we designed our own streaming SQL database that supports both SQL and LINQ expressions. The later is closer to the M3QL language proposed in the design document and the language most customers used to use.

I've also created an internal proposal in StarTree to discuss about this feature. It can be read here. My document is not focused on the language itself. I believe SQL is expressive enough to be able to execute this kind of queries, but I understand that the market is not there and supporting M3QL and specially PromQL will be probably more useful to attract users.

I have some concerns about the designed document:

1. My main concern about the proposed design document is that I don't think we need to add too many new concepts on top of the multi-stage query engine (MSQE). Instead, what I think we need to add is some key functions and some special operators (specifically for aggregate, window and join) that should work in streaming fashion.
2. I think the design is too broad and ambitious. I would split it in the following:
   1. Add streaming capabilities to the current MSQE. Basically the tumble functions and window base aggregate, join and probably window functions.
   2. Add the plugable syntax/protocol support. There is where we would think about how to add M3QL, etc.
   3. Add the ability to extend aggregates. The current design assumes this is going to be needed to support other syntaxes, but I don't think this should be part of the time series design given that is also something that would help normal queries. We have recently added the ability to create new scalar and transformation functions using typesafe and it was a hard job to do, so I think we should think about long term support when adding a way to add aggregate functions.

Anyway I know I'm late to the party and several key PRs have been merged so it doesn't make sense to discuss too much about the past, but please take me also in consideration for future developments.

Again, I don't want to sound very pessimistic. I think this is an awesome feature for Apache Pinot and it is great to see it progress that fast!

[ankitsultana]

@gortiz thanks Gonzalo for sharing your views. There are several points made by you here and in the design document, but I think it's best to first establish the broader picture in which we are doing this.

As you know, most companies have been using Pinot for OLAP on mainly "events". The idea being that you can power user-facing dashboards or other end-user App UI elements like counters based on Realtime data with low latency.

At least at Uber, over the last 2 years, we have seen an increase in the number of users who are using Pinot for alerting and monitoring ("metrics"). And recently we also shared in the Uber Meetup that we are now using Pinot for our logging platform too. (side-note: Pinot specifically solves the problem of High Cardinality Metrics)

The term being used these days to cover this landscape of use-cases is MELT: Metrics, Events, Logs and Traces.

Pinot is able to solve all of these problems, but its capability to solve all of these problems differs. What our proposal aims to do is to immediately improve Pinot's ability to tackle the Metrics and the Logging problem, and IMO bring it up to the state of the art in the HICAM space. (note that CH also announced in their latest release that they plan to support PromQL).

re: your proposal about Streaming queries in Pinot, I think it's talking about a different problem.

re: your specific points about this not being necessary and we can leverage the MSE or add enable some SQL extensions, I shared the transformNull (gapfill) example in the doc which should help clear that up. For other readers, see the comment thread at the bottom of this doc

Finally, I think this time-series engine may stick out like a sore thumb right now, but once we integrate it with the Multistage Engine Shuffle Framework, I think it will sit very neatly with the rest of the code. And from a use-case perspective, it takes Pinot forward and expands its capabilities around use-cases that it theoretically can already support, but can't support as well. (side-note: from a high-level, I am thinking in the direction that we will have a common Operator interface in the SPI that will be oblivious of the data-model: relational, time-series, etc.)

A potential future direction is that we may make the entire engine (above the V1 server level engine) pluggable, but it might make sense to consider that after making sure that the time-series engine becomes a success first.