Generic Event Markers populated by Influx queries

[mkarlesky]

(I believe @ebb-tide may want to be aware of this submission specifically; hence, tagging her. By the way, thank you again, @ebb-tide, for your work on the CSV export functionality in Chronograf from way back. That was clutch for us.)

Background: Originally, I submitted the core of this feature request as Issue #5165, an enhancement to Chronograf's Annotation feature. My request got some love from others and was on the development roadmap for a short while. I assume that with the shift to Influx v2 it fell to the wayside. Most recently it was closed as stale.

I've heard from an insider the rough outline of the Event Marker feature in the Influx v2 roadmap. As I understand it, the first version to be rolled out will be tied specifically to Alerting & Monitoring, displaying notification events overlaid on data plots. Subsequently, Event Markers are intended to be made more generic and available throughout dashboards and data exploration.

I was encouraged to write up a feature request for sake of providing example use cases and providing some input on how the future generic Event Markers might operate.

Proposal: Event Marker overlays throughout the InfluxDB v2 interface (e.g. Dashboards & Data Explorer) populated by Influx query results able to display both singular events and events with a time range. Event Markers would be centrally configured and available to be selectively enabled in the various views throughout the interface.

Manually entered events (akin to Chronograf's Annotation feature) would also be supported. However, it would probably make the best sense if manual annotations were stored in an Influx bucket and then queried for display using the same mechanisms needed to support the generic ability discussed here.

Notes on events:

• Many events are singular moments (e.g. a piece of equipment transitioned from low to high speed). These correspond well to individual rows in an Influx query response. The timestamp is the time of the event and its fields and tags contain its details.
• Some events are a time range (e.g. server downtime 7:39am – 8:02am or cleaning 1:37pm – 2:09pm). Mapping an event with a time range to Influx query results is a bit more challenging. A few options that might work:
  • In a single query result row, the timestamp represents the beginning of the event while a field stores the end of the time range. This is only practical when the entirety of an event is known after its conclusion.
    • Duration is stored as an integer field (cast with toDuration() in Flux).
    • End time is stored as an integer field (cast with toTime() in Flux).
    • End time is stored as a string field (parsed with time() in Flux).
  • The event's time range is represented by two distinct rows in query results linked together logically by some unique ID in common. Care in display would be necessary in circumstances where the end of a time range is not yet stored. In the simplest case the beginning of the range could be displayed just like a singular event. Alternatively, some sort of “smear” of transparent color could visually indicate an event that has begun but has not concluded.

Current behavior: To my knowledge Event Markers are in some form of development but only in relation to Alerting & Monitoring at present.

Desired behavior: In essence, my proposal is essentially Grafana's query-based Annotations plus the following. The first two in this list are enhancements to Grafana's existing abilities. The last item in this list is an all new feature.

1. Not only single event markers but also event time ranges (see Proposal above) both populated by query results.
2. The ability to create centrally managed Event Marker queries whose results are available for overlay anywhere within Dashboards and the Data Explorer interface.
   1. In Dashboards these would be selectable (toggle on / toggle off) for an entire dashboard but also per individual plot (dashboard cell).
   2. In the Data Explorer interface not only would the user be able to enable preconfigured Event Marker overlays, they would also be able to author ad hoc Event Marker queries for overlay on the existing query submission / display interface.
3. Event Marker overlays on tabular data views. Some query results are not suitable for graphing, but understanding events in relation to query results is still desirable.

Alternatives considered: At present we can use Grafana for query-filled “Annotations” in dashboards, albeit only for singular events and not time ranges. With some custom work Power BI can be coerced into displaying rudimentary event markers / annotations in dashboards and reports. Otherwise, we have not found a good way to present discrete events or event time ranges in a user-driven data explorer interface for InfluxDB time-series data. Chronograf was previously our best hope for this had my requested enhancement to Annotations come to pass.

Use case: Our time-series data largely falls in three categories:

1. Sensor readings (environmental monitoring)
2. Equipment operation (on/off, cycle times, state transitions)
3. Production line events — e.g. Batch XYZ entered/exited a production stage

All of the above are interrelated and also influenced by other events — maintenance, system changes, etc. We want to begin tracking external events also as time-series data (a fourth category that does not yet exist for us) so we can visually correlate cause and effect across our systems.

Examples:

• A particular environmental reading may change as the result of equipment operation. A reading may rise or fall in step with a particular piece of equipment turning on or off.
• Equipment operation can be influenced by maintenance or lack thereof. A machine's cycle time may be correlated with when its filters were last cleaned.
• Production line events may shift in relation to environmental conditions or equipment downtime. Such shifts correlated to other events can reveal unreported operations issues.

Being able to visualize our events in relation to plotted data such as above would help us:

• Troubleshoot quality, environmental, and performance issues
• Provide greater situational awareness during regular operations on the shop floor
• Provide valuable input to various R&D & continuous improvements efforts

[ebb-tide]

@mkarlesky thank you so much for this thoughtful writeup. I was just working on some polish on event markers (as part of the monitoring feature) today, so your timing is great for us to consider ways to generalize the implementation. Thanks so much for sharing your perspective. I'll be referencing this issue as I move forward.

[mkarlesky]

@ebb-tide Thanks for the quick reply and so nice to hear from you. I realize there's a lot in my feature request. I'm happy to clarify or provide further input in the future if it's desired.

We've come to learn that correlating events to time-series data is incredibly valuable. InfluxDB is nearly perfect for supporting this sort of insight.

Thank you again.

[mkarlesky]

Woohoo! Thank you, @russorat.

[russorat]

@mkarlesky sure thing. sorry about the bot, it has been removed.

We have this on our roadmap but we might not get to it for a while. We will keep this issue updated with status.

[afausti]

@mkarlesky @russorat I'm happy to see this proposal, having event marker overlays populated by Influx queries would help some of the use cases I described in #16954, thank you.

[mkarlesky]

@afausti Thank you for your contributions on this product direction!

The story on my side is that I originally contributed feature requests in relation to Annotations in Chronograf (https://github.com/influxdata/chronograf/issues/5165); eventually these requests became irrelevant as InfluxDB 2 development came to the fore. I was involved in the InfluxCloud 2 beta and saw the early versions of Event Markers in relation to the Alerts and Notifications features. Having seen those designs I wanted to bring my original Chronograf feature requests for Annotations forward to the InfluxDB 2 product and was encouraged to do so by internal InfuxData people. I did so but then my requests here from last October were closed by stalebot. Thankfully, @russorat revived my issue a little over a month ago.

Your needs and feature requests as described in https://github.com/influxdata/influxdb/issues/16954 aligns quite well with our own in spirit. We talk about things a bit differently, but I think these our two Github issues are quite a bit in alignment.