Scope adding support for streaming updates

[mukeshsahay]

Is your feature request related to a problem? Please describe. Our customers are asking us to not trigger a full reload of data on every refresh of the panel. We should cache the time series data and only request data history for the time range not found in our cache. Currently, if the dashboard is set to auto-refresh every 15 seconds it will reload entire data every 15 seconds and re-draws the entire panel. This makes it hard for the users to use the dashboard.

Describe the solution you'd like We would be satisfied with a streaming plugin implemented for our datasource. Add support for streaming updates for queries on the panel. Also, the request looping logic should be based on the caching logic defined here.

Describe alternatives you've considered No other solution seems to work in this case.

[sarahzinger]

A great suggestion! I'm putting this in our backlog for now since we don't have an immediate timeline on when we would implement this, but we don't want to forget it about it!

[mukeshsahay]

This is a high priority request rather I would say it is the most critical and important request from our customers, that needs immediate attention. @diehbria - Please help setting the priority.

[jhoggood]

@mukeshsahay @sarahzinger I am a Principal Industrial Specialist working closely with our delivery and engineering team on this particular issue relative to an important Industrial customer. I cannot stress enough the importance of resolving this peformance issue in the next 24-48 hours for this customer who's urgently trying to finish UAT this week with the goal being to scale to other plants soon. I will be grateful if you can prioritize and correct this situation in the next 24-48 hours. For more details please don't hesitate to email me. Thank you!

[taleena]

Hello: I understand you are working towards a deadline that we were not a part of, and the frustrations of running into something unexpected. For all new requests (bugs, features and/or requirements) we generally require additional information to help assess. Is this for an Amazon Managed Grafana customer, or self-hosted? What version of Grafana is running, and what version of the plugin? If this is a performance issue, are you seeing any errors? Would you be willing to share your dashboard/some panels with us? What other approaches have you tried already (eg. if you are on Enterprise, is query caching enabled? what prevents lengthening the dashboard refresh time? etc)

To better set expectations: requests for new features typically require design, planning, and prioritization against other feature requests. We've already completed our quarterly planning for Q4, which means taking this on ad hoc would greatly disrupt our other commitments. That being said, we'll look into this for you once provided the information above, and see if there is a workaround available that would help you complete your testing.

[TheEvilDev]

@mukeshsahay @sarahzinger I am a Principal Industrial Specialist working closely with our delivery and engineering team on this particular issue relative to an important Industrial customer. I cannot stress enough the importance of resolving this peformance issue in the next 24-48 hours for this customer who's urgently trying to finish UAT this week with the goal being to scale to other plants soon. I will be grateful if you can prioritize and correct this situation in the next 24-48 hours. For more details please don't hesitate to email me. Thank you!

This isn't going to be feasible. What we need is a timeline and effort estimate we can provide our PMs, and they can reach out to help prioritize with other pressing Grafana changes. This is a significant feature request that will take time to design and plan, and has implications for other customers, it's not a bug fix that just requires a few tweaks and we can push out.

[sarahzinger]

Hey folks, just wanted to let you know that on our end, we're working on an implementation for this now, and are hoping to have something before the second week of January.

[sarahzinger]

Hey, sorry, this is taking a bit longer than expected. I had hoped to at least push a WIP by now, but it's been taking a bit longer than I thought, as we don't have a streaming tutorial just yet.

Theoretically it's a matter of implementing 3 methods on the backend server file: SubscribeStream, RunStream, and PublishStream, and then setting a streaming channel in QueryData as we have done in twinmaker here: https://github.com/grafana/grafana-iot-twinmaker-app/blob/main/pkg/plugin/datasource.go#L145

In twinmaker we seem to have a streaming toggle in the query editor that controls this:

When it's enabled, we send along an isStreaming field on the query object and then it sets the channel meta data.

But I would prefer to not add that, as when you set streaming: true on the plugin.json, it seems to auto add this Grafana Live button which seems to do a similar thing:

I'm having a bit of trouble figuring out how to identify if that is on or not though from the plugin itself.

In anycase, I believe this is do-able as we seem to have implemented something similar with TwinMaker. Here's the PR for that which hopefully gives a bit of a sense of the scope of work: https://github.com/grafana/grafana-iot-twinmaker-app/pull/83

Still hoping to do this by Second week of Jan, but it might be closer to third week of Jan.

[sarahzinger]

Sharing what I have now even though it's not quite working yet to get some more feedback: https://github.com/grafana/iot-sitewise-datasource/pull/277

I want to validate that this seems like this right general approach to this problem before continuing to invest time in making this solution work. Hoping the rest of the team can take a look and give some fresh perspectives

[sarahzinger]

Ok after talking with the team a bit on this, I think it makes sense to pause on implementing this work and get more clarity about what user problem we are trying to solve (is it to handle large amounts of queries or is to stream live data) and we pick and build one solution for it (maybe it's backend streaming over a websocket, or maybe its actually something else like improving performance).

The draft I have up can be a good point of reference should we decide we want to move forward with this feature, but should not be considered a working/finished solution, but rather more exploratory work. I had originally hoped that this would be as simple as implementing 3 methods on the backend, but it's a little bit more than that. So I'm shifting the purpose of this github issue to be a "scoping" ticket. If we do decide to move forward to work on backend streaming over a websocket, it probably makes the most sense to then break that work into multiple smaller prs that are a bit more planned and well tested.

As a rough scoping of how we could break this work up going forward, and the amount of effort it could take getting there:

Backend Changes:

• (small effort) Create a new handler “handleStreaming” that updates a response to have the streaming channel
• (medium effort) Update server.go to implement “RunStream” “PublishStream” and “SubscribeStream”. The Bulk of this will be implementing RunStream which will need to start a loop to identity and process each next query

Frontend Changes:

• (small effort) Update query method to no longer call nextToken on the frontend (this should happen on the backend now)
  • Open Question: Should this happen for non-streaming queries as well?
• (small effort) Add ui elements to let users toggle streaming, and select a time range, should only show on queries that are designed to be streaming queries
• (small effort) Add support for Grafana Live Button
  • Open question: Is this the same feature as Grafana Live? Does it make sense to have both a streaming toggle and a grafana live button? Twinmaker has both the Grafana Live button and the streaming toggle, but there are circumstances where a query is not appropriate to stream (like listing the columns in a database).