spotify / spotify-heroic-datasource

Heroic datasource for Grafana
Apache License 2.0
16 stars 7 forks source link

Add warning caching to datasource #117

Closed ryanvazquez closed 4 years ago

ryanvazquez commented 4 years ago

This PR contains the following changes:

Currently, "inline" error warnings are lost between dashboard <--> panel navigations. Heroic (not Grafana) warnings are only generated after data is received in the query panel. When users navigate away from the editor to the dashboard and back to the editor - without requerying - these warnings are lost and can lead to confusion.

Warnings will now persist in the query editor across panel/dashboard navigations until they are manually dismissed or the user requeries Heroic. This PR does not address dashboard level warnings (aka "dogear" warnings) generated by Grafana and may be addressed in a separate PR.

ryanvazquez commented 4 years ago

I've been holding off on moving forward with this until I've had a clearer picture for how Heroic error handling could work and thus play with this cacheing implementation.

Since Heroic responses could contain time series data, even those containing errors/limits/etc - relying on Grafana's native alerting to display warnings to users is not possible. Datasource responses must be rejected and doing so will discard the data.

The datasource currently relies on the onDataReceived hook to generate warnings. However, this hook is only fired while the user is in editor mode and only after a response has been resolved by the datasource. This means the user must either navigate directly to the panel and manually refresh the query to view warnings. Users refreshing dashboards, for example, do not receive them.

The alternative approach would be to raise warnings manually at the time of the query, before the response is passed to Grafana. Since many different components within Grafana can trigger a datasource query, these warnings could be triggered excessively. The datasource receives limited context at the time of the query, so being able to conditionally render warnings whether in dashboard or panel mode is not possible.

Additionally, Grafana offers no way to map option.panelIds that datasource.query receives to the panel that generated the query see related. This means panel properties such as the panel's title, etc. are not accessible to the datasource and can't be used when generating warnings while in dashboard mode. Thus users in dashboard mode could see warnings for all of their panels but not be able to map those warnings to the panel that generated them.

I'm going to stick a pin in this until a cleaner solution becomes available.

ryanvazquez commented 4 years ago

Several updates were made since this PR and makes changes to how errors are displayed in the datasource

warnings