DatasourceError: Plugin.unavailable

[tangkong]

Describe the bug We get sporadic errors where grafana reports that the datasource plugin is unavailable. The error raises an exception/error and triggers a notification to be sent as in the screenshot below. These errors are sporadic, and alerts will often oscillate between "Normal" and "Error" status, even though data has

I apologize if this description is somewhat vague. I haven't been able find settings that prevent this error from occurring, and our logs have not elucidated anything for us. The time range of the alert does not seem to affect the error rate.

To Reproduce (I'll share one of our alert configurations, but there were many that saw this error) Steps to reproduce the behavior: State history showing oscillation between "normal" and "error" state A variety of queries showing that the query can return real values that evaluate to normal The resulting error notifications

Expected behavior We'd like for plugin.unavailable errors to not occur 😄

Desktop (please complete the following information):

• OS: rhel-7
• Browser: firefox, chrome

[sasaki77]

Thank you for reporting this. I'm afraid I haven't seen this error. Could you provide the following information? It might help me to solve this issue.

• Grafana server version
• AA datasource plugin version
• Logs related to this issue in /var/log/grafana/grafana.log

[tangkong]

Sure thing:

• Grafana: v9.3.6
• AA: 1.3.5

I do not currently have access to the logs, but will work on that now. Thanks for the help!

[tangkong]

A snippet of our logs that involve plugin.unavailable plugin_unavail.txt

[sasaki77]

It appears that you may still be using an older version of the plugin. Is it possible to update the plugin to v1.4.4 or later? I'm not sure if the later version solves the problem, but it may help me to investigate the problem more easily.

[tangkong]

We tried installing an updated plugin on our dev server, and replaced the plugin.unavailable errors with some slightly different errors...

• grafana: 9.4.7
• AA: 1.5.0

Similar to before, the errors are intermittent. In some cases I can see archiver data without error

but in other cases I get a query error.

In this grafana instance, the error shows up in the dashboard panel as well

I've verified that this behavior is not isolated to a specific PV, and also occurs for the mean operator. The logs don't seem to have any additional information aside from the error message shown in the screenshots. I have not yet observed the plugin.unavailable error we see on our prod instance

Thanks again for the help!

[sasaki77]

Thank you for your detailed report! I'm not sure why, but Grafana said that it couldn't parse the requested query. This error only occurs for alerting query, right? I'm wondering if this error also occurs for the query from the dashboard panel.

[tangkong]

Sorry for not being clear, the last screenshot was actually a view of the dashboard panel, not the alerting query.

[sasaki77]

One thing I noticed is that your Grafana might try to parse datasource from query. However, the original plugin codes have no datasource field for ArchiverQueryModel.

The original codes are available from archiverappliance-datasource/models.go.

I tried adding the datasource field as a string and Grafana returned a similar error. Unfortunately, I can't reproduce this error in the alerting query.

type ArchiverQueryModel struct {
    // It's not apparent to me where these two originate from but they do appear to be necessary
    Format    string      `json:"format"`
    Constant  json.Number `json:"constant"`  // I don't know what this is for yet
    QueryText string      `json:"queryText"` // deprecated

    // Parameters added in AAQuery's extension of DataQuery
    Target       string                         `json:"target"`       //This will be the PV as entered by the user, or regex searching for PVs
    Alias        string                         `json:"alias"`        // What to refer to the data as in the table - I think this only for the frontend rn
    AliasPattern string                         `json:"aliasPattern"` // use for collecting a large number of returned values
    Operator     string                         `json:"operator"`     // ?
    Regex        bool                           `json:"regex"`        // configured by the user's setting of the "Regex" field in the panel
    Functions    []FunctionDescriptorQueryModel `json:"functions"`    // collection of functions to be applied to the data by the archiver
    Datasource   string                         `json:"datasource"`

Do you have any idea about this?

[sasaki77]

I have confirmed that ArchiverQueryModel has the datasource field in v1.3.4 or earlier. Could you check if the older version is still running or not?

[sasaki77]

I found the issue (#75) that mentions "json: cannot unmarshal" error.

[tangkong]

Argh sorry, I thought I confirmed the server restarted and AA 1.5 was running, but checking again it appears 1.3 is currently active. I'll revisit

[tangkong]

Sorry again for the late response, we've done some testing and this issue was resolved with an update to 1.5.0. Thanks for all the help here! We appreciate the help and responsivness