Filter empty responses in Alerts query using Regex

[tangkong]

Is your feature request related to a problem? Please describe. Thanks for all the hard work! I'm not sure if the archiver datasource supports this, but figured I'd ask.

In normal dashboard construction you can specify groups of PVs with regex. This works great

However if I make an alert with a similar query, after tweaking the regex slightly I get the following error "PLC:CRIX:MOT:TaskInfo:2:ExceedCount_RBV Response is Empty"

For regex alert queries where all the PVs are properly responding, things seem to work fine

The alerts can be created fine with single PVs in the query, but I'd like to avoid that.

This PV is not included in the dashboard query, presumably due to some filtering on the query side. If this is the case, is it possible to apply the same filter to the alerts query?

Perhaps I'm misguided, feel free to let me know if so.

Describe the solution you'd like A good way to define alerts with regex queries

[sasaki77]

Thanks for asking. The plugin has two types of implementation to retrieve the archived data, TypeScript frontend and Go backend. In normal dashboard, you can choose which implementation to use on Data source setting page. On the other hand, the alert query must use Go backend implementation.

I think you are using TypeScript frontend to retrieve the data in normal dashboard. That's the why the behavior between the dashboard query and the alert query is different.

One way to filter these PVs is that to add a new function that filters the empty response PVs. If you accept this approach, I will try to implement this function. Or if you have another idea, please let me know.

[tangkong]

Thanks! I don't have the ability to see the datasource settings page, so I unfortunately can't verify your hypothesis.

Either way I think having that empty-rejecting-function would be a nice solution. I don't have any other ideas anyhow.

Thanks again!

[sasaki77]

I confirmed that the plugin can filter an empty response with a new function. The ignoreEmptyErr function will be added to the Options functions category. You can use it to filter empty response errors in the alert query.

I will merge this change. v1.6.1 will be released after that. Please be patient.

[sasaki77]

v1.6.1 is released with the ignoreEmptyErr function. Could you check if the plugin works properly? If the plugin does not work the way you want, don't hesitate to reopen the issue.

[tangkong]

We tried out the plugin, and saw some weird behavior when running alert query. Running the regex query in a dashboard works fine, but once we attempt to make an alert with that same query, it causes the archiver appliance to quickly consume all the memory it has available. (The drop at the end is from us restarting the archiver appliance)

This isn't something we saw in the past, though it's possible the query was short-circuited by the empty response exception?

[tangkong]

@sasaki77 , since I'm not sure how to reopen the issue ...

[sasaki77]

I have some questions.

• Does this graph show the usage of the Archiver Appliance and not the Grafana plugin?
• Does this memory consumption occur when you use ignoreEmptyErr on alert query? Could you check if the consumption occurs when you don't use ignoreEmptyErr function?
• Can you check the log of Grafana server and find some clues? It is located in /var/log/grafana by default.
• Can you check the log of Archiver Appliance and find some clues?

[tangkong]

1. Yes, this is graph is the memory usage of the archiver appliance
2. The memory consumption actually occurs without the ignoreEmptyErr function.
3. / 4. I'll do some digging

[tangkong]

Ok, we found the issue to be the query itself. The default settings on the query used a max_1 operator, with a huge history interval, resulting in oversized responses.

I think this is outside the domain of the plugin, ideally we have grafana supply more reasonable defaults. Thanks for your patience and help with this 👍