regexp filter partly not working

[dhoffend]

It seems that some regexp filter (Filter: host_name ~~ searchfilter) are not working with the lmd livestatus interface while they're working with naemon-livestatus.

Example hostnames:

srvaa01, srvbb01, srvcc01,
srvaa02, srvbb02, srvcc02,
... 03, 04, etc.

Works with both:

Filter: host_name ~~ srv.*01

Works with naemon, but not with lmd

Filter: host_name ~~ srv..01

Works with lmd, but not with naemon

Filter: host_name ~~ srv.{2}01

It looks like a single . (dot) without * or + or {count} doesn't get accepted in the filter syntax and lmd returns an empty resultset

[sni]

LMD tries to be smart and changes the regex search to a way faster string search if no regex characters are present. And LMD does not count dots as regex characters, see https://github.com/sni/lmd/blob/8ebe7ed6fb076212e2d296169f5c539967b124a5/lmd/filter.go#L292 and https://github.com/sni/lmd/blob/8ebe7ed6fb076212e2d296169f5c539967b124a5/lmd/filter.go#L807 I did not add dots, because that would match most hostnames and those filters are the ones used most. Maybe this should be made optional or only enabled in larger setups with millions of services.

[jacobbaungard]

We have also noticed some issues with the regex filtering, but haven't gotten around to taking a closer look at it.

Here's the specific example that was flagged:

# echo -e "GET hosts\nColumns: name\nFilter: custom_variables ~~ BAR ." | unixcat /path_to_lmd
[]
# echo -e "GET hosts\nColumns: name\nFilter: custom_variables ~~ BAR ." | unixcat /path_to_livestatus
172.16.40.1

Inspecting of the actual data:

# echo -e "GET hosts\nColumns: name custom_variables" | unixcat /path_to_lmd | grep BAR
,["172.16.40.1",{"BAR":"baz","TEMPLATE_USED":"default-host-template"}]

[sni]

yeah, same reason here, a dot is not recognized as regular expression character, so the hole string is not recognized as regular expression and simple string filter are used instead.

[dhoffend]

@sni Thanks for taking a look into it. It would be great if this optimization would be configurable.

We would like to use lmd to take away some load from the naemon core process (which has enough to do receiving 80k passive services via qh). Currently the regex filtering and some performance issues (slow queries) in combination with nagvis are preventing us. Not sure if it helps in the end, but moving lots of frontend calls away from the main core process towards lmd sounds like a good try to me.

[sni]

regex detection has been improve with 057af8939a5eca91fac65574685bcb7b1abb6e84 let me know if you got some more cornercases. At least the are some unit tests now.

[dhoffend]

@sni thanks.

It would still be nice to enable the regex search when the search contains dots (via configuration option). In our environment we have only hostnames (no fqdns). In that case every dot present in the search is 100% a regex search. I can understand that the default regex search detection makes sense in a normal use case.

[jacobbaungard]

Another example with $:

Livestatus:

echo "GET hosts\nFilter: name ~~ monitor-chief-hagfish$\nColumns: name" | unixcat /opt/monitor/var/rw/livestatus
monitor-chief-hagfish

LMD:

echo "GET hosts\nFilter: name ~~ monitor-chief-hagfish$\nColumns: name" | unixcat /opt/monitor/var/rw/lmd
[]

[sni]

thanks, the dollar sign was missing in regex character detection. I added it now.