Option to combine filters with logical AND instead of the default OR

[piotr-dobrogost]

I would like to filter-in log entries matching all instead of any of the filters set. It would be especially useful as it is not easy to create regular expression matching a series of phrases/expressions as can be seen for instance in the following unix.stackexchange.com question – How to run grep with multiple AND patterns?

[piotr-dobrogost]

Coincidentally such a feature was just added to less – Multiple &-filters can now be in effect at the same time.

[tstack]

I would say that the order of the patterns is most often known, so one could write a regex like "pattern1.*pattern2". I would also say that the desire for ANDing or ORing the filters together is arbitrary. And, now that filtering with a SQL expression is possible, I don't see a reason to implement this option.

[piotr-dobrogost]

I understand your points but at the same time I'm afraid that filtering with SQL expression will set too high a bar to access this feature for average user. People are used to regular expressions when searching/filtering but nobody expects SQL there…

[tstack]

too high a bar to access this feature for average user

Sorry, but I'm not seeing why you consider this a common use case. I don't disagree that it's a use case that needs to be supported, I just don't see what is common about it.

I would say the most common use cases are:

1. Filter out spam. This is handled well with multiple OUT filters. As mentioned in the less PR, as more spam shows up, people want to filter those messages out.
2. Filter in known IDs. This is handled by OR'ing multiple IN filters. For example, if someone is particularly interested in one or more IPs, UUIDs, or whatever, they want to just see that stuff. Doing an AND of the IN filters doesn't make sense for this case.

[elad-eyal]

My use case is that I want to find all mentions of a particular id which have been generated by a particular library. Within the library name will sometimes appear before the id and sometimes after it. I think I would use this often.

Perhaps what we're really after is filter-out-unless statement which will treat all lines not matching the regex as spam. In this case I would use filter-out-unless id and filter-out-unless libname

[tstack]

My use case is that I want to find all mentions of a particular id which have been generated by a particular library. Within the library name will sometimes appear before the id and sometimes after it.

Can you give some concrete examples of this, I can't really understand what you're trying to say here.

[elad-eyal]

So multiple services all use the same library. The library outputs some debug messages and if I'm lucky they will include the resource ID I'm looking for.

Sometimes the log message will be

[app1] mylib: Starting to evaluate resource 656475647

and sometimes I will see

[app3] Failed to do something with resource 656475647 (mylib.py:700)

So I would want to filter in only lines showing both mylib and resource 656475647

Right now my only option is something like

:filter-in mylib.*656475647|656475647.*mylib

If I had a "logical and" feature I would do

:filter-in mylib
:filter-in 656475647
:filter-mode AND

or I proposed you can add this

:filter-out-unless mylib

would filter out all the lines except those with mylib. and then

:filter-out-unless 656475647

will filter out all the lines not containing 656475647 and then I'm left with only lines holding both mylib and 656475647

hopefully this makes sense?

[tstack]

Hmm... I can maybe see adding an extension to the PCRE syntax to allow for inverting the match, something like (*NOT). So, you could run:

:filter-in mylib
:filter-out (*NOT)656475647

The first filter would narrow the view to the messages from mylib and then the second filter would remove messages that didn't match the ID.

My reasoning is that I don't think something like :filter-mode scales with a lot of filters and it can easily lead to confusion when it's set or not set. It would require more indicators and what not in the UI, which is already a Christmas tree.

I favor embedding the (*NOT) option in the regex instead of :filter-out-unless because then it could be used in other contexts. For example, if someone wanted to search for lines that did not contain a particular string, they could do /(*NOT)foobar.

But, again, I don't find any of these options particularly good. I'd much rather rely on knowledge that people might already have, such as writing a SQL expression or using straight PCRE (like "mylib.656475647|656475647.mylib"). Making folks learn too many lnav-specific things isn't a good use of their time. If they need to learn a little SQL to write a sophisticated filter, so be it, that knowledge is likely usable in other parts of their work.

[elad-eyal]

I think /(*NOT)foobar would find all lines that contain anything other than foobar. So If a line has a timestamp and foobar it will be matched too?

I agree that :filter-mode doesn't scale very well, but isn't editing PCRE mean people have to learn something very very very lnav specific?

Maybe I didn't read the docs well enough - can you show me how sql solves the filtering scenario I brought up?

Huh I had another idea, maybe

:filter-in mylib && 65644331

[tstack]

I think /(*NOT)foobar would find all lines that contain anything other than foobar. So If a line has a timestamp and foobar it will be matched too?

The (*NOT) option would invert the match, so /foobar would find all lines that contain "foobar" and /(*NOT)foobar would find all lines that do NOT contain "foobar".

but isn't editing PCRE mean people have to learn something very very very lnav specific?

Somewhat, yes, but it will translate to other contexts in lnav and I can add an example to the online help so it's easy to find.

Maybe I didn't read the docs well enough - can you show me how sql solves the filtering scenario I brought up?

A new feature in the top-of-tree is the addition of a :filter-expr command that takes a SQL expression to use when filtering log lines (see this comment for an example). In your case, it would be something like:

:filter-expr :log_body LIKE '%mylib%' AND :log_body: LIKE '%65644331%'

[elad-eyal]

/foobar would find all lines that contain "foobar"

I thought /foobar does not "find lines" but it searches for foobar is lines admitted by the other filters.

I see how filter-expr is very powerful but unfortunately it seems too verbose for some quick'n'dirty searches which I would usually do with |grep mylib|grep 65644331. I will be looking forward if you add an easy way to filter for lines matching 2 regexps. thanks

(*NOT) would be cool.

[piotr-dobrogost]

I'm revisiting this due to the newly created duplicate issue #1051 and leaving comments I should have left a long time ago.

I would say that the order of the patterns is most often known, so one could write a regex like "pattern1.*pattern2".

I'm afraid the order is often not known as in the example given by @eladeyal-intel in https://github.com/tstack/lnav/issues/790#issuecomment-753492077 In addition even in cases where it's known it might be useful to be able to temporarily disable additional filtering (by disabling some of the filters) which is possible only when patterns are in separate filters. If two or more patterns are placed in one filter as you suggest then to disable some of them requires editing the filter.

I would also say that the desire for ANDing or ORing the filters together is arbitrary.

Not sure why you think so as other users are asking for the same feature due to the problems they have with filtering using current functionality.

And, now that filtering with a SQL expression is possible, I don't see a reason to implement this option.

Just wanted to reiterate that I firmly believe that SQL is not a viable option here. I guess most users of lnav are not familiar enough with SQL to use it in this context. Also, forcing someone to use SQL for something so conceptually simple as combining two or more filters feels odd.

Filter in known IDs. This is handled by OR'ing multiple IN filters. For example, if someone is particularly interested in one or more IPs, UUIDs, or whatever, they want to just see that stuff. Doing an AND of the IN filters doesn't make sense for this case.

Imagine logs, where you have identifiers for for clients, for sessions, for transactions and for single operations inside transaction. If you want to focus on specific client then you filter-in this client's id. Then if you want to focus further you want to be able to extend the filter with specific session's id. Then if you want to focus further you want to be able to extend the filter with specific's transaction's id etc.

Maybe a new kind of filter like filter-require? One could define as many such filters as needed and they all would need to be matched (effectively filtering out lines not matching all such filters).

I also like the idea of filter-out-unless but I feel it's much more natural to specify :filter-require pattern1 and :filter-require pattern2 than to specify :filter-out-unless pattern1 :filter-out-unless pattern2 due to the usage of double negation in the latter case.

Having NOT operator available in regular expressions seems also useful. Using it in the current filter-out would provide functionality requested in this issue without further complicating filter system which I have an impression Tim is not fond of.

[trantor]

I agree with @piotr-dobrogost across the line @tstack . There is also the fact that regex matching in SQLite is not based on PCRE, from what I can see. That is not something obvious to people and quite limiting in cases where full PCRE functionality is required (something not at all unheard of as far as I am concerned). On top of that toggling filtering expressions and incrementally adding or removing them in a SQL query is far from user-friendly and basically a hassle. It does not make sense to switch from individual PCRE-based regexes to a single complex SQL query merely to "intersect" filtering conditions. On the UI side of things, I think that indentation and highlighting might provide clues as to AND/OR conditions, although how to implement the thing would require some good thought and facilities for multiple selection, unless it's quite simplified to prevent "nested" AND/OR groupings (which would be fabolous to have imho).

[tstack]

I would also say that the desire for ANDing or ORing the filters together is arbitrary.

Not sure why you think so as other users are asking for the same feature due to the problems they have with filtering using current functionality.

But, why would the group for whom the current behavior works need to say anything? Just because you hear from one side does not mean the other side does not exist. For example, there have been plenty of requests to be able to "force" the log format used instead of relying on the auto-detection algorithm. But, I don't see random people popping into those threads to mention that they like the auto-detection. I'm grateful for the engagement, but there are also users that don't have the time or interest in engaging and they shouldn't be ignored.

Both use cases are valid. At no point have I ever said otherwise.

And, now that filtering with a SQL expression is possible, I don't see a reason to implement this option.

Just wanted to reiterate that I firmly believe that SQL is not a viable option here. I guess most users of lnav are not familiar enough with SQL to use it in this context. Also, forcing someone to use SQL for something so conceptually simple as combining two or more filters feels odd.

It isn't conceptually simple, though. I think you underestimate how quickly it will get out of hand.

Filter in known IDs. This is handled by OR'ing multiple IN filters. For example, if someone is particularly interested in one or more IPs, UUIDs, or whatever, they want to just see that stuff. Doing an AND of the IN filters doesn't make sense for this case.

Imagine logs, where you have identifiers for for clients, for sessions, for transactions and for single operations inside transaction. If you want to focus on specific client then you filter-in this client's id. Then if you want to focus further you want to be able to extend the filter with specific session's id. Then if you want to focus further you want to be able to extend the filter with specific's transaction's id etc.

I don't have to imagine this, it's how things work in the logs I work with. Except, the log messages do not contain all of those identifiers together, so it's not a matter of narrowing. It's more like a chain. One messages has the user ID and the transaction ID, but the next message only has the transaction ID. So, I can filter in by user ID, but then I need to :filter-in <transaction-id1> and :filter-in <transaction-id2>. If the IN filters were AND'd together, it would not work.

Maybe a new kind of filter like filter-require? One could define as many such filters as needed and they all would need to be matched (effectively filtering out lines not matching all such filters).

That's worth considering.

I also like the idea of filter-out-unless but I feel it's much more natural to specify :filter-require pattern1 and :filter-require pattern2 than to specify :filter-out-unless pattern1 :filter-out-unless pattern2 due to the usage of double negation in the latter case.

Yes, double negatives are a pet peeve of mine.

[tstack]

There is also the fact that regex matching in SQLite is not based on PCRE, from what I can see. That is not something obvious to people and quite limiting in cases where full PCRE functionality is required (something not at all unheard of as far as I am concerned).

SQLite has LIKE, REGEXP, and GLOB^1 for matching text. The REGEXP is implemented with the same machinery as used in other parts of lnav.

On top of that toggling filtering expressions and incrementally adding or removing them in a SQL query is far from user-friendly and basically a hassle.

Yes, toggling sub-expressions would not be easy. Can you elaborate on why updating the query is not user friendly? Typing :filter-expr will pre-populate the prompt with the current expression, so it should be easy enough to make changes. The biggest problem is that it's not multiline.

It does not make sense to switch from individual PCRE-based regexes to a single complex SQL query merely to "intersect" filtering conditions.

Any non-trivial filtering is going to require using logical operators (e.g. AND, OR, &&, ||) to combine the search terms. There's a reason log services like SumoLogic have their own filtering language with these types of operators. Since lnav already integrates with SQLite, it was more straightforward to leverage SQL for complex filtering rather than having a separate language. That being said, I've always had an interest in incorporating angle-grinder into lnav since it's language can be more friendly for interactive use.

On the UI side of things, I think that indentation and highlighting might provide clues as to AND/OR conditions, although how to implement the thing would require some good thought and facilities for multiple selection, unless it's quite simplified to prevent "nested" AND/OR groupings (which would be fabolous to have imho).

Any sort of grouping would still need to be representable via commands. It's not just a matter of being able to construct a TUI that can do stuff. The user should be able to write a script that accomplishes the same stuff.

[trantor]

SQLite has LIKE, REGEXP, and GLOB1 for matching text. The REGEXP is implemented with the same machinery as used in other parts of lnav.

Nice, so my concerns where misplaced in this regard. I did not know there was a custom implementation of the function.

Yes, toggling sub-expressions would not be easy. Can you elaborate on why updating the query is not user friendly? Typing :filter-expr will pre-populate the prompt with the current expression, so it should be easy enough to make changes. The biggest problem is that it's not multiline.

Yep, when you've got 10+ patterns (as it's often the case in my use) going back and forth through a single-line prompt is quite a pain and rather error-prone, especially if you are iteratively trying to add/remove pattern to narrow your search results.

As for the complexity of combining things with logical operations, I agree it's rather complicated, especially going beyond the TUI. However allow me to imagine one scenario, than you can tell me what you think about it. Normally, at least in my years of experience processing logs, one would employ something like grep to filter through lines of text to gather a partial set of those lines for process/analysis. Normally grep works providing patterns which allow alternations, as in the case of PCRE patterns, or by providing multiple patterns, which are used in alternation (OR). To filter further, one would pipe the output to another grep instance, so effectively you've got

pat1 OR pat2 OR ...    # 1st "tier" (first grep command)
AND
pat3 OR pat4 OR ...    # 2nd "tier" (second grep command)
AND
....

In addition with lnav one combines in the same "tier" (allow me the label) both positive and negative matching, to include/exclude things, so you have, in the current system

( INCLUDE pat1 OR INCLUDE pat2 OR ...) AND (EXCLUDE pat3 OR EXCLUDE pat4)    # single "tier"
...

What if we (perhaps enabling it in the UI only in a sort of "advanced mode" not to confuse basic users) created a tier (an ordering tag of sorts) which would determine the order in the "chaining" of the filters (pipeline-style) and could be passed as an optional argument to :filter-in and :filter-out.

( INCLUDE pat1 OR INCLUDE pat2 OR ...) AND (EXCLUDE pat3 OR EXCLUDE pat4 OR...)    # "tier" 1 filters
AND
( INCLUDE pat5 OR INCLUDE pat6 OR ...) AND (EXCLUDE pat7 OR EXCLUDE pat8 OR ...)    # "tier" 2 filters
AND
...

This could be represented as different blocks separated by an horizontal line or something in the TUI, changing the "active editing" tier as you cross it with the cursor. In the commands a mere number as an argument might be used to select the right tier to add/delete filters, leaving tier 1 as the default value if not specified. It's not as complex as multiple logical groupings summed together, it's, in a way, an incremental step from the current situation and it matches most of what, I believe, users already expect from experience with cli tools such as grep.

Do you think this might make sense? Thanks in advance

[tstack]

Circling back to this again... How about using terms like HAS, MUST HAVE, and MUST NOT HAVE for the types of filters? So, the current :filter-in is HAS and :filter-out is MUST NOT HAVE. So, the HAS filters are OR'd together and whatever is left is check with the MUST NOT HAVE filters. So, what is missing is MUST HAVE, where the message must have the specified value. Using these terms might help to make things clearer and address most use cases that don't have really complicated logic.

I can see adding a single :filter command that would work something like this:

:filter has term1
:filter has term2
:filter must-not-have spam
:filter must-have myID

(Of course, TAB-completion would make this easy to type... and the existing commands would still work. But, I would like a common place and terminology to use, so ditching "in" / "out" for something clearer would be nice.)

I can also see adding a --field option so that you can specify the log message field that the pattern is matched against.

[piotr-dobrogost]

This is an interesting take on the problem which seems to address most (all?) use cases in this thread indeed. The only problem I see is that everyone seeing has and must-have will instantly ask himself What the heck? What is the difference between them? In other words it's not intuitive that has filters in and must-have filters out. Having said this maybe it's still the best solution in the end.

[y120]

Hi; first off, thank you for this wonderful tool!

I wanted to provide a counterargument to:

So, I can filter in by user ID, but then I need to :filter-in <transaction-id1> and :filter-in <transaction-id2>. If the IN filters were AND'd together, it would not work.

This is true, but it's also directly supported by regex right now, e.g. with :filter-in <transaction-id1>|<transaction-id2>. In contrast, in cases where the relative ordering is not known or stable, there is no trivial equivalent for ANDing; one alternative is (pat1).*(pat2)|(pat2).*(pat1) but this requires duplication, and grows factorially with the number of patterns, instead of linearly with ORs.

How about using terms like HAS, MUST HAVE, and MUST NOT HAVE for the types of filters?

This works for my typical usage, and I think the terms are ok. Folks used to certain document search engines, or to reading RFCs, might find should, must, must not more intuitive. I think I would personally prefer the terms include, exclude, require, possibly retaining the existing commands for backwards-compatibility and brevity.