Unable to filter more than 22 eventIDs

[Alreanaes]

• Version: winlogbeat 5.0.0-alpha1
• Operating System: Windows Server 2008R2 (also confirmed on 2012R2)
• Steps to Reproduce: In config file, add line: event_id: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24

The relevant output from -e -d "*"

DBG  WinEventLog[ForwardedEvents] using subscription query=<QueryList>
  <Query Id="0">
    <Select Path="ForwardedEvents">*[System[(EventID=1 or EventID=2 or EventID=3 or EventID=4 or EventID=5 or EventID=6 or EventID=7 or EventID=8 or EventID=9 or EventID=10 or EventID=11 or EventID=12 or EventID=13 or EventID=14 or EventID=15 or EventID=16 or EventID=17 or EventID=18 or EventID=19 or EventID=20 or EventID=21 or EventID=22 or EventID=23 or EventID=24)]]</Select>
  </Query>
</QueryList>
WARN EventLog[ForwardedEvents] Open() error. No events will be read from this source. The specified query is invalid.

According to KB9704531 , more than 22 event sources need to be split into seperate queries.

I have successfully tested the following query in event viewer where the above query fails

<QueryList>
  <Query Id="0" Path="ForwardedEvents">
    <Select Path="ForwardedEvents">*[System[(EventID=1 or EventID=2 or EventID=3 or EventID=4 or EventID=5 or EventID=6 or EventID=7 or EventID=8 or EventID=9 or EventID=10 or EventID=11 or EventID=12 or EventID=13 or EventID=14 or EventID=15 or EventID=16 or EventID=17 or EventID=18 or EventID=19 or EventID=20 or EventID=21 or EventID=22 or EventID=23)]]</Select>
  </Query>
  <Query Id="1" Path="ForwardedEvents">
    <Select Path="ForwardedEvents">*[System[(EventID=24)]]</Select>
  </Query>
</QueryList>

[andrewkroh]

When I implemented the first round of "filtering" I was thinking in the future Winlogbeat could support more advanced queries like shown below. This would be a work around if it existed.

  event_logs:
    # Possible today:
    - name: Application
      include_xml: true
      ignore_older: 24h 
      event_ids: 4600-4700, 8, -4650
      level: crit, err, warn
    # INVALID CONFIG BELOW - DO NOT USE
    # Each of the named queries becomes a Query in the XML QueryList
    - name: External Media Detection
      queries:
        New Device Info:
          log: Microsoft-Windows-USBUSBHUB3-Analytic
          provider: Microsoft-Windows-USBUSBHUB3
          event_ids: 43
          level: info
        New Mass Storage Installation:
          log: Microsoft-Windows-KernelPnP/Device Configuration
          provider: Microsoft-Windows-KernelPnP
          event_ids: 400, 410 
          level: info
    # Then there's also a the possibility of writing the XML yourself.
    - name: Pass the Hash Detection
      xml_query: >
        <QueryList>
         <Query Id="0" Path="ForwardedEvents">
         <Select Path="ForwardedEvents">
         *[System[(Level=4 or Level=0) and (EventID=4625)]]
         and 
         *[EventData[Data[@Name='LogonType'] and (Data='3')]]
         and 
         *[EventData[Data[@Name='AuthenticationPackageName'] = 'NTLM']]
         and 
         *[EventData[Data[@Name='TargetUserName'] != 'ANONYMOUS LOGON']]
         and 
         *[EventData[Data[@Name='TargetDomainName'] != '<DOMAIN NAME>']]
        </Select>
         </Query>
        </QueryList>

We don't have plans to add this in the immediate future (5.0 or 5.1) because we're working on other Beat features. But if anyone is interested in contributing, this would be a cool feature. Probably the raw XML query is the easiest to implement.

[gmoskovicz]

@andrewkroh given this restriction, should we add a mention in the documentation? We could add a disclaimer at https://www.elastic.co/guide/en/beats/winlogbeat/current/configuration-winlogbeat-options.html#_event_logs_event_id . What do you think?

Looks like an API issue, so nothing that can be actually fixed in Winlogbeat. Other than splitting the event_ids if the list is larger than 22 items into 2, without letting the user know that this internally happen.

[ruflin]

@andrewkroh I see we updated the docs related to this. Is this still a limitation? If yes, should we change this from bug to enhancement?

[andrewkroh]

This can be closed. We have a workaround documented. And a separate enhancement request for advancing the query syntax in #1054.

[JPvRiel]

Hi, even if this is closed because there's a proposed enhancement for direct XML query syntax, I'm noticed that the limitation of 22 logical boolean operands (I only confirmed with 'or') is a "top level" limit, but grouping operands can work around the limit. E.g. *[System[(EventID=1000 or EventID=1001) or (EventID=2000 or EventID=2001)]] seems to only count as 2 in the limit of 22.

I confirmed this in powershell:

(10, 22, 23, 30) | %{
  $lower=1000
  $upper=1000 + $_
  $XPathString = '*[System[EventID=' + ($lower..$upper -join ' or EventID=') + ']]'
  Write-Information "`nEvent IDs in query: $_"
  Write-Information "XPathStringLength: $($XPathString.Length)"
  Get-WinEvent -FilterXml "<QueryList><Query Id=`"0`"><Select Path=`"System`">$XPathString</Select></Query></QueryList>" -MaxEvents 1
}
# Grouped?
$XPathString = '*[System[(EventID=' + (1000..1020 -join ' or EventID=') + ') or (EventID=' + (2000..2020 -join ' or EventID=') + ')]]'
Write-Information "`nEvent IDs in query: 20 in each of 2 groups = 40"
Write-Information "XPathStringLength: $($XPathString.Length)"
Get-WinEvent -FilterXml "<QueryList><Query Id=`"0`"><Select Path=`"System`">$XPathString</Select></Query></QueryList>" -MaxEvents 1