Problem when mixing <Exec> and <QueryXML> configuration

[cmiscloni]

I am facing to a strange problem since the upgrade to Graylog 3 and Sidecar Before we used Graylog Collector, I change to Sidecar with the new version and I imported the same configuration but it appears that since, when i 'm mixing and in the same nxlog configuration, the regex in Exec menu doesn't seems to be executed. I specify that the regex are the same and NXLog's logs tell me that configuration is OK and no errors

• Graylog Version: 3
• Elasticsearch Version: 6
• MongoDB Version:
• Operating System: CentOS 7
• Browser version: Chrome, Firefox, IE
• NXLog CE : 2.10.2150 or lower

[mariussturm]

Hi @cmiscloni, thanks for the report! Could you please provide us the configurations you have put in the web interface and the same that was rendered on the Sidecar host? Maybe there is an escape issue so we need to compare both.

[cmiscloni]

Hi @mariussturm,

Yes, see below:

Sidecar Webinterface:

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
LogLevel INFO

<Extension logrotate>
    Module  xm_fileop
    <Schedule>
        When    @daily
        Exec    file_cycle('%ROOT%\data\nxlog.log', 7);
     </Schedule>
</Extension>

<Extension gelfExt>
  Module xm_gelf
  # Avoid truncation of the short_message field to 64 characters.
  ShortMessageLength 65536
</Extension>

<Input eventlog>
        Module im_msvistalog
        PollInterval 1
        SavePos True
        ReadFromLast True

     <QueryXML>   
        <QueryList>
            <Query Id='1'>
                <Select Path='System'>*[System[Provider[@Name='Service Control Manager'] and (EventID=7036 or EventID=4697 or EventID=7045)]] and (*[EventData[Data[@Name ='param1']="PSEXESVC"]] or *[EventData[Data[@Name ='ServiceName']="PSEXESVC"]])</Select>
                <Select Path='Security'>*[System[(Level=1 or Level=2 or Level=3)]]</Select>
                <Select Path='HardwareEvents'>*[System[(Level=1 or Level=2 or Level=3)]]</Select>
                <Select Path='Application'>*[System[Provider[@Name='Trend Micro OfficeScan Server'] and (Level=1 or Level=2 or Level=3)]]</Select>
                <Select Path='Microsoft-Windows-TerminalServices-LocalSessionManager/Operational'>*</Select>
            </Query>
        </QueryList>
    </QueryXML>

        <Exec> 
                if ($IpAddress =~ /::ffff:(.*)/) $IpAddress = $1;
                if $EventID == 21 or $EventID == 25 and $message =~ /(?:Remote Desktop Services|Services Bureau à distance).*(?:.|\n)*.*(?:User:|Utilisateur.*:)\s+(.*).*(?:.|\n)*.*(?:Address:|Source.*:)\s+(.*)$/i
                    {
                        $UserSource = lc($1);
                        $IPV4 = $2;
                    }
                if $SourceName == 'Trend Micro OfficeScan Server' and $message =~ /(?:Virus\/Malware|Spyware\/Grayware):\s+(.*)(?:.|\n)Endpoint:\s+(.*)(?:.|\n)Domain:\s+(.*)(?:.|\n)File:\s(.*)/i
                {
                    $FileName= lc($1);
                    $Endpoint= uc($2);
                    $Domain= $3;
                    $Path= $4;
                }
                    if not ($Severity == 'ERROR' or $Severity == 'CRITICAL') drop();
        </Exec>
</Input>

<Output gelf>
    Module om_udp
    Host xxxxxxxxxxxxxxx
    Port xxxxx
    OutputType  GELF_UDP
    <Exec>
      # These fields are needed for Graylog
      $gl2_source_collector = '${sidecar.nodeId}';
      $collector_node_id = '${sidecar.nodeName}';
    </Exec>
</Output>

<Route route-1>
  Path eventlog => gelf
</Route>

on Host:

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
LogLevel INFO

<Extension logrotate>
    Module  xm_fileop
    <Schedule>
        When    @daily
        Exec    file_cycle('%ROOT%\data\nxlog.log', 7);
     </Schedule>
</Extension>

<Extension gelfExt>
  Module xm_gelf
  # Avoid truncation of the short_message field to 64 characters.
  ShortMessageLength 65536
</Extension>

<Input eventlog>
        Module im_msvistalog
        PollInterval 1
        SavePos True
        ReadFromLast True

     <QueryXML>   
        <QueryList>
            <Query Id='1'>
                <Select Path='System'>*[System[Provider[@Name='Service Control Manager'] and (EventID=7036 or EventID=4697 or EventID=7045)]] and (*[EventData[Data[@Name ='param1']="PSEXESVC"]] or *[EventData[Data[@Name ='ServiceName']="PSEXESVC"]])</Select>
                <Select Path='Security'>*[System[(Level=1 or Level=2 or Level=3)]]</Select>
                <Select Path='HardwareEvents'>*[System[(Level=1 or Level=2 or Level=3)]]</Select>
                <Select Path='Application'>*[System[Provider[@Name='Trend Micro OfficeScan Server'] and (Level=1 or Level=2 or Level=3)]]</Select>
                <Select Path='Microsoft-Windows-TerminalServices-LocalSessionManager/Operational'>*</Select>
            </Query>
        </QueryList>
    </QueryXML>

        <Exec> 
                if ($IpAddress =~ /::ffff:(.*)/) $IpAddress = $1;
                if $EventID == 21 or $EventID == 25 and $message =~ /(?:Remote Desktop Services|Services Bureau à distance).*(?:.|\n)*.*(?:User:|Utilisateur.*:)\s+(.*).*(?:.|\n)*.*(?:Address:|Source.*:)\s+(.*)$/i
                    {
                        $UserSource = lc($1);
                        $IPV4 = $2;
                    }
                if $SourceName == 'Trend Micro OfficeScan Server' and $message =~ /(?:Virus\/Malware|Spyware\/Grayware):\s+(.*)(?:.|\n)Endpoint:\s+(.*)(?:.|\n)Domain:\s+(.*)(?:.|\n)File:\s(.*)/i
                {
                    $FileName= lc($1);
                    $Endpoint= uc($2);
                    $Domain= $3;
                    $Path= $4;
                }
                    if not ($Severity == 'ERROR' or $Severity == 'CRITICAL') drop();
        </Exec>
</Input>

<Output gelf>
    Module om_udp
    Host xxxxxxxxxxxxxxx
    Port xxxxx
    OutputType  GELF_UDP
    <Exec>
      # These fields are needed for Graylog
      $gl2_source_collector = 'xxxxxxxxxx';
      $collector_node_id = 'xxxxxxxxx';
    </Exec>
</Output>

<Route route-1>
  Path eventlog => gelf
</Route>

It looks like the same

Thanks

[mariussturm]

Ok so the rendered configuration actually looks good. I think from a Sidecar perspective we can't do much more then that. I tried your configuration with a simplified QueryXML\Exec block and everything worked fine:

     <QueryXML>   
        <QueryList>
            <Query Id='1'>
                <Select Path='System'>*</Select>
                <Select Path='Security'>*</Select>
                <Select Path='Application'>*</Select>
            </Query>
        </QueryList>
    </QueryXML>
    <Exec>
        if $raw_event =~ /.*/ $new_field = 'foobar';
    </Exec>

Maybe you can ask in the nxlog community forum for more help regarding your Regex but for the Sidecar I can't see a bug here. Closing this, if I am wrong feel free to re-open.