[Cisco FTD] Pipeline Error for Event 302013

[MakoWish]

The Cisco FTD integration is unable to parse events with Message ID 302013 when the events are TCP Probes. This is causing pipeline errors:

Provided Grok expressions do not match field value: [Built inbound Probe TCP connection 412755190 for Inside:10.10.10.10/31547 (81.2.69.142/22617) to FTD-B-Outside:89.160.20.128/443 (89.160.20.128/443)]

The original event that caused the above parsing error is this:

<166>Feb 16 2024 23:16:39 firewall1 : %FTD-6-302013: Built inbound Probe TCP connection 412755162 for Inside:10.10.10.10/31547 (81.2.69.142/22617) to Outside:89.160.20.128/443 (89.160.20.128/443)

The GROK pattern with the issue is here:

https://github.com/elastic/integrations/blob/d72cb49e1dd8a7f2550bd33e7e0442896f0a2ee4/packages/cisco_ftd/data_stream/log/elasticsearch/ingest_pipeline/default.yml#L403

I would like to suggest it be changed to the following to account for the possible "Probe" string:

         - Built %{NOTSPACE}( Probe)? %{NOTSPACE:network.transport} connection %{NUMBER:_temp_.cisco.connection_id} for %{NOTCOLON:_temp_.cisco.source_interface}:%{IPORHOST:source.address}/%{NUMBER:source.port} \(%{IPORHOST:_temp_.natsrcip}/%{NUMBER:_temp_.cisco.mapped_source_port}\)(\(%{CISCO_USER:_temp_.cisco.source_username}\))? to %{NOTCOLON:_temp_.cisco.destination_interface}:%{NOTSPACE:destination.address}/%{NUMBER:destination.port} \(%{NOTSPACE:_temp_.natdstip}/%{NUMBER:_temp_.cisco.mapped_destination_port}\)(\(%{CISCO_USER:_temp_.cisco.destination_username}\))?( \(%{CISCO_USER:_temp_.cisco.termination_user}\))?%{GREEDYDATA}

[MakoWish]

The same issue exists for Message ID 302014.

https://github.com/elastic/integrations/blob/d72cb49e1dd8a7f2550bd33e7e0442896f0a2ee4/packages/cisco_ftd/data_stream/log/elasticsearch/ingest_pipeline/default.yml#L938-L944

I would like to also suggest the above be changed to this:

        - ^Teardown (Probe )?%{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:_temp_.cisco.connection_id} (?:for|from) %{NOTCOLON:_temp_.cisco.source_interface}:%{DATA:source.address}/%{NUMBER:source.port:int}\s*(?:\(?%{CISCO_USER:_temp_.cisco.source_username}\)? )?to %{NOTCOLON:_temp_.cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int}\s*(?:\(?%{CISCO_USER:_temp_.cisco.destination_username}\)? )?duration (?:%{DURATION:_temp_.duration_hms} bytes %{NUMBER:network.bytes}) %{NOTCOLON:event.reason} from %{NOTCOLON:_temp_.cisco.termination_initiator} \(%{CISCO_USER:_temp_.cisco.termination_user}\)
        - ^Teardown (Probe )?%{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:_temp_.cisco.connection_id} (?:for|from) %{NOTCOLON:_temp_.cisco.source_interface}:%{DATA:source.address}/%{NUMBER:source.port:int}\s*(?:\(?%{CISCO_USER:_temp_.cisco.source_username}\)? )?to %{NOTCOLON:_temp_.cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int}\s*(?:\(?%{CISCO_USER:_temp_.cisco.destination_username}\)? )?duration (?:%{DURATION:_temp_.duration_hms} bytes %{NUMBER:network.bytes}) %{NOTCOLON:event.reason} from %{NOTCOLON:_temp_.cisco.termination_initiator}
        - ^Teardown (Probe )?%{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:_temp_.cisco.connection_id} (?:for|from) %{NOTCOLON:_temp_.cisco.source_interface}:%{DATA:source.address}/%{NUMBER:source.port:int}\s*(?:\(?%{CISCO_USER:_temp_.cisco.source_username}\)? )?to %{NOTCOLON:_temp_.cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int}\s*(?:\(?%{CISCO_USER:_temp_.cisco.destination_username}\)? )?duration (?:%{DURATION:_temp_.duration_hms} bytes %{NUMBER:network.bytes}) %{NOTCOLON:event.reason} \(%{CISCO_USER:_temp_.cisco.termination_user}\)
        - ^Teardown (Probe )?%{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:_temp_.cisco.connection_id} (?:for|from) %{NOTCOLON:_temp_.cisco.source_interface}:%{DATA:source.address}/%{NUMBER:source.port:int}\s*(?:\(?%{CISCO_USER:_temp_.cisco.source_username}\)? )?to %{NOTCOLON:_temp_.cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int}\s*(?:\(?%{CISCO_USER:_temp_.cisco.destination_username}\)? )?duration (?:%{DURATION:_temp_.duration_hms} bytes %{NUMBER:network.bytes}) \(%{CISCO_USER:_temp_.cisco.termination_user}\)
        - ^Teardown (Probe )?%{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:_temp_.cisco.connection_id} (?:for|from) %{NOTCOLON:_temp_.cisco.source_interface}:%{DATA:source.address}/%{NUMBER:source.port:int}\s*(?:\(?%{CISCO_USER:_temp_.cisco.source_username}\)? )?to %{NOTCOLON:_temp_.cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int}\s*(?:\(?%{CISCO_USER:_temp_.cisco.destination_username}\)? )?duration (?:%{DURATION:_temp_.duration_hms} bytes %{NUMBER:network.bytes}) %{NOTCOLON:event.reason}
        - ^Teardown (Probe )?%{NOTSPACE:network.transport} (?:state-bypass )?connection %{NOTSPACE:_temp_.cisco.connection_id} (?:for|from) %{NOTCOLON:_temp_.cisco.source_interface}:%{DATA:source.address}/%{NUMBER:source.port:int}\s*(?:\(?%{CISCO_USER:_temp_.cisco.source_username}\)? )?to %{NOTCOLON:_temp_.cisco.destination_interface}:%{DATA:destination.address}/%{NUMBER:destination.port:int}\s*(?:\(?%{CISCO_USER:_temp_.cisco.destination_username}\)? )?duration (?:%{DURATION:_temp_.duration_hms} bytes %{NUMBER:network.bytes})
        - ^Teardown (Probe )?%{NOTSPACE:network.transport} connection for faddr (?:%{NOTCOLON:_temp_.cisco.source_interface}:)?%{ECSDESTIPORHOST}/%{NUMBER}\s*(?:\(?%{CISCO_USER:_temp_.cisco.destination_username}\)? )?gaddr (?:%{NOTCOLON}:)?%{MAPPEDSRC}/%{NUMBER} laddr (?:%{NOTCOLON:_temp_.cisco.source_interface}:)?%{ECSSOURCEIPORHOST}/%{NUMBER}\s*(?:\(%{CISCO_USER:_temp_.cisco.source_username}\))?(\s*type %{NUMBER:_temp_.cisco.icmp_type} code %{NUMBER:_temp_.cisco.icmp_code})?

[MakoWish]

To note... I have already made the above changes in our production cluster, and it does resolve the issue with no adverse affects. Hopefully the above change can be incorporated into the next update so our changes don't get overwritten.

[elasticmachine]

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

[jrmolin]

I have not been able to make the pipeline tests that we have pass with your suggested edits, though I am working on edits very similar to what you proposed. Thank you for the sample data and example grok patterns!

[MakoWish]

I have not been able to make the pipeline tests that we have pass with your suggested edits, though I am working on edits very similar to what you proposed. Thank you for the sample data and example grok patterns!

Interesting! It works perfectly fine on my end. I was getting thousands of event.type: pipeline_error on those messages, and as soon as I made those exact changes, the messages are being parsed properly now, and I am no longer getting any pipeline errors.

[jrmolin]

I'm new to the team, and I used the elastic-package test pipeline utility, and also the stack-based grok debugger. Your suggested edits only worked with (Probe )?? in the utility and not at all in the debugger. It's likely that there is a difference between my system and yours. Please feel free to comment on my PR if you have concerns! PR: https://github.com/elastic/integrations/pull/9223

[MakoWish]

That's interesting! I just tested locally, and it also fails for me with Elastic Package as you mentioned (I used v98.1). Must be an issue with Elastic Package, because the exact same changes work just fine in production for me. Oh, well! I'll let you run with it, hahaha.

[taylor-swanson]

      patterns:
        - Built %{NOTSPACE} (?:Probe )?%{NOTSPACE:network.transport} (... and so on)

This pattern worked fine for me (all tests pass). The space before (?:Probe )? needs to be preserved, which was likely throwing things off. I added those as suggested changes on the PR.

[jrmolin]

Package cisco_ftd - 3.1.3 containing this change is available at https://epr.elastic.co/search?package=cisco_ftd