search

elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
23 stars 435 forks source link

Sophos integration for XG firewalls does not work due to Syslog RFC5424, plus other issues #3482

Closed colin-stubbs closed 1 year ago

colin-stubbs commented 2 years ago

Sophos XG firewalls send syslog RFC5424 by default, refer to: https://docs.sophos.com/nsg/sophos-firewall/18.5/Help/en-us/webhelp/onlinehelp/AdministratorHelp/SystemServices/LogSettings/index.html

The current Sophos integration for XG firewalls uses the default delimiter of \n, when it should default to using \0 as specified by RFC5424.

As such the integration just does not work out of the box, and the logged content is buffered by the filebeat TCP input until either a \n is received, or the collection timeout occurs. In which case content parsing fails due to the garbled mess of multiple messages.

By adding the following custom TCP options it can be made to work,

framing: delimiter
line_delimiter: "\0"

However due to the configuration of custom TCP options, the Elastic Agent rendering of the integration config for filebeat fails to include the pipeline name now, so it needs to be manually specified.

e.g. I now need to use the following custom TCP options to get logs parsed properly.

framing: delimiter
line_delimiter: "\0"
pipeline: logs-sophos.xg-2.2.1

Obviously, this kind of static reference to the pipeline version is not ideal and it will stop working if the integration gets upgraded and the 2.2.1 pipeline ceases to exist.

In addition, the event.timezone is statically set to +00:00 so when logs are logged in a non-UTC timezone they get the incorrect timestamp.

The integration is missing a webUI option to configure event.timezone field value, so this is statically set by Elastic Agent / the integration as +00:00.

So when firewalls that log in local time, e.g. +10:00, send logs; @timestamp winds up being that many hours wrong.

I have had to duplicate all ingest pipelines as custom variants, and point the integration at the initial entry ingest pipeline which now looks at event.custom_timezone instead of event.timezone.

Much fail for this integration in the current state.

elasticmachine commented 2 years ago

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

taylor-swanson commented 2 years ago

@colin-stubbs As far as I know, RFC 5424 doesn't dictate anything about framing. For TCP, it appears it's ambiguous (either non-transparent framing or octet counting may be used (https://datatracker.ietf.org/doc/html/rfc6587#section-3.4)), and TLS should use octet counting (https://datatracker.ietf.org/doc/html/rfc5425#section-4.3). Overriding the delimiter in the TCP config is the correct way to handle this. We can change the default in this case if we know the firewall is sending \0.

Defining the pipeline name is not necessary in the TCP input config. Agent will automatically configure filebeat to forward to the data stream, which will in turn run through the pipeline.

The sample logs that I have access to have the time zone already defined in the message. For example:

<30>device="SFW" date=2020-05-18 time=14:38:36 timezone="CEST" device_name="XG230" ...

The pipeline will use the time zone provided there and fallback to UTC if it's missing. We can provide a field in Kibana for it, but I find it odd that it's missing from the original message.

Would you be willing to provide some example logs (with any sensitive information redacted)? We can feed these into our tests to prevent this from being an issue in the future.