This update allows fw1-loggrabber to output CheckPoint logs over TCP.
I implemented this due to the number of issues I was having trying to process the file output, too many edge cases lead to eventual loss of log data, especially when using builtin log rotation, logrotated, or anything else. Even logrotated with pre/post scripts completely stopping and starting the service was becoming an issue. Due to the 32bit process limitation in only being able to write up to 2GB files I've had to resort to restarting services every 10 minutes* to prevent fw1 from crashing at my log ingest rate. With TCP output I no longer have to deal with any of these issues, also with this option there's much less IO involved which makes the SSD's happy.
I didn't implement UDP because when payloads are too large the data is sent with multiple packets (as expected). The issue is that so far every SIEM or log platform I've used can't seem to deal with single log events sent over multiple UDP packets and I always see those systems trying to parse truncated log events - so there's no point in using UDP for Checkpoint logs, which I know will always be very large.
The combination of this change and wrapping fw1 with a systemd service file to auto-recover from the occasional unexpected failure turns this into a decently resilient service and I hope others will find this useful. I'm also working on having the Makefile install a systemd service for fw1 automatically, I'll also create a pull request when I have that working.
This update allows fw1-loggrabber to output CheckPoint logs over TCP.
I implemented this due to the number of issues I was having trying to process the file output, too many edge cases lead to eventual loss of log data, especially when using builtin log rotation, logrotated, or anything else. Even logrotated with pre/post scripts completely stopping and starting the service was becoming an issue. Due to the 32bit process limitation in only being able to write up to 2GB files I've had to resort to restarting services every 10 minutes* to prevent fw1 from crashing at my log ingest rate. With TCP output I no longer have to deal with any of these issues, also with this option there's much less IO involved which makes the SSD's happy.
I didn't implement UDP because when payloads are too large the data is sent with multiple packets (as expected). The issue is that so far every SIEM or log platform I've used can't seem to deal with single log events sent over multiple UDP packets and I always see those systems trying to parse truncated log events - so there's no point in using UDP for Checkpoint logs, which I know will always be very large.
The combination of this change and wrapping fw1 with a systemd service file to auto-recover from the occasional unexpected failure turns this into a decently resilient service and I hope others will find this useful. I'm also working on having the Makefile install a systemd service for fw1 automatically, I'll also create a pull request when I have that working.