search

elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
32 stars 448 forks source link

F5's logs (using syslog) are not parsed #7236

Open daixque opened 1 year ago

daixque commented 1 year ago

Overview

We are using F5 and trying to onboard the logs to Elasticsearch via Elastic Agent. We realized there are (at least) 2 types of logs:

  1. Telemetry streaming data, which is target of our integration
  2. Usual log data which is available via syslog

My understanding is our integration supports only 1st one. But we need to onboard 2nd one as well. So it will be super helpful for us if integration team introduces new input for 2nd usual syslog parsing.

Sample logs

Currently we are using Custom TCP Logs integration to collect the syslog. Below are just example of the syslog messages which are taken from message field. Note that these are not complete list of the message type.

info logger[32640]: [ssl_acc] 10.200.20.200 - admin [03/Aug/2023:15:31:17 +0800] "/mgmt/shared/file-transfer/ucs-downloads/f5.bigiq-analytics-BIG-IQ.gz" 200 1111
info logger[32641]: [ssl_req][03/Aug/2023:15:31:17 +0800] 10.247.22.238 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "/mgmt/shared/file-transfer/ucs-downloads/f5.bigiq-analytics-BIG-IQ.gz" 1111
info dhclient[11078]: XMT: Solicit on mgmt, interval 111800ms.
notice tmsh[9707]: 01420002:5: AUDIT - pid=9707 user=root folder=/ module=(tmos)# status=[Command OK] cmd_data=cd / ;
info
info systemd[1]: Starting user-0.slice.
info CROND[31708]: (root) CMD (/usr/bin/diskmonitor)
debug perl[31708]: OpenSSL is initialized in FIPS mode.
elasticmachine commented 1 year ago

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

andrewkroh commented 1 year ago

Could you provide some more samples, but captured from the event.original field so that we can get the complete raw data. In the integration settings you would need to toggle on the "preserve original event" setting.

Also would it be possible to share some screenshots of the available configuration settings. I did not get a lot information from the docs about what is configurable (like can you choose tcp, tcp+tls, udp; are there tcp framing options?, can you pick and choose logs types, etc.). This will also help us in writing the setup documentation in the README.

nicpenning commented 11 months ago

How can these be retrieved using the agent today? I have gathered some logs using the Custom TCP logs and Syslog Parsing enabled. Is that close enough to the raw event?

nicpenning commented 11 months ago

Looks like I found the log messages source :) https://techdocs.f5.com/kb/en-us/products/big-ip_ltm/releasenotes/related/log-messages.html

Sample RFC5424 message. This could take a awhile to implement.

<134>1 2023-12-28T16:06:44-06:00 InternalF5a tmm 10564 - - Rule /Common/apinodogov_2pool <HTTP_REQUEST>: API.nodo.gov allowed from IP: 192.168.226.128 - TCPClientPayload: - TCPClientPort: 55304