search

splunk / splunk-connect-for-syslog

Splunk Connect for Syslog
Apache License 2.0
153 stars 108 forks source link

VMWare Logging Regression - vmware:esxlog:crond #2562

Closed tlay1 closed 1 month ago

tlay1 commented 2 months ago

I am unaware if this issue has been replicated by support.

SC4S v3.29.0 on OEL(RHEL) 7.9 on Docker and Podman no PCAP The issue is not believed to be related to external software. Data is not lost due to the issue. systems are 2-8 CPU, 8-32G mem lastchance is available if needed There are no customizations for the vmware sourcetypes. All of the default indexes are created

We noticed last month when we ran the latest image that our vmware logs were misclassified as index=osnix sourcetype=nix:syslog Whereas before it was... index=infraops sourcetype=vmware:esxlog:crond

This appears not to have impacted the other sourcetypes we have seen over the last 24 hours: vmware:cb:protect vmware:vgauthsvc.log vmware-vmsvc-root vmware-vmtools-root

We first noticed the issue when we were upgrading our sc4s images from v2->v3 a few months ago and believed it was related to 2390 so we downgraded all of our hosts to 2.46.0. We saw that 2390 merged with 2462 and was released with 2465 so we updated last night to the latest rev and the issue still occurred with the cron logs, which is why we wanted to file this report.

Some source data examples of the types of logs which had classifications change are:

- - CROND 22365 - [meta sequenceId="2267788"](syscheck) CMD (/usr/bin/system_check -q)
- - CROND 22366 - [meta sequenceId="2267695"](root) CMD (/usr/lib64/sa/sa1 1 1)
- - CROND 22364 - [meta sequenceId="2267681"](root) CMD (nice -n 19 ionice -c 3 /usr/sbin/logrotate -s /var/lib/logrotate-adm.status /etc/adm/adm_logrotate.conf)
- - CROND 22352 - [meta sequenceId="2271400"](root) MAIL (mailed 62 bytes of output but got status 0x0001
- - crond 2829 - [meta sequenceId="2271458"]sendmail: Cannot open localhost:25
- - CROND 23419 - [meta sequenceId="2369643"](root) CMD (run-parts /etc/cron.hourly)
- - CROND 23420 - [meta sequenceId="2369651"](root) CMD (/usr/bin/diskmonitor)
- - CROND 24630 - [meta sequenceId="2467463"](syscheck) CMD (/usr/bin/system_check -q)
- - CROND 26760 - [meta sequenceId="2658256"](syscheck) CMD (/usr/bin/system_check -q)
- - CROND 27785 - [meta sequenceId="2757280"](root) CMD (nice -n 19 ionice -c 3 /usr/sbin/logrotate -s /var/lib/logrotate-monitors.status /etc/monitors/monitors_logrotate.conf)
- - CROND 27787 - [meta sequenceId="2757391"](root) CMD (nice -n 19 ionice -c 3 /usr/share/ts/bin/asm_logrotate)
- - CROND 27789 - [meta sequenceId="2757393"](root) CMD (nice -n 19 ionice -c 3 /usr/sbin/logrotate -s /var/lib/logrotate-adm.status /etc/adm/adm_logrotate.conf)
- - CROND 27786 - [meta sequenceId="2757622"](root) CMD (nice -n 19 ionice -c 3 /usr/sbin/logrotate -s /var/lib/logrotate-bwafconf.status /etc/bwafconf/bwafconf_logrotate.conf)

To Reproduce Steps to reproduce the behavior:

  1. Go to /usr/lib/systemd/system/sc4s.service
  2. Change Environment="SC4S_IMAGE=ghcr.io/splunk/splunk-connect-for-syslog/container2:2.46.0" to Environment="SC4S_IMAGE=ghcr.io/splunk/splunk-connect-for-syslog/container3:latest"
  3. Run searches for: index=osnix sourcetype=nix:syslog index=infraops sourcetype=vmware:esxlog:crond
  4. Note the break in logs of the vmware:esxlog:crond logs and the new classification as nix:syslog

If it was some other odd sourcetype we could change it in the splunk_metadata.csv file, but we can't do that for nix:syslog as that is a common catchall.

Thanks! -Tony

cwadhwani-splunk commented 2 months ago

Please check this GitHub issue that was raised earlier. Since crond is a very generic program, we removed it earlier and also provided a custom local parser that can be used. Please refer to this comment.