search

sabre1041 / openshift-logforwarding-splunk

Demonstration of forwarding logs from OpenShift to Splunk
Apache License 2.0
30 stars 26 forks source link

No logs in Splunk #23

Closed dtrowbri closed 1 year ago

dtrowbri commented 1 year ago

Hello, I have followed the instructions and everything appears to have been deployed properly. Kibana does have logs, but I see no logs listed in the Splunk instance that was created. I verified that the HEC token is correct. There are no errors for the Fluentd Forwarders in either the logs or the events. I am at a loss and not sure to do or try next. I'm not sure if it is an issue with the forwarder or the Splunk instance/configurations. Is there some way I can test the Fluentd Forwarders to verify if they are or are not working?

sabre1041 commented 1 year ago

@dtrowbri can you check to see if there are any errors in the OpenShift fluentd DaemonSet?

dtrowbri commented 1 year ago

Hello @sabre1041, Thank you for your response. I rebuilt the EFK stack and the cluster log forwarders. Still not able to forward to Splunk, but was able to get some useful logs from the Fluentd DaemonSet. I think, based on these errors, I may need to disable the ssl option for the Fluentd forwarder or get a new cert. The Openshift cluster I am working on is 4.10.31, if that helps.

2022-10-03 21:18:19 +0000 [warn]: suppressed same stacktrace 2022-10-03 21:19:20 +0000 [warn]: [openshift_logforwarding_splunk] failed to flush the buffer. retry_times=23 next_retry_time=2022-10-03 21:20:25 +0000 chunk="5ea213e900c510ee01ad0b7387c05c89" error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired)"

This error is now showing in the logs for all pods in the Fluentd DaemonSet.

dtrowbri commented 1 year ago

I got it writing to the Splunk instance deployed on Openshift by running the helm upgrade command with the forwarding.fluentd.ssl=false option. I got more work to do to get this right for our environment, but I consider the issue I posted as resolved.