MP+ logging to Splunk

[jpopelka]

Our image for sending logs to Splunk

• will reach End of Support on January 1, 2024
• tries to send the logs to http-inputs-rhcorporate.splunkcloud.com which is blocked by the firewall. The firewall allows http-inputs-osdsecuritylogs.splunkcloud.com which I tried, but our token doesn't work with it.

Per internal docs the logs from MP+ nodes should automatically go to Splunk if we log to stdout so we probably don't need to do the log-forwarding ourselves anymore.

For example this search query seems to work OK.

The task here is to:

• [ ] If the above query doesn't return any results, request access to rh_paas index for yourself and tell the other team members to do so as well
• [ ] Make sure our logs are there
• [ ] Update the docs
• [ ] Stop using the fluentd sidecar (just set the with_fluentd_sidecar: false as a start)

[mfocko]

IIRC the ESS requires logging, am I to naïve, if I expect them to have some logging in-place within the cluster?

[jpopelka]

The SEC-MON-REQ-1 (Logging & Monitoring) is only for RH Internal/Restricted classified data (not our case) and the SEC-NET-REQ-5 (Monitor Egress - outgoing internet traffic) is about logging outgoing network connections (i.e. logging on the infra/networking level, not application level, AFAIK), so we're not required to do that.

Yes, I was also expecting something more streamlined than "build your own splunk forwarder image from our Dockerfile and run it as a sidecar container in each pod".

[lachmanfrantisek]

Please, ask if you need any help when working on that.

[jpopelka]

I was either blind or reading too hastily because the docs is quite clear that we just need to log to stdout to see the logs in Splunk. I updated the description above.

[majamassarini]

Wait for both stg and prod instances be in MP+ before merging PR.

[mfocko]

We're actually using it for some time already, I just need to clean up my branch and merge it, so I think that we can close this as done. :)