search

cloudfoundry-community / splunk-firehose-nozzle

Send CF component metrics, CF app logs, and CF app metrics to Splunk
Apache License 2.0
29 stars 29 forks source link

[BUG] Events of other apps get lost when a single app is using Index Routing with invalid index #295

Closed JuergenSu closed 2 years ago

JuergenSu commented 2 years ago

What happened We have a foundation with 1000s of apps and a splunk nozzle reporting to the HEC of Splunk. Index Routing as described here https://github.com/cloudfoundry-community/splunk-firehose-nozzle#index-routing works for us. But if a single app specifies an environment variable with an illegal index, like SPLUNK_INDEX " " (empty) this events get included in to the post request sent to the HEC and the HEC discards the whole POST even if there are many other evnets in with no or correct index. This results in Loss of LogEvents transported to splunk, when a single application uses a wrong environment variable

What you expected to happen: Logs should not be lost when this feature is used wrong, or at least the documentation should warn about this effect.

How to reproduce it (as minimally and precisely as possible): have a CF foundation running some apps and have one app setting SPLUNK_INDEX to empty

Anything else we need to know?: Add any other context about the problem here.

Environment: This happens in every Version

luckyj5 commented 2 years ago

@JuergenSu Thanks for reporting this issue. Yes per app index routing feature relies on having the correct index in place when using this feature. We will update the docs for the same to make it more clear. Thank you!

hvaghani221 commented 2 years ago

@JuergenSu I have added this as warning in the docs. You can refer #292 Thanks.

JuergenSu commented 2 years ago

Thanks, i hope this helkps others because in large deployments with creative developers this can happen, For example when someone tries to avoid logs being sent to splunk and simply sets the index to something illegal but not using the F2S_DISABLE functionallity.

i guess this can be closed