hurricanenick
commented
5 years ago https://www.function1.com/2017/12/tips-tricks-splunks-monitoring-console
This article has a section on how you would find a "missing" universal forwarder which may help in providing a more accurate output (if we want to get that granular). It uses the DMC asset lookup generator. We could bundle that in as a scheduled search to generate a file that we use for this functionality.
We should implement some kind of system that allows us to monitor the status of Splunk on a UF. If a UF is not sending any logs to the "_internal" index, that is a good indication that Splunk is not running, there is a problem with the configuration, or there is a network issue. It would make a lot of sense to aggregate all of this into a single alert instead of having multiple alerts for all of the data sources that UF would normally be sending (obviously if logs can't get to the indexers, everything will be "broken"). This kind of alert is a lot clearer on what issue to look into. Perhaps another way to think of this would be the heartbeat concept.