splunk / splunk-sdk-csharp-pcl

Splunk's next generation C# SDK
https://dev.splunk.com/enterprise/docs/csharp
Apache License 2.0
64 stars 46 forks source link

Stopgap fix for graceful shutdown of modular inputs. #38

Closed brattonc closed 9 years ago

brattonc commented 9 years ago

This set of changes is a stopgap attempt to fix some of the issues around graceful shutdown of modular inputs when Splunk itself is shutting down. See this issue for details.

This change attempts to avoid the issue of lost events by:

  1. Providing a mechanism to stop writing events to Splunk. After the "Stop writing" is triggered, any queued events are simply not written, and thus are not passed to the progress reporter.
  2. Watching the Splunk Windows service to enter the Stopping or Stopped states. On average this gives the modular input 1-2 seconds lead time before Splunk signals the modular input to shut down via ctrl+break.
brattonc commented 9 years ago

I've been using this set of changes for a couple weeks now and I haven't experienced any issues with lost events since. And while this doesn't fix the root cause of the issue, a race condition still exists, the additional 1-2 seconds that watching the windows service provides is enough to prevent data loss in the vast majority of situations.