Using Serilog.Sinks.PeriodicBatching when Network Connectivity Might Be Unavailable or Metered at Times?

[mesheets]

What might be a "best practice" approach for using Serilog.Sinks.PeriodicBatching when the device that is running the software from which the logs are collected might be offline and/or on a metered connection for substantial portions of time? Ideally, logs would only be forwarded to Splunk when on an unmetered connection.

In looking at the code, Serilog.Sinks.PeriodicBatching.PeriodicBatchingSink instantiates a captive instance of the non-public class Serilog.Sinks.PeriodicBatching.BatchedConnectionStatus (e.g. no opportunity to inject an alternative implementation).

Several values within BatchConnectionStatus appear to be hard-coded, including the following:

• MinimumBackoffPeriod
• MaximumBackoffInterval
• FailuresBeforeDroppingBatch
• FailuresBeforeDroppingQueue

I understand reasons and use cases for backoff logic, but in scenarios with part of the time spent in offline or metered connectivity, a backoff scenario that is also dropping batches and queues would not be desirable. (Also, logs aren't the only data, being collected, and backend systems are designed for the type of usage in question.)

The following perhaps might be a ways to enable such capability with minimal disturbance to the existing code base, but might others more familiar with the code have other suggestions?

• Add the following properties to PeriodicBatchingSyncOptions:
  • MinimumBackoffPeriod : also provide a way to disable?
  • MaximumBackoffInterval : also provide a way to disable?
  • FailuresBeforeDroppingBatch : also provide a way to disable?
  • FailuresBeforeDroppingQueue : also provide a way to disable?
  • ShouldAttempt : this would be a lambda expression that would evaluate to a Boolean and would facilitate providing a custom set of criteria to use in determining whether or not to attempt to connect (e.g. if offline or on a metered connection, don't attempt).
• Update PeriodicBatchingSink.OnTick() to evaluate ShouldAttempt before executing the try/catch block that currently exists.

[nblumhardt]

Hi @mesheets, hope you were able to find a satisfactory solution! Closing this one as stale.

[mesheets]

Are there any best practices for dealing with metered connections, such as being able to configure in a way that transmission only occurs when on a non-metered connection?

[nblumhardt]

None I can pass along, but Serilog and its sinks are intended to be hackable. The core sink is only a few types, so grabbing those and tweaking them to add the features you need should be reasonably straightforward, as long as your EmitBatchAsync() can format up a payload for Splunk.

Another option, not 100% watertight, is to wrap the sink you're using with a (custom built) "queueing sink", which would queue events when a metered connection is detected, and pass them through otherwise. You'd still run the risk of the internal queue in the Splunk sink being flushed over the metered connection, but setting it to a small maximum size would help mitigate that.

Good luck with it!