Handle JSON serialization errors of log events

[angularsen]

When logging events that include an exception that has a getter-property that throws an exception, it results in a JSON serialization exception that drops the entire log event instead of just the problematic property.

This was observed with Akka.NET when there was a problem with instantiating a new actor due to missing dependency injection configuration.

2023-03-14T07:04:49.5779163Z Exception while emitting periodic batch from Serilog.Sinks.Logz.Io.LogzioSink: Newtonsoft.Json.JsonSerializationException: Error gett
ing value from 'Dispatcher' on 'Akka.Actor.DeadLetterMailbox'.
 ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Akka.Dispatch.Mailbox.get_Dispatcher()
   at lambda_method357(Closure , Object )
   at Newtonsoft.Json.Serialization.ExpressionValueProvider.GetValue(Object target)
...
   at Serilog.Sinks.Logz.Io.LogzIoSerializer.Serialize[T](T value)
   at Serilog.Sinks.Logz.Io.LogzioSink.FormatLogEvent(LogEvent loggingEvent)
   at System.Linq.Enumerable.SelectEnumerableIterator`2.ToArray()
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Serilog.Sinks.Logz.Io.LogzioSink.FormatPayload(IEnumerable`1 events)
   at Serilog.Sinks.Logz.Io.LogzioSink.EmitBatchAsync(IEnumerable`1 events)
   at Serilog.Sinks.PeriodicBatching.PeriodicBatchingSink.OnTick()

By handling JSON errors, we can omit the problematic property instead of dropping the entire log event. https://www.newtonsoft.com/json/help/html/serializationerrorhandling.htm

Changes

• Handle JSON serialization errors to only lose problematic properties instead of the whole event
• Move FormatPayload into try-catch to handle any other exceptions during formatting

Testing

• Add unit test case that failed before this fix
• The observed case with Akka.NET no longer drops the log event

[mantasaudickas]

Looks good. Even though - it feels a bit strange as constants handled exceptions might reduce whole application performance and it will be very hard to notice.. But seems like we do not have a lot of options here.

[angularsen]

To add some more context, these were the log sinks we used:

• Seq: OK
• Console: OK
• Application Insights trace: ❌ no log event (not sure about SelfLog for this one)
• logz.io (this): ❌ no log event, error described in SelfLog