This change allows users to inject a custom ITextFormatter implementation to change the output format of the logs before they are sent to Datadog.
Background
In order to correctly support this - our existing JSON serialization needed to be refactored.
Datadog has a set of reserved fields for logs. Previously these were injected alongside the user attributes in the final JSON payload. However this prevented us from using a custom text formatter as we would still need to inject these reserved fields which would conflict with the custom format.
To fix this - we need to change how logs are packaged. Datadog will treat any JSON in the message fields as JSON in the app and parse it appropriately. So the new flow is:
Receive log event
Apply the ITextFormatter to the log message
Encode the message in the message field of another JSON payload (+ the source/service/host/tags)
Send to backend
By doing this - users can format their logs any way they want (even non-json) while still preserving the source/service/host/tags on the log message. (This is how the datadog-agent works)
Challenges
As part of this change we need to have our own default JSON formatter (an ITextFormatter implementation). Ideally we would use serilog-expressions (the new recommended way to format logs) - however it does not support some old .net versions that we (and Serilog core) does.
The second option was to subclass JsonFormatter (the existing formatter we were using) - however this is deprecated.
As a result we have to use the recommended approach - a custom JsonValueFormatter based serializer that gives us the best balance of backwards compatibility and simplicity.
Ensure exceptions are handled by the global exception handler (they previously were not)
Drop Newtonsoft.Json as we are now dependent solely on Serilog APIs.
Future work
We should consider truncated logs like the Datadog agent does instead of dropping them and throwing an exception. Previously this was difficult because truncating the payload would damage the JSON encoding and lose the source/service/host/tags context (making them impossible to find). But now that logs are wrapped in a parent JSON structure - we can truncate logs without losing context.
This change allows users to inject a custom
ITextFormatterimplementation to change the output format of the logs before they are sent to Datadog.Background
In order to correctly support this - our existing JSON serialization needed to be refactored. Datadog has a set of reserved fields for logs. Previously these were injected alongside the user attributes in the final JSON payload. However this prevented us from using a custom text formatter as we would still need to inject these reserved fields which would conflict with the custom format.
To fix this - we need to change how logs are packaged. Datadog will treat any JSON in the
messagefields as JSON in the app and parse it appropriately. So the new flow is:ITextFormatterto the log messagemessagefield of another JSON payload (+ the source/service/host/tags)By doing this - users can format their logs any way they want (even non-json) while still preserving the source/service/host/tags on the log message. (This is how the datadog-agent works)
Challenges
As part of this change we need to have our own default JSON formatter (an
ITextFormatterimplementation). Ideally we would use serilog-expressions (the new recommended way to format logs) - however it does not support some old .net versions that we (and Serilog core) does.The second option was to subclass
JsonFormatter(the existing formatter we were using) - however this is deprecated.As a result we have to use the recommended approach - a custom
JsonValueFormatterbased serializer that gives us the best balance of backwards compatibility and simplicity.Additional changes
Newtonsoft.Jsonas we are now dependent solely on Serilog APIs.Future work
We should consider truncated logs like the Datadog agent does instead of dropping them and throwing an exception. Previously this was difficult because truncating the payload would damage the JSON encoding and lose the source/service/host/tags context (making them impossible to find). But now that logs are wrapped in a parent JSON structure - we can truncate logs without losing context.
Tests performed