Adding Attributes at Root

[m-pedersen]

Hi

I have a use case where I would like to add more attributes on root level (like service and host) in the JSON sent to datadog. An example is I would like to know whether the logs come from the production or QA environment, or something else in the future. Using Enrich.WithProperty adds them in the Properties object, which is not what I need as other parts of my organization uses different tech than C# and are able to do this.

Is this possible in the current implemention?

If not, a suggestion could be allowing the user to provide at custom LogFormatterto handle this. A simpler solution could be something like a RootAttributesobject, containing the desired attributes and values. The values of the attributes does not need to be modifiable after configuration.

In case this is a desired feature, I will offer to do the implementation.

[dwainbrowne]

I have a similar request. I need to customize the tags on every request. I'm currently using Azure Functions with the ILogger implementation, and currently don't have a way to do this that I can see.

[ogaca-dd]

Hello @m-pedersen,

If you want to know if a log come from QA or production environment, you can use tags like"env:qa" or "env:prod". More information on tags are available at Getting started with tags.

Here is an example how to set tags:

var log = new LoggerConfiguration()
    .WriteTo.DatadogLogs(
        "<API_KEY>",
        source: "<SOURCE_NAME>",
        service: "<SERVICE_NAME>",
        host: "<HOST_NAME>",
        tags: new string[] {"env:qa"},
        configuration: config
    )
    .CreateLogger();

Let me know if I misunderstand your issue.

[ogaca-dd]

Hello @dwainbrowne,

Can you give us more context about your needs? What kind of tags would you like to send? Do you need to send log or sending a metric would be enough? In the later case maybe https://github.com/DataDog/dogstatsd-csharp-client can help.

[michalsteyn]

I just ran into the same issue.

The problem is that for dynamic context (for example a correlationID) there is already mechanisms for enriching your logs using enrichers. This ends up adding additional "Properties" on the Serilog LogEvent.

At the moment, the DataDot LogFormatter simply serializes the LogEvent this results in all the Enriched Fields showing up in DataDog as a nested attribute under Properties.

So imagine a situation where you are adding Global Context to a JS DataDog Logger sending the CorrelationID for all API Calls. This results in all logs coming from JS to have a field: CorrelationID that is added under the root level of the log.

On the back-end side, by adding an out of the box Serilog CorrelationID Enricher, results in that same field now added user Root->Properties->CorrelationID in the Log.

So you can no longer easily track the same fields coming from the front-end and the back-end.

So if there is a way to tell the DataDog Serilog Client to serialize the LogEvent in such a way that the Properties end up at the root level, then everything gets solved. One way to achieve this would be to make it possible to swap out the LogFormatter. That way we have more control over how we want to serialize the LogEvent.

I will create a PR for this.

[andreiardel]

I just ran into the same issue.

The problem is that for dynamic context (for example a correlationID) there is already mechanisms for enriching your logs using enrichers. This ends up adding additional "Properties" on the Serilog LogEvent.

At the moment, the DataDot LogFormatter simply serializes the LogEvent this results in all the Enriched Fields showing up in DataDog as a nested attribute under Properties.

So imagine a situation where you are adding Global Context to a JS DataDog Logger sending the CorrelationID for all API Calls. This results in all logs coming from JS to have a field: CorrelationID that is added under the root level of the log.

On the back-end side, by adding an out of the box Serilog CorrelationID Enricher, results in that same field now added user Root->Properties->CorrelationID in the Log.

So you can no longer easily track the same fields coming from the front-end and the back-end.

So if there is a way to tell the DataDog Serilog Client to serialize the LogEvent in such a way that the Properties end up at the root level, then everything gets solved. One way to achieve this would be to make it possible to swap out the LogFormatter. That way we have more control over how we want to serialize the LogEvent.

I will create a PR for this.

Is this still in progress?

[jzabroski]

@michalsteyn Thanks for your explanation above.

If I follow what you're saying, then "Standard Attributes" in the datadog logs UI won't get automatically populated if they're not also Serilog standard attributes. For example, Duration is a "Standard Attribute" in the datadog logs UI (seen screenshot below).

Apparently, Duration is a standard attribute, but when looking at the "send logs" HTTP API (https://docs.datadoghq.com/api/latest/logs/#send-logs), it doesn't specify Duration. This is kind of annoying, because this particular GitHub repo shows an "elapsed" example, which I think actually will put the data in the Properties section rather than the ddtags section (where I think it belongs?). It's super difficult to find the DataDog documentation where they explain the JSON log format, but it's located here: https://docs.datadoghq.com/logs/log_configuration/attributes_naming_convention/ under the Performance domain attributes

Duration is a unique attribute in that it doesn't always make sense to populate. Most logs are events, not activities. Duration should only be populated for activities.

Ultimately, I want to write "activity logging" code like @jonsequitur 's github.com/jonsequitir/Its.Log/ so that performance logging is a higher-order function of regular logging. And I'd like to use Log Categorization so I can bucket Duration by intervals, as seen here: https://www.datadoghq.com/blog/how-to-categorize-logs/#categorize-logs-based-on-response-duration

UPDATE: I found the original author if Serilog, Nick Blumhardt, created the nuget package SerilogTimings to address this from the caller perspective, but it does not address the main issue, which is adding attributes at the root so that they map to Standard/Reserved attributes: https://github.com/nblumhardt/serilog-timings

[jszwedko]

I believe this is achievable now by way of https://github.com/DataDog/serilog-sinks-datadog-logs/pull/86 but feel free to reopen this if I'm missing something.

[jzabroski]

Can I see some demo code?