search

Nebo15 / logger_json

JSON logger formatter with support for Google Cloud, DataDog and other for Elixir.
https://nebo15.github.io/logger_json/
MIT License
237 stars 92 forks source link

duration_μs cannot be handled by DataDog #124

Closed pirvudoru closed 4 weeks ago

pirvudoru commented 2 months ago

Hello,

Seems that DataDog does not like the "duration_μs" as path anywhere -- cannot remap, cannot create a metric from it. -- From LoggerJSON.Plug

image image

Can we change it to something else ?

I've just updated from 5.1.3 to latest 6.03

AndrewDryga commented 2 months ago

I think a good way to solve this would be to sanitize all the metadata key names in DataDog adapter, a PR would be welcome.

AndrewDryga commented 1 month ago

Hello @bvobart @pirvudoru and @guzishiwo. Sorry for taking so long to reply. I pushed the simplest possible fix to main - replace μs with us. Most destinations do not support unicode so I feel like there is no good reason trying to fight to keep it.

Please give it a try. I will push the release to hex in a few days.

bvobart commented 1 month ago

Alright that sounds like a good solution for the general case, @AndrewDryga, but for the Elastic formatter, duration_us is still not a valid ECS field, so it would still be best to map that to event.duration. I don't think Elastic and Kibana will crash or error when they encounter duration_us, but they might well ignore it or at least not do anything useful with it. I'll rebase my MR (#129) and reopen it to fix that.

I see DataDog has a similar predefined format for a duration field (they recommend just using duration with the value in nanoseconds), so a similar remap will need to happen for the DataDog formatter. Same for the Google Cloud formatter too, though I'm not sure what field they use.

To be clear, I do think it's good to keep the generic duration field as duration_us because that at least specifies which unit the duration is in. But the product-specific formatters will need to adhere to those products' log specs.

EDIT: see https://github.com/Nebo15/logger_json/pull/132

AndrewDryga commented 1 month ago

@bvobart I'm sorry for missing that out, lets merge your fix.

AndrewDryga commented 1 month ago

TODO for myself:

AndrewDryga commented 4 weeks ago

The fix will be released in 6.2.0.