search

spandex-project / spandex

A platform agnostic tracing library
MIT License
333 stars 53 forks source link

fix: logger metadata can mangle span_ids #117

Closed gerbal closed 3 years ago

gerbal commented 4 years ago

We are deploying Spandex to our elixir services and have discovered in some circumstances it the Elixir Logger can lose precision on very large integers. This has resulted in oddly truncated trace and span ids in our logs like 6123256812116107000. The underlying cause is unclear, but the whole issue can be avoided if the trace_id and span_id are stored as strings in logger metadata.

Is this a useful change for anyone else?

zachdaniel commented 4 years ago

@gerbal so the trace/span ids are in datadog w/o the trailing zeroes? I suspect its related to this https://github.com/spandex-project/spandex_datadog/blob/3344868ac5eca8f7ce6d9558d970edfd0d109073/lib/spandex_datadog/adapter.ex#L24

gerbal commented 4 years ago

The trace_id is actually 6123256812116107452 and what's logged is 6123256812116107000

zachdaniel commented 4 years ago

Yeah, that is really strange...I'd really suggest seeing if you can reproduce the specific issue of integers losing precision when being transferred to logger metadata. That would be an issue w/ Elixir/logger I imagine

GregMefford commented 4 years ago

I didn’t have a chance to try to repro this yet, but if you’re able to give some more details about what versions of Elixir and OTP you’re seeing this on, that would be helpful. I know there are some recent changes to the way Logger works and others may have just not updated/noticed yet if there are issues there. ❤️ 🚀

zachdaniel commented 4 years ago

Yeah, I tried to reproduce it but I couldn't.

aselder commented 3 years ago

The PR I just submitted is a dup of this. I can confirm the behavior, see #126 for details.

aselder commented 3 years ago

For clarity, the issues is with log ingesters (namely datadog), treating JSON integers as JS Numbers (aka float64) and losing precision.