Work around silent truncation

[mausch]

Datadog silently truncates log messages to 4000 characters (found this the hard way). Perhaps this integration library could work around that by emitting multiple messages whenever that limit is hit? Do you think this should be done by default? Should this be a flag in DatadogConfiguration?

[stefansedich]

Thanks @mausch

I never actually hit this issue myself, and with my Datadog use case I was not logging large log entries to it.

To me 4000 characters to me is a pretty large log, if we were to do the multiple entry approach how would we link them together?

[mausch]

It is pretty large, but if you think about stack traces 4KB is not enough. Also sometimes we crank up the logging to DEBUG-level which outputs Elasticsearch queries (for diagnosis) that are pretty big.

One way to correlate multiple entries would be with a Datadog tag with a GUID identifier value for each set of entries e.g. logentry:33b37ee7-6663-4077-bc91-d42eecffaa8f. We could make the tag name logentry configurable in DatadogConfiguration.

What do you think?

[stefansedich]

I must say I did not make use of Datadog for actually looking at logs, we would only log errors so that things like our Pagerduty alerting could be driven off these, I never logged info/debug to it.

We would use SEQ or Splunk to view and navigate logs as I found they are much better at doing this.

If you are interested submit a PR that covers this, I would like to see how the split logs look inside Datadog and how easy they were to navigate/work with before commiting to anything.

[mausch]

I've implemented this but then hit that other issue with the UDP datagram length limit. The event statsd API is just not suitable for this at all, so there's no point in adding this feature until #3 is solved.

[nblumhardt]

Closing this one as stale