search

app-insights-php / monolog-handler

Microsoft App Insights monolog handler for trace tracking
MIT License
3 stars 7 forks source link

Truncate long messages, instead of dropping. #11

Open meidlinga opened 3 years ago

meidlinga commented 3 years ago

Hello,

As mentioned in the readme, here and here, log messages are dropped, if they exceed a certain size limit.

What is the reason behind dropping, instead of truncating them?

thanks and kind regards

norberttech commented 3 years ago

hey @meidlinga that's excellent question, I don't remember now all details but something tells me this was the easiest and most reliable solution.

TellemetryData is an array data structure that is internally serialized by app insights sdk to JSON. Truncating would need to be done at the data level, but how to select which part should be trimmed and how much, to make output JSON smaller than 64Kb?

Any thoughts?

meidlinga commented 3 years ago

Thanks for the fast response. I understand the problem now. To keep complexity manageable and runtime fast, I would like to propose the following:

The number of messages shall be limited with a configurable value. A good default value would be 20. This prevents taking to long in case of a lot of small elements. Those 20 fields would then be trimmed in relation to their total length. A free quota of for example 200 bytes would ensure short elements are kept as is, reducing the budget for the other fields.

If you like this approach, I could work on a PR some time soon.

norberttech commented 3 years ago

First of all, thanks for thinking this one through! I'm afraid it's a bit more complex than that. So the thing is that Each TelemetryData object represents one "message". It can be one of the following types:

To visualize the problem better, please look at following code: 

TelemetryData::message($message, $properties);

TelemetryData::message('message', ['foo' => 'foo', 'bar' => 'bar', 'message' => 'this is your log entry')]);

Under the hood, a limitation of 64kb is set to the combination of message and properties serialized together into one json string. So it's not really about the number of messages but the size of a single message. In order to solve that problem, you would need to choose what to trim, message, or maybe properties? And you would need to make that decision across different types of TelemetryData types.

Does it make sense?