Truncate messages that are too long + make max message size configurable

[gh123man]

This change adds:

• Configurable max message size (default 256k)
• Truncates that are larger than the max message size instead of dropping them and throwing an exception.
• Enforces the maximum payload count of 1000 messages (as per our docs)
• Raises the max payload size to 5MB instead of 2MB (as per our docs - see above)
• Refactors payload assembly to be cleaner and easier to understand + adde tests.

Messages that exceed the max message size will now be split into chunks with ...TRUNCATED... appended and or prepended to the truncated chunks. This is consistent with the datadog-agent behavior. With this change - logs will never be dropped.

[strigona]

I'm not sure if there's a better place to discuss (I can open an issue if that's better?), but this feature is kind of awkward as splitting the the JSON request in the middle breaks the ingestion and yields 2 log entries with no parsed metadata, just Source & Service by the looks of it.

Any correlation ID's are lost and all parts of the split message is not consumable if you have pretty much any filters.

That being said, I don't really have a good solution for how to handle this any better at the Sink level...

[gh123man]

@strigona You are absolutely correct. This is a challenging issue. The reason this behavior was introduced to the serilog sink is that it aligns with how the Datadog agent behaves. We are tracking this and investigating better ways to handle such cases.

In the short term there are a couple ways to deal with this if you are hitting the size limits:

1. you can raise maxMessageSize up to 1MB to get more headroom.
2. you can introduce a custom log formatter with your own truncation rule to ensure the payload lands under the maximum size. (we can't include this by default because it's impossible decide what part of the payload to truncate if we overflow).
3. Similar to 2 - you can use a custom formatter to drop the template and properties fields which will reduce the final payload size by ~50% (because it's a duplicate of what appears in the rendered message).

[strigona]

@gh123man Thanks for the response! I'm glad it's on your radar, it's certainly an awkward problem. I will increase the maxMessageSize for now and see how frequently we encounter truncation/splitting.