Losing log messages

[provegard]

Hi!

Every now and then, we're unable to find log messages in StackDriver. Our containers log to stdout/stderr, so we can see that we have a text log message, but the corresponding structured JSON log entry isn't there.

We turned on Serilogs's SelfLog logging, and got this:

2022-02-04T15
54:43.1948133Z Exception while emitting periodic batch from Serilog.Sinks.PeriodicBatching.PeriodicBatchingSink: Grpc.Core.RpcException: Status(StatusCode="InvalidArgument", Detail="Log entry with size 2.30M exceeds maximum size of 256.0K", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1643990083.192067968","description":"Error received from peer ipv4:[xx.xx.xx.xx:443](http://xx.xx.xx.xx:443/)","file":"/var/local/git/grpc/src/core/lib/surface/call.cc","file_line":1062,"grpc_message":"Log entry with size 2.30M exceeds maximum size of 256.0K","grpc_status":3}")

I'm not sure if it's the total of all entries sent to StackDriver that exceeds the max, or if it is a single entry in the batch. If it's the total, would it be possible to calculate the size beforehand and send in smaller chunks? If it's a single entry that is too big, would it be possible to drop just that one (with some suitable error logging) but send the rest?

[manigandham]

Thanks for reporting. That error is for a single line (logged as JSON structure), you can see the quotas here: https://cloud.google.com/logging/quotas

I'll see what I can do to handle it. 2.3 megabytes is very large by the way, just FYI in case you weren't expecting something that big to be logged.

[provegard]

Thanks! Yes, we'll adjust our logging to not exceed that limit.

[manigandham]

Version 3.7.0 has been released with checks for log entry size. The max size is approximate so I've set it to 250KB to be safe with API overhead. Oversized messages will be skipped and logged to Serilog's debug log.

You will still need to enable that debug log and monitor it to see if large messages are still occuring.

Let me know if that resolves it for you.

[provegard]

Very nice, thank you!