Logs from LoggerJSON.Plug show as "" in Google Cloud Logging

[marceldegraaf]

We're using Elixir/Phoenix and LoggerJSON in our Plug pipelines to format log lines for Google Cloud Logging. These log lines used to be shown with proper labels in the GCL summary line, using fields from httpRequest:

However, today we noticed these logs are no longer shown that way in GCL, instead showing up as an empty string ("") with no labels applied:

Expanding one of these log lines still shows the required httpRequest fields are present, but GCL seems to take the jsonPayload.message (which is logged as an empty string in LoggerJSON.Plug) and for some reason no longer shows the httpRequest labels:

Things I've tried in a fork of LoggerJSON:

• Log nil in LoggerJSON.Plug, to see if that would make GCL disregard the jsonPayload.message and fall back to the properly formatted log line. This doesn't change anything.
• Remove the message field from the payload altogether, if it's empty. This makes GCL show the JSON payload as the log line, instead of showing the properly formatted log line.

Is anyone else experiencing these issues, and do you have any idea how these log lines should be formatted to get proper formatting back in GCL?

[marceldegraaf]

Update: I've spent the last few hours debugging this, pouring over documentation and trying to add/remove various fields in LoggerJSON to see if that helps. No luck so far.

I've opened a support ticket at Google Cloud, to check if this is caused by a recent change on their end. Will update this issue as I learn more.

[stevencorona]

Happens for me as well

[marceldegraaf]

Google Cloud Support confirmed this was caused by an error in Google Cloud Logging and Google Kubernetes Engine. It seems to have been fixed in the latest GKE updates.