GCP stackdriver logs for diag service are captured at the wrong level

[alexionescu]

Describe the bug The ambassador diag service will output everything as error severity to GCP logs due to outputting to stderr and the structured JSON field having levelname as the key used for logging severity. Per https://cloud.google.com/logging/docs/structured-logging#special-payload-fields the suggest field would be severity (I believe level can also be used). I have a PR here: https://github.com/emissary-ingress/emissary/pull/4941 which renames the field from levelname to severity which would annotate logs from this service at the right level.

[cindymullins-dw]

To check if I understand, are you saying that - of the severity levels noted below - only ERROR will be output currently? And that's because the wrong key is being used? Have you tested this - could you demonstrate some sample output that shows this?

DEFAULT | (0) The log entry has no assigned severity level. DEBUG | (100) Debug or trace information. INFO | (200) Routine information, such as ongoing status or performance. NOTICE | (300) Normal but significant events, such as start up, shut down, or a configuration change. WARNING | (400) Warning events might cause problems. ERROR | (500) Error events are likely to cause problems. CRITICAL | (600) Critical events cause more severe problems or outages. ALERT | (700) A person must take an action immediately. EMERGENCY | (800) One or more systems are unusable.

[LanceEa]

@alexionescu - Thanks for creating the issue and for the PR.

A few things from my perspective:

• I would want to make sure this is not a GCP only change, so can you look at the other major cloud vendors and or log providers to determine if making this change will work in other environments? Ideally, we try to avoid putting vendor specific hooks in if possible but I understand not everything is a one size fits all.
• If we can't find a non-vendor specific default, then I would prefer we keep the current defaults so we don't break users that maybe already make assumptions about the formatting of these log lines. We could provide a mechanism for users to provide a format similar to the way we support modifying the AccessLog through some Environment variable probably (but haven't put too much thought into that so open to suggestions).
• The PR is only adjusting the JSON formatter, if we were to accept something then I would want to have a consistent default format regardless of using json or console style logging so this would need to be extended.