search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.59k stars 584 forks source link

Logs are being sent to `stderr` when viewing in GCP Logs Explorer #9643

Open vuldin opened 1 year ago

vuldin commented 1 year ago

It looks like all Redpanda logs are shown as ERROR when viewing them in GCP Logs Explorer. The reason seems to be that the logs are using an unstructured textPayload and Google doesn't know how to parse this (and defaults to ERROR). image

Below is an example of another log event that uses jsonPayload and has not issue since the payload can be parsed: image

More details: https://cloud.google.com/logging/docs/structured-logging https://cloud.google.com/stackdriver/docs/solutions/gke/managing-logs#best_practices

JIRA Link: CORE-1228

dotnwat commented 1 year ago

we could do some structured logging to address this. grafana seems to be able to cope. maybe there are some gcp options we can use to fix it in the short term?

piyushredpanda commented 1 year ago

@weeco would you happen to know anything here? Or @sbocinec?

jcsp commented 1 year ago

So this is a duplicate of https://github.com/redpanda-data/redpanda/issues/3700, right? It's not a bug, just an observation that Redpanda doesn't do structured logging today.

jcsp commented 1 year ago

@vuldin is there anything strongly GCP specific about this, or should we close this one in favor of the existing #3700?