GCP GKE INFO logs are showing ERROR

[jheyer159]

Followed the README to install the operator using kubectl and it seems to be running as normal, but every minute I'm getting a chunk of error logs that show INFO in the message and are showing as ERROR.

Is there something I need to change to correct the output?

...
logName: "projects/project_id/logs/stderr",
severity: "ERROR"
textPayload: "2023-03-10T14:26:47.498Z  INFO    ..."
...

[nmanoogian]

Hey @jheyer159! Thanks for writing in about this. The operator uses the go-logr to write those messages and it does look like they're being sent with log.Info which should end up on stdout.

Let me do a bit of digging and see if I'm able to reproduce this.

[mrasnake-doppler]

Hello @jheyer159, I wanted to update you as we were able to determine that logs sent with log.Info are sent to stderr which is likely why they are being categorized with severity: "ERROR".

However, this is the default setting and not something we are directly defining. So we are going to to do some research into the appropriate conventions for similar operators before deciding to make a change or not.

[jheyer159]

I tried applying this patch using kustomize

kind: Deployment
metadata:
  labels:
    control-plane: controller-manager
  name: doppler-operator-controller-manager
  namespace: doppler-operator-system
spec:
  template:
    spec:
      containers:
        - name: manager
          env:
            - name: ZAP_OPTIONS
              value: |
                {
                  "level": "info",
                  "outputPaths": ["stdout"],
                  "errorOutputPaths": ["stderr"],
                  "encoding": "console"
                }

it seems to be active, but i'm still getting error logs. Any idea if this should work? I'm not sure how to debug it further.

[nmanoogian]

@jheyer159 That could be an interesting solution! Where did you see the ZAP_OPTIONS environment variable? I couldn't find it in the zap or controller-runtime docs.

[jheyer159]

You're right - that was an incorrect suggestion from GPT. It seems the CLI options are found here and there's not one for changing DestWriter

https://github.com/kubernetes-sigs/controller-runtime/blob/0ef07535542bb44dc642fe5c67dcd4f7a9c06a22/pkg/log/zap/zap.go#L263

I will explore changing this in a fork.