Error with stacktrace in gitjob pod after sending a webhook event with wrong credentials

thehejik commented 4 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

After sending a webhook event with wrong credentials, for eg. from Azure DevOps on push, there is following error in gitjob logs with stacktrace:

{
  "level": "error",
  "ts": "2024-05-29T15:19:24Z",
  "logger": "webhook",
  "msg": "Webhook processing failed",
  "error": "basic auth verification failed",
  "stacktrace": "github.com/rancher/fleet/pkg/webhook.(*Webhook).logAndReturn\n\t/home/runner/work/fleet/fleet/pkg/webhook/webhook.go:301\ngithub.com/rancher/fleet/pkg/webhook.(*Webhook).ServeHTTP\n\t/home/runner/work/fleet/fleet/pkg/webhook/webhook.go:183\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212\nnet/http.serverHandler.ServeHTTP\n\t/opt/hostedtoolcache/go/1.22.3/x64/src/net/http/server.go:3137\nnet/http.(*conn).serve\n\t/opt/hostedtoolcache/go/1.22.3/x64/src/net/http/server.go:2039"
}

Reproducer

install rancher:v2.9-c42447977787af1230f82e3dcf92c4895d5723df-head with fleet 104.0.0+up0.10.0-rc.14
login to rancher and let fleet to initialize
add a gitrepo pointing to a azure repository
configured HTTPS webhook for the azure repo on push event using auth credentials

Add a secret with the credentials into you cluster - just set another password

kubectl delete create secret generic gitjob-webhook -n cattle-fleet-system --from-literal=azure-username=username --from-literal=azure-password=wrong-password

configure gitjob to be able accept the webhook requests (expose gitjob on public address by using ngrok or so)
Either do a push or send a testing webhook request from Azure
See the logs kubectl logs -n cattle-fleet-system -lapp=gitjob

Expected Behavior

The error itself makes sense, but the stacktrace is a bit too much.

weyfonk commented 3 months ago

This comes from the way errors are logged through logr. We could mitigate this by making these Info logs, but those basic auth logs refer to actual errors.

Not sure there's an easy fix for this.

manno commented 3 months ago

I think we can configure our loggers. Both controllers should have these args, where --debug controls --zap-devel:

      --debug                             Turn on debug logging
      --debug-level int                   If debugging is enabled, set klog -v
      --zap-devel                         Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error) (default true)
      --zap-encoder encoder               Zap log encoding (one of 'json' or 'console')
      --zap-log-level level               Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
      --zap-stacktrace-level level        Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
      --zap-time-encoding time-encoding   Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'.

However

[ ] --debug-level doesn't set --zap-log-level (https://github.com/rancher/fleet/blob/main/internal/cmd/controller/gitops/operator.go#L79, https://github.com/rancher/fleet/blob/main/internal/cmd/controller/root.go#L92).
[ ] ~~if we are sure this is a user error, we could indeed change it to "info". Not sure if we want to, even if it's not a "fleet" error, it's still an error condition which needs the users attention?~~

rancher / fleet