Events are not being filtered by namespace or log level

framp commented 3 months ago

Environment

I'm running on Ubuntu 22.04 with microk8s 1.30/edge

  sentry-logging:
    Image ID:       docker.io/getsentry/sentry-kubernetes@sha256:6ac37974fd2af86eff6db6424949f5741fab3589fd5a5a693f4d406698d78f60
    Environment:
      SENTRY_DSN:                   <set to the key 'sentry-dsn' in secret 'sentry-dsn'>  Optional: false
      SENTRY_ENVIRONMENT:           production
      SENTRY_K8S_WATCH_NAMESPACES:  default
      SENTRY_K8S_LOG_LEVEL:         error
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dfm96 (ro)

Steps to Reproduce

Setup a new microk8s cluster
Create a daemonset for sentry

Expected Result

I would expect the sentry agent to forward events only from the default namespace with severity error or above

Actual Result

I get on sentry info and warnings event from containers running in the kube-system namespace

Namespace filtering doesn't seem to work (I've tried to leave the variable unset too but the default is default so nothing changes). I'm not sure if my understanding of LOG_LEVEL variable is correct. Is it supposed to filter events? If so, filtering is not working there too.

theneva commented 3 months ago

I'm not a maintainer of the project, but… I interpret SENTRY_K8S_LOG_LEVEL as a filter for the log messages outputted by the sentry-kubernetes process/container/deployment itself – not as a filter for which events get exported to Sentry.

I have no idea about the namespace filtering though.

Edit: I think maybe the image hosted on Docker Hub is the old/legacy Python version of the app, with different environment variables. The Go image that's documented in the readme seems to only be uploaded to GitHub's container registry. Maybe try with the Go image? The latest one at the time of writing is ghcr.io/getsentry/sentry-kubernetes:c71f3b24737e972262b094d36767394d570e24db

tonyo commented 2 months ago

As @theneva pointed out, DockerHub currently only hosts the legacy Python implementation (https://github.com/getsentry/sentry-kubernetes/tree/master/py-legacy) To use the latest version, please use the GHCR-hosted images: https://github.com/getsentry/sentry-kubernetes/pkgs/container/sentry-kubernetes (we'll need to clarify this in README)

SENTRY_K8S_LOG_LEVEL only controls the logging level of the (stdout) output of the agent itself.

Regarding filtering events by type, there are actually only two types of those: "Normal" and "Warning"; sentry-kubernetes currently only reports "Warning" events (that include the events that we generally treat as errors).

theneva commented 2 months ago

@tonyo it's only half-related to this issue, but… In addition to clarifying which image to use in the readme, I would greatly appreciate some kind of named/versioned tags for the new images published to ghcr – the hashes make the deployment pretty hard to maintain, since there's no quick way to know how my deployed image is – or if there's a new version 😅

tonyo commented 2 months ago

Noted, thanks for the feedback. Moved to a separate issue: https://github.com/getsentry/sentry-kubernetes/issues/94

getsentry / sentry-kubernetes