Send messages to different slack channels / tags per team

[stephanwesten]

Hi,

We are looking into a way to send slack notifications to various application dev teams with only their violations. Let me explain in more detail:

In our case, a team can own one or more k8s namespaces. We put a label on the namespace to identify the team. What we would like to do is send a slack message to a team that violates some Kyverno rules (either audit mode or enforced). It could even be just a notification with a link to a Kyverno dashboard or another system, but at least the team should get notified - this is the crucial part. Another important aspect is that we do not want to bother a team with violations in namespaces they do not own.

Would something like this be possible? We can even contribute if necessary

[fjogeleit]

Hey,

it is possible to route notifications based on the namespace of a resource, but not based on labels of the resource nor namespace. The limitation is based on the information I get from the policy report which not includes any labels.

So with the current feature set you could do something like:

target:
  slack:
    minimumPriority: "warning"
    skipExistingOnStartup: true
    channels:
    - webhook: "https://hooks.slack.com/services/123..."
      filter:
        namespaces:
          include: ["team-a-*"]
    - webhook: "https://hooks.slack.com/services/456..."
      filter:
        namespaces:
          include: ["team-b-*"]

The namespaces filter support include and exclude, as you see you can use wildcards.

[stephanwesten]

thank you.

I want to avoid name-based conventions. That scales only to a certain degree. This is why we have the team labels.

We could update the 'include' entries of the configuration by a k8s job that runs every hour or so. The job would iterate across the namespaces and based on the labels make the appropriate change.

The solution is a bit lame, but probably would work. Or do you foresee any issues with this approach? Maybe we can share the job as an example for other companies having similar desires - not sure how generic we can make it.

[fjogeleit]

Yeah, its not optimal, I can have a look on a selector support by loading the ns and cache it for some time to do not to much overhead on the API server, but this will take some time.

So you could start with the current approach by name and I'll check in the next weeks an possible selector support.

[stephanwesten]

Ack,

thank you

[fjogeleit]

Hey @stephanwesten, I just want to mention that I added selector support for namespace filter, with a map of label/value pairs to the upcoming Policy Reporter v3 which is in an alpha phase, together with the new UI right now.

If you have a test/dev cluster and want to try it out you can find install instructions here:

https://github.com/kyverno/policy-reporter/tree/3.x

An example filter looks like:

target:
  telegram:
    chatID: "XXX"
    token: "X:X-X"
    minimumPriority: "warning"
    skipExistingOnStartup: true
    customFields:
      Cluster: Minikube
    filter:
      namespaces:
        selector:
          group: system

One minor things, there is currently an issue if your label contains a dot, its an issue in an underlaying dependency.

[stephanwesten]

That is great! The team just started looking into this. I will inform them.

Thanks!

[fjogeleit]

Great, let me know if you need any help or encounter any issue. I am glad to help.

[bkuzmic]

Hi @fjogeleit! I'm part of @stephanwesten team.

Thanks for the great work! I Just wanted to let you know that, even though I didn't test Slack target integration, I did try the UI custom board feature and it works ok.

There is just one annoying thing - the UI doesn't refresh if I specify the query parameters selecting only some kinds. For example, when I specify that I want to filter by kinds=Pod, like this: http://127.0.0.1:8081/#/custom-boards/team-garuda?kinds=Pod The UI doesn't populate the dropdown/selection box with the Kinds and doesn't filter only on the Pods.

[fjogeleit]

Hey @bkuzmic thanks a lot for your feedback, did you recently update to the latest version? I fixed an issue with the kind selector for custom boards in one of the latest releases.

[bkuzmic]

@fjogeleit I tried it on Friday - installed the 3.0.0-alpha.8. I'll check the latest release also.

[bkuzmic]

@fjogeleit It's still an issue.

If I open in browser: http://127.0.0.1:8081/#/custom-boards/team-garuda?kinds=Pod I see this:

But I should be seeing also all the failed reports on Pods:

Which I do, but only when I remove the Pods from Kinds list and add them again

[fjogeleit]

Okay, got it. Thanks for reporting. I will check it

[fjogeleit]

@bkuzmic I published a new helm version with a fix

[bkuzmic]

thanks @fjogeleit, it works now as expected.