Not receiving Slack notifications

[CapKenR]

I deployed kubewatch via the Helm chart. I have it set to watch most resources and all namespaces. I see a bunch of Processing add to Event and Processing update to Event entries in the kubewatch pod log. However, I'm not receiving any messages in the Slack channel I specified in the configuration. And there are no Slack related messages in the pod log. How can I get more information about what's going on and/or troubleshoot this problem?

[arikalon1]

Hi @CapKenR There seems to be a bug in the latest chart version. As a workaround until we fix it, you can install the previous chart version by adding --version 3.3.5 to the helm install command

Please update if it works for you

[CapKenR]

Version 3.3.5 of the Helm chart works, i.e. I'm getting messages in Slack. Now I need to go back and figure out how to get rid of some of the noise, i.e. pods and jobs in one namespace. I did notice another issue which I'll post separately.

[CapKenR]

Version 3.3.5 of the Helm chart works, i.e. I'm getting messages in Slack. Now I need to go back and figure out how to get rid of some of the noise, i.e. pods and jobs in one namespace.

[juanpedrojose]

Same problem with 3.3.6. Getting log information as this: msg="Processing update to Event: preprod/chat-84476b5d98-pgnkl.173ec84990dee20d" pkg=kubewatch-Event But not getting messages to slack.

When using 3.3.5 I get the following in logs: invalid_auth Installation command: helm install --version=3.3.5 kubewatch robusta/kubewatch --set='rbac.create=true'

configmap:

 .kubewatch.yaml: |
   namespace: "dev,preprod"
   handler:
     slackwebhook:
       channel: "dev-cluster-events"
       slackwebhookurl: "https://hooks.slack.com/services/<...>"
   resource:
     clusterrole: false
     configmap: false
     daemonset: false
     deployment: true
     ingress: false
     job: false
     namespace: false
     node: true
     persistentvolume: false
     pod: false
     replicaset: false
     replicationcontroller: false
     secret: false
     serviceaccount: false
     services: false

[benm-stm]

Same issue here with latest and 3.3.5 version kubewatch logs time="2023-02-03T08:38:05Z" level=info msg="Processing add to pod: default/nginx2" pkg=kubewatch-pod time="2023-02-03T08:38:05Z" level=info msg="Processing update to pod: default/nginx2" pkg=kubewatch-pod time="2023-02-03T08:38:05Z" level=info msg="Processing update to pod: default/nginx2" pkg=kubewatch-pod time="2023-02-03T08:38:09Z" level=info msg="Processing update to pod: default/nginx2" pkg=kubewatch-pod

Configmap
`.kubewatch.yaml: |
    handler:
      slackwebhook:
        enabled: true
        channel: "#r-tests"
        username: "test"
        emoji: ":BOT:"
        slackwebhookurl: "https://hooks.slack.com/services/xxx"
    resource:
      daemonset: false
      deployment: true
      job: false
      persistentvolume: false
      pod: true
      replicaset: false
      replicationcontroller: false
      services: false
    namespace: ""

Any thoughts ?

[arikalon1]

Hi @benm-stm

We're aware of the issue with version 3.3.6. We're going to release a fix in the coming days.

Regarding version 3.3.5, can you please describe the issue you're experiencing ? Do you have the invalid_auth message in the logs? Are you just not receiving events on slack?

[benm-stm]

You can use the webhook conf instead, it works for me:

 data:
  .kubewatch.yaml: |
    handler:
      webhook:
        enabled: true
        url: "https://hooks.slack.com/services/xxxx/xxxx"
    resource:
      daemonset: false
      deployment: true
      job: false
      persistentvolume: false
      pod: true
      replicaset: false
      replicationcontroller: false
      services: false
    namespace: ""

[benm-stm]

Hi @arikalon1,

I just don't receive messages on slack, here is the whole stack

time="2023-02-03T08:38:01Z" level=info msg="Starting kubewatch controller" pkg=kubewatch-pod
time="2023-02-03T08:38:01Z" level=info msg="Starting kubewatch controller" pkg=kubewatch-deployment
time="2023-02-03T08:38:01Z" level=info msg="Processing add to deployment: kube-system/metrics-server" pkg=kubewatch-deployment
time="2023-02-03T08:38:01Z" level=info msg="Processing add to deployment: kube-system/traefik" pkg=kubewatch-deployment
time="2023-02-03T08:38:01Z" level=info msg="Processing add to deployment: default/kubewatch" pkg=kubewatch-deployment
time="2023-02-03T08:38:01Z" level=info msg="Processing add to deployment: kube-system/coredns" pkg=kubewatch-deployment
time="2023-02-03T08:38:01Z" level=info msg="Processing add to deployment: kube-system/local-path-provisioner" pkg=kubewatch-deployment
time="2023-02-03T08:38:01Z" level=info msg="Processing add to pod: kube-system/metrics-server-5f9f776df5-nwvcn" pkg=kubewatch-pod
time="2023-02-03T08:38:01Z" level=info msg="Processing add to pod: kube-system/svclb-traefik-05f0e883-zf9sn" pkg=kubewatch-pod
time="2023-02-03T08:38:01Z" level=info msg="Processing add to pod: default/kubewatch-6788c7ff4b-jj65n" pkg=kubewatch-pod
time="2023-02-03T08:38:01Z" level=info msg="Processing add to pod: kube-system/helm-install-traefik-crd-zgx8j" pkg=kubewatch-pod
time="2023-02-03T08:38:01Z" level=info msg="Processing add to pod: kube-system/helm-install-traefik-dhbnh" pkg=kubewatch-pod
time="2023-02-03T08:38:01Z" level=info msg="Processing add to pod: kube-system/coredns-597584b69b-lrng7" pkg=kubewatch-pod
time="2023-02-03T08:38:01Z" level=info msg="Processing add to pod: kube-system/local-path-provisioner-79f67d76f8-74dp6" pkg=kubewatch-pod
time="2023-02-03T08:38:01Z" level=info msg="Processing add to pod: kube-system/traefik-66c46d954f-x8twp" pkg=kubewatch-pod
time="2023-02-03T08:38:01Z" level=info msg="Kubewatch controller synced and ready" pkg=kubewatch-deployment
time="2023-02-03T08:38:01Z" level=info msg="Kubewatch controller synced and ready" pkg=kubewatch-pod
time="2023-02-03T08:38:02Z" level=info msg="Processing update to pod: default/kubewatch-6788c7ff4b-jj65n" pkg=kubewatch-pod
time="2023-02-03T08:38:02Z" level=info msg="Processing update to deployment: default/kubewatch" pkg=kubewatch-deployment
time="2023-02-03T08:38:05Z" level=info msg="Processing add to pod: default/nginx2" pkg=kubewatch-pod
time="2023-02-03T08:38:05Z" level=info msg="Processing update to pod: default/nginx2" pkg=kubewatch-pod
time="2023-02-03T08:38:05Z" level=info msg="Processing update to pod: default/nginx2" pkg=kubewatch-pod
time="2023-02-03T08:38:09Z" level=info msg="Processing update to pod: default/nginx2" pkg=kubewatch-pod

[arikalon1]

Thanks for the information @benm-stm

Can you try installing using the following:

helm repo add robusta https://robusta-charts.storage.googleapis.com && helm repo update
helm install kubewatch robusta/kubewatch --set='rbac.create=true,slack.channel=#YOUR_CHANNEL,slack.token=xoxb-YOUR_TOKEN,resourcesToWatch.pod=true,resourcesToWatch.daemonset=true,resourcesToWatch.deployment=true' --version 3.3.5

Please just replace your slack channel and slack token. Does it work for you?

[RoiGlinik]

@CapKenR this should be working for you now with the new version v2.1.0 thanks to @maxospiquante please reopen if thats no the case