search

krkn-chaos / cerberus

Guardian of Kubernetes clusters. Tool to monitor clusters health and signal/alert on failures.
Apache License 2.0
92 stars 41 forks source link

Slack Integration "Exception: name 'thread_ts' is not defined" #118

Open gnunn1 opened 3 years ago

gnunn1 commented 3 years ago

I'm integrating cerberus with my slack workspace and after working through things thought I had it working. However today cerberus started returning a no-go signal based on an issue and I was expecting to see it pop up in slack but didn't. Looking at the output for Cerberus I see this error message appearing regularly:

Feb 11 09:30:49 lab-server bash[368944]: 2021-02-11 09:30:49,399 [INFO] Encountered issues in cluster. Hence, setting the go/no-go signal to false
**Feb 11 09:30:49 lab-server bash[368944]: 2021-02-11 09:30:49,399 [INFO] Exception: name 'thread_ts' is not defined**
Feb 11 09:30:49 lab-server bash[369025]: 2021-02-11 09:30:49,756 [INFO] Iteration 205: Node status: True
Feb 11 09:30:50 lab-server bash[369028]: 2021-02-11 09:30:50,071 [INFO] Iteration 205: Cluster Operator status: True
Feb 11 09:30:50 lab-server bash[369027]: 2021-02-11 09:30:50,219 [INFO] Iteration 205: openshift-machine-api: True
Feb 11 09:30:50 lab-server bash[369032]: 2021-02-11 09:30:50,224 [INFO] Iteration 205: openshift-apiserver: True
Feb 11 09:30:50 lab-server bash[369033]: 2021-02-11 09:30:50,225 [INFO] Iteration 205: openshift-kube-controller-manager: True
Feb 11 09:30:50 lab-server bash[369029]: 2021-02-11 09:30:50,240 [INFO] Iteration 205: openshift-kube-apiserver: True
Feb 11 09:30:50 lab-server bash[369024]: 2021-02-11 09:30:50,242 [INFO] Iteration 205: openshift-sdn: True
Feb 11 09:30:50 lab-server bash[369026]: 2021-02-11 09:30:50,253 [INFO] Iteration 205: openshift-etcd: True
Feb 11 09:30:50 lab-server bash[369023]: 2021-02-11 09:30:50,253 [INFO] Iteration 205: openshift-kube-scheduler: True
Feb 11 09:30:50 lab-server bash[369031]: 2021-02-11 09:30:50,254 [INFO] Iteration 205: openshift-ingress: False
Feb 11 09:30:50 lab-server bash[369034]: 2021-02-11 09:30:50,295 [INFO] Iteration 205: openshift-monitoring: True
Feb 11 09:30:50 lab-server bash[368944]: 2021-02-11 09:30:50,296 [INFO] HTTP requests served: 1
Feb 11 09:30:50 lab-server bash[368944]: 2021-02-11 09:30:50,296 [INFO] Iteration 205: Failed pods and components
Feb 11 09:30:50 lab-server bash[368944]: 2021-02-11 09:30:50,296 [INFO] openshift-ingress: ['router-default-7645688499-8j5c4']
Feb 11 09:30:50 lab-server bash[368944]: 2021-02-11 09:30:50,296 [INFO]

I do not have any watcher or team alias defined in my config:

   slack_integration: True
    watcher_slack_ID:                                        
        Monday:
        Tuesday:
        Wednesday:
        Thursday:
        Friday:
        Saturday:
        Sunday:
    slack_team_alias:
chaitanyaenr commented 3 years ago

@gnunn1 Thanks for reporting the issue. I will run some tests to see what might be the issue and get back to you.

chaitanyaenr commented 3 years ago

@gnunn1 There have been couple of issues with the slack integration, this should get fixed in the latest images after https://github.com/cloud-bulldozer/cerberus/pull/125 is in.

amitsagtani97 commented 3 years ago

@gnunn1 hey, could you please confirm if you are receiving the same issue after the #125 fix?