[kube-prometheus-stack] Windows monitoring is not working on AKS ( level=error msg="failed to waitForNotification: callbackNumber does not exist in callbackMap") #4391
Describe the bug a clear and concise description of what the bug is.
Hi Team,
We are using the Prometheus-kube-stack chart for monitoring the metrics, it's working fine and gathering the metrics for all the pods pertaining to Linux nodes under node pools hosted in azure Kubernetes cluster, but its not monitoring any metrics for windows pods hosted in the windows nodes hosted in the same azure Kubernetes cluster.
When checked the pod logs of the windows exporter its throwing below error.
level=error msg="failed to waitForNotification: callbackNumber does not exist in callbackMap" callbackNumber=72
Describe the bug a clear and concise description of what the bug is.
Hi Team,
We are using the Prometheus-kube-stack chart for monitoring the metrics, it's working fine and gathering the metrics for all the pods pertaining to Linux nodes under node pools hosted in azure Kubernetes cluster, but its not monitoring any metrics for windows pods hosted in the windows nodes hosted in the same azure Kubernetes cluster.
When checked the pod logs of the windows exporter its throwing below error.
level=error msg="failed to waitForNotification: callbackNumber does not exist in callbackMap" callbackNumber=72
Detailed error information below.
windows-exporter ts=2024-03-26T08:16:36.027Z caller=config.go:63 level=info msg="Loading configuration file: C:\hpc/config.yml" ││ windows-exporter ts=2024-03-26T08:16:36.046Z caller=service.go:93 level=warn collector=service msg="No where-clause specified for service collector. This will ││ windows-exporter ts=2024-03-26T08:16:36.046Z caller=textfile.go:103 level=info collector=textfile msg="textfile collector directories: C:\hpc" ││ windows-exporter ts=2024-03-26T08:16:36.205Z caller=exporter.go:165 level=info msg="Running as NT AUTHORITY\SYSTEM" ││ windows-exporter ts=2024-03-26T08:16:36.205Z caller=exporter.go:172 level=info msg="Enabled collectors: cs, logical_disk, net, service, system, textfile, memor ││ windows-exporter ts=2024-03-26T08:16:36.206Z caller=exporter.go:225 level=info msg="Starting windows_exporter" version="(version=0.25.1, branch=heads/tags/v0.2 │ │ windows-exporter ts=2024-03-26T08:16:36.206Z caller=exporter.go:226 level=info msg="Build context" build_context="(go=go1.21.5, platform=windows/amd64, user=ru │ │ windows-exporter ts=2024-03-26T08:16:36.207Z caller=tls_config.go:313 level=info msg="Listening on" address=:9182 │ │ windows-exporter ts=2024-03-26T08:16:36.207Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=:9182 configure-firewall ││ configure-firewall ││ configure-firewall Name : {16776e90-a6dd-4c9e-9c09-07e4b53b75a0} │ │ configure-firewall DisplayName : windows-exporter │ │ configure-firewall Description : │ │ configure-firewall DisplayGroup : │ │ configure-firewall Group : │ │ configure-firewall Enabled : True │ │ configure-firewall Profile : Any ││ configure-firewall Platform : {} ││ configure-firewall Direction : Inbound ││ configure-firewall Action : Allow ││ configure-firewall EdgeTraversalPolicy : Block ││ configure-firewall LooseSourceMapping : False ││ configure-firewall LocalOnlyMapping : False ││ configure-firewall Owner : ││ configure-firewall PrimaryStatus : OK ││ configure-firewall Status : The rule was parsed successfully from the store. (65536) ││ configure-firewall EnforcementStatus : NotApplicable ││ configure-firewall PolicyStoreSource : PersistentStore ││ configure-firewall PolicyStoreSourceType : Local ││ configure-firewall │ │ configure-firewall │ │ configure-firewall ││ Stream closed EOF for prometheus/prometheus-prometheus-windows-exporter-42m22 (configure-firewall)
windows-exporter time="2024-03-26T08:19:58Z" level=error msg="failed to waitForNotification: callbackNumber does not exist in callbackMap" callbackNumber=72
Kindly do the needful.
Thanks Naresh
What's your helm version?
version.BuildInfo{Version:"v3.14.3", GitCommit:"f03cc04caaa8f6d7c3e67cf918929150cf6f3f12", GitTreeState:"clean", GoVersion:"go1.21.7"}
What's your kubectl version?
Client Version: v1.29.1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.3
Which chart?
prometheus-kube-stack
What's the chart version?
kube-prometheus-stack-57.1.1
What happened?
No response
What you expected to happen?
No response
How to reproduce it?
No response
Enter the changed values of values.yaml?
windowsMonitoring: enabled: true
prometheus-windows-exporter: prometheus: monitor: enabled: true jobLabel: jobLabel releaseLabel: true podLabels: jobLabel: windows-exporter config: |- collectors: enabled: '[defaults],memory,container'
Enter the command that you execute and failing/misfunctioning.
We are running the prometheus-kube-stack chart with the values mentioned above from Octopus deploy .
Anything else we need to know?
No response