zabbix-tooling / k8s-zabbix

Kubernetes autodiscovery monitoring
GNU General Public License v2.0
18 stars 9 forks source link

Error with pvc and node discovery #23

Open options2015 opened 1 month ago

options2015 commented 1 month ago

Node error

Exception in thread Thread-1 (run):
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/threading.py", line 1031, in _bootstrap_inner
    self.run()
    ^^^^^^^^^^
  File "/app/base/watcher_thread.py", line 34, in run
    getattr(self.daemon_object, self.daemon_method)(self.resource)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/base/daemon_thread.py", line 264, in watch_data
    self.watch_event_handler(resource, obj)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/base/daemon_thread.py", line 332, in watch_event_handler
    self.send_object(resource, resourced_obj, event_type,
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/base/daemon_thread.py", line 495, in send_object
    self.send_data_to_zabbix(resource, obj=resourced_obj)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/base/daemon_thread.py", line 579, in send_data_to_zabbix
    metrics = obj.get_zabbix_metrics()
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/k8sobjects/node.py", line 60, in get_zabbix_metrics
    data = self.resource_data
           ^^^^^^^^^^^^^^^^^^
  File "/app/k8sobjects/node.py", line 36, in resource_data
    data = super().resource_data
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/k8sobjects/k8sobject.py", line 119, in resource_data
    raise RuntimeError("name_space is None for %s" % self.name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: name_space is None for aks--vmss000000

pvc errors


Traceback (most recent call last):
  File "/app/base/timed_threads.py", line 54, in run
    self.run_requests()
    ^^^^^^^^^^^^^^^^^^^
  File "/app/base/timed_threads.py", line 75, in run_requests
    getattr(self.daemon_object, self.daemon_method)(self.resource)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/base/daemon_thread.py", line 281, in watch_data
    pvc_volumes = get_pvc_volumes_for_all_nodes(api=api,
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/k8sobjects/pvc.py", line 102, in get_pvc_volumes_for_all_nodes
    pvc_volumes = _get_pvc_data_for_node(api=api, node=node,
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/k8sobjects/pvc.py", line 53, in _get_pvc_data_for_node
    pvc_volumes = _process_volume(item=item, namespace_exclude_re=namespace_exclude_re, node=node,
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/k8sobjects/pvc.py", line 86, in _process_volume
    volume['inodesUsed'] / volume['inodes'])) * 100
    ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~
ZeroDivisionError: division by zero`
``

The nodes are discovered but without data, but no pods are discovered at all.
scoopex commented 1 month ago

Thanks for your bug report. I'm sorry that you have problems. I reformatted your bug report to make it more readable.

You would probably have to delve deeper into the analysis of the problem yourself, as I currently do not have an Azure environment available.

The information provided is a bit too scarce to understand exactly what went wrong. It could be that k8s-zabbix has too few rights, but it could also be that certain structures of this Kubernetes no longer match AKS.

To find out, I recommend that you run the project locally and adjust the config so that you get the details.

The following parameters can help you in the configuration: (see https://github.com/zabbix-tooling/k8s-zabbix/blob/master/config-dev.ini)

debug = False
debug_k8s_events = False
# this excludes all ressources, remove the one you want to debug
resources_exclude = components, statefulsets, daemonsets, nodes, services, pvcs, deployments