search

openvstorage / openvstorage-health-check

The health check is classified as a monitoring and detection tool for Open vStorage.
3 stars 7 forks source link

ovs.dal.exceptions.ObjectNotFoundException during DTL checkup #193

Closed kinvaris closed 7 years ago

kinvaris commented 7 years ago

Problem description

This happens when running the healthchecks parallel on many nodes. Occurs on fargo

Possible root of the problem

Related to : https://github.com/openvstorage/openvstorage-health-check/issues/172

Logs

  File "/opt/OpenvStorage/ovs/lib/healthcheck.py", line 35, in <module>
    class HealthCheckController(object):
  File "/opt/OpenvStorage/ovs/lib/healthcheck.py", line 411, in HealthCheckController
    HealthCheckController.run_method(*arguments)
  File "/opt/OpenvStorage/ovs/lib/healthcheck.py", line 363, in run_method
    return HealthCheckController.check_attended()
  File "/usr/lib/python2.7/dist-packages/celery/local.py", line 188, in __call__
    return self._get_current_object()(*a, **kw)
  File "/usr/lib/python2.7/dist-packages/celery/app/task.py", line 420, in __call__
    return self.run(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/healthcheck.py", line 70, in check_attended
    return HealthCheckController.execute_check(unattended, silent_mode)
  File "/usr/lib/python2.7/dist-packages/celery/local.py", line 188, in __call__
    return self._get_current_object()(*a, **kw)
  File "/usr/lib/python2.7/dist-packages/celery/app/task.py", line 420, in __call__
    return self.run(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/healthcheck.py", line 108, in execute_check
    HealthCheckController.check_volumedriver(logger)
  File "/usr/lib/python2.7/dist-packages/celery/local.py", line 188, in __call__
    return self._get_current_object()(*a, **kw)
  File "/usr/lib/python2.7/dist-packages/celery/app/task.py", line 420, in __call__
    return self.run(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/healthcheck.py", line 179, in check_volumedriver
    VolumedriverHealthCheck.run(logger)
  File "/opt/OpenvStorage/ovs/extensions/healthcheck/volumedriver/volumedriver_health_check.py", line 193, in run
    VolumedriverHealthCheck.check_dtl(logger)
  File "/opt/OpenvStorage/ovs/extensions/healthcheck/volumedriver/volumedriver_health_check.py", line 52, in check_dtl
    vdisk = VDiskHelper.get_vdisk_by_guid(vdisk_guid)
  File "/opt/OpenvStorage/ovs/extensions/healthcheck/helpers/vdisk.py", line 66, in get_vdisk_by_guid
    return VDisk(vdisk_guid)
  File "/opt/OpenvStorage/ovs/dal/hybrids/vdisk.py", line 65, in __init__
    DataObject.__init__(self, *args, **kwargs)
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 233, in __init__
    self.__class__.__name__, self._guid
ovs.dal.exceptions.ObjectNotFoundException: VDisk with guid '9da5037f-8598-49ed-9b67-9cd02da631b5' could not be found

Additional information

Setup

Packages

root@stor-01:~# dpkg -l | grep openvstorage
ii  openvstorage                         2.7.6-fargo.2-1                         amd64        openvStorage
ii  openvstorage-backend                 1.7.6-fargo.1-1                         amd64        openvStorage Backend plugin
ii  openvstorage-backend-core            1.7.6-fargo.1-1                         amd64        openvStorage Backend plugin core
ii  openvstorage-backend-webapps         1.7.6-fargo.1-1                         amd64        openvStorage Backend plugin Web Applications
ii  openvstorage-core                    2.7.6-fargo.2-1                         amd64        openvStorage core
ii  openvstorage-hc                      1.7.6-fargo.1-1                         amd64        openvStorage Backend plugin HyperConverged
ii  openvstorage-health-check            3.1.2-fargo.14-1                        amd64        Open vStorage HealthCheck
ii  openvstorage-sdm                     1.6.6-fargo.2-1                         amd64        Open vStorage Backend ASD Manager
ii  openvstorage-webapps                 2.7.6-fargo.2-1                         amd64        openvStorage Web Applications
jeroenmaelbrancke commented 7 years ago

Open this ticket again: still not fixed on fargo repo.

ii  openvstorage-health-check            3.1.4-fargo.1-1                         amd64        Open vStorage HealthCheck

added 

VolumedriverHealthCheck.MACHINE_DETAILS.invalidate_dynamics('vdisks_guids')

in /ovs/extensions/healthcheck/volumedriver/volumedriver_health_check.py