Missing/broken disk used by vPool causes error during rescan

JeffreyDevloo commented 8 years ago

Problem description

I've added disks to my node and got 'An error occurred while rescanning. selection_036

/var/log/upstart/ovs-workers.log :

2016-08-22 13:55:40 05300 +0200 - ovs-node1 - 15206/140602219763520 - celery/celery.worker.job - 104 - ERROR - Task ovs.disk.sync_with_reality[861492e5-28cc-4f34-8248-37769d9c487f] raised unexpected: LinkedObjectException('There are 3 items left in self.storagedrivers',)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/helpers/decorators.py", line 301, in new_function
    output = function(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/disk.py", line 211, in sync_with_reality
    partition.delete()
  File "/opt/OpenvStorage/ovs/dal/dataobject.py", line 700, in delete
    raise LinkedObjectException('There {0} left in self.{1}'.format(multi, key))
LinkedObjectException: There are 3 items left in self.storagedrivers

Possible root of the problem

A disk/ partition seems to be missing. OVS will attempt to fix this in the model (by removing the information) but it's not able to due to a link to the storagedriver (most likely the used/reserved size).

Additional information

Setup

Hyperconverged setup

Three nodes with each three disks for the back-end
Package information
openvstorage 2.7.2-rev.3859.42b3488-1 amd64 openvStorage
openvstorage-backend 1.7.2-rev.675.37ca5b8-1 amd64 openvStorage Backend plugin
openvstorage-backend-core 1.7.2-rev.675.37ca5b8-1 amd64 openvStorage Backend plugin core
openvstorage-backend-webapps 1.7.2-rev.675.37ca5b8-1 amd64 openvStorage Backend plugin Web Applications
openvstorage-cinder-plugin 1.2.2-rev.32.948a8c1-1 amd64 OpenvStorage Cinder plugin for OpenStack
openvstorage-core 2.7.2-rev.3859.42b3488-1 amd64 openvStorage core
openvstorage-hc 1.7.2-rev.675.37ca5b8-1 amd64 openvStorage Backend plugin HyperConverged
openvstorage-health-check 2.0.0-rev.117.3212ea9-1 amd64 Open vStorage HealthCheck
openvstorage-sdm 1.6.2-rev.330.f06c8de-1 amd64 Open vStorage Backend ASD Manager
openvstorage-test 2.7.2-rev.980.0dbc80f-1 amd64 openvStorage autotest suite
openvstorage-webapps 2.7.2-rev.3859.42b3488-1 amd64 openvStorage Web Applications

JeffreyDevloo commented 8 years ago

khenderick commented 8 years ago

Actually, I'm not sure whether this can or should be resolved as such; if a disk is in use by a vPool and for some reason that disk breaks, we have currently no functionality in place that can cope with the failed disk (we don't have disk replacements on a StorageRouter yet). Any thoughts?

wimpers commented 8 years ago

After discussion with @khenderick : this should be tackled in a broader scope where we monitor the disks used by storage routers https://github.com/openvstorage/framework/issues/775

JeffreyDevloo commented 7 years ago

Closing because the broader scope has been tackled.

openvstorage / framework