Calculate safety results in RuntimeError: (Failure "unknown osd XXX")

JeffreyDevloo commented 7 years ago

Problem description

When running a scenario that initializes and claims asds multiple times with an increasing number of asds, I experienced a strange bug. All the calculate_safety calls return the calculate_safety error from an older, failed asd:

2016-10-17 13:22:18 51200 +0200 - cmp01 - 15218/140064603072320 - celery/celery.worker.job - 57096 - DEBUG - Task accepted: alba.calculate_safety[a24fa001-a8a4-42b6-b8d6-c07c6af5b248] pid:12791
2016-10-17 13:22:18 60400 +0200 - cmp01 - 12791/140064603072320 - celery/celery.redirected - 57108 - WARNING - 2016-10-17 13:22:18 60300 +0200 - cmp01 - 12791/140064603072320 - extensions/albacli - 57107 - ERROR - Error: (Failure "unknown osd 3cx6qk7om59tsTR9G6qDRHaB7HF4WM69")
Traceback (most recent call last):
  File "/opt/OpenvStorage/ovs/extensions/plugins/albacli.py", line 102, in run
    raise RuntimeError(output['error']['message'])
RuntimeError: (Failure "unknown osd 3cx6qk7om59tsTR9G6qDRHaB7HF4WM69")
2016-10-17 13:22:18 60400 +0200 - cmp01 - 12791/140064603072320 - celery/celery.redirected - 57110 - WARNING - 2016-10-17 13:22:18 60400 +0200 - cmp01 - 12791/140064603072320 - extensions/albacli - 57109 - DEBUG - Command: /usr/bin/alba get-disk-safety --config=arakoon://config/ovs/arakoon/mybackend-abm/config?ini=%2Fopt%2FOpenvStorage%2Fconfig%2Farakoon_cacc.ini --to-json --include-decommissioning-as-dead --long-id=ug6z1HwIXykMXLdP3tdliYjHfO3QhRH7 --long-id=zBuphlaYZIIHYVzpgNirPmLmUuDKBm90 --long-id=3cx6qk7om59tsTR9G6qDRHaB7HF4WM69
2016-10-17 13:22:18 60400 +0200 - cmp01 - 12791/140064603072320 - celery/celery.redirected - 57112 - WARNING - 2016-10-17 13:22:18 60400 +0200 - cmp01 - 12791/140064603072320 - extensions/albacli - 57111 - DEBUG - stderr:
2016-10-17 13:22:18 60500 +0200 - cmp01 - 12791/140064603072320 - celery/celery.redirected - 57114 - WARNING - 2016-10-17 13:22:18 60500 +0200 - cmp01 - 12791/140064603072320 - extensions/albacli - 57113 - DEBUG - stdout: {"success":false,"error":{"message":"(Failure \"unknown osd 3cx6qk7om59tsTR9G6qDRHaB7HF4WM69\")","exception_type":"unknown","exception_code":0}}
2016-10-17 13:22:18 64800 +0200 - cmp01 - 15218/140064603072320 - celery/celery.worker.job - 57097 - ERROR - Task alba.calculate_safety[a24fa001-a8a4-42b6-b8d6-c07c6af5b248] raised unexpected: RuntimeError(u'(Failure "unknown osd 3cx6qk7om59tsTR9G6qDRHaB7HF4WM69")',)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/albacontroller.py", line 1007, in calculate_safety
    safety_data = AlbaCLI.run(command='get-disk-safety', config=config, to_json=True, extra_params=extra_parameters)
  File "/opt/OpenvStorage/ovs/extensions/plugins/albacli.py", line 102, in run
    raise RuntimeError(output['error']['message'])
RuntimeError: (Failure "unknown osd 3cx6qk7om59tsTR9G6qDRHaB7HF4WM69")

A full traceback log of what happened:

2016-10-12 01:07:05 91000 +0200 - cmp01 - 2442/140064603072320 - lib/albanode - 5917 - DEBUG - Removing ASD 3cx6qk7om59tsTR9G6qDRHaB7HF4WM69 at node 172.19.5.30
2016-10-12 01:07:07 15500 +0200 - cmp01 - 2442/140064603072320 - celery/celery.redirected - 5919 - WARNING - 2016-10-12 01:07:07 15500 +0200 - cmp01 - 2442/140064603072320 - extensions/asdmanagerclient - 5918 - INFO - Request "get_
asds" took 1.23 seconds (internal duration 1.21 seconds)
2016-10-12 01:07:07 29600 +0200 - cmp01 - 2442/140064603072320 - lib/albanode - 5920 - DEBUG - Safety OK for ASD 3cx6qk7om59tsTR9G6qDRHaB7HF4WM69 on backend fcb9a487-a8ae-4745-b348-7427aed06584
2016-10-12 01:07:07 29600 +0200 - cmp01 - 2442/140064603072320 - lib/albanode - 5921 - DEBUG - Purging ASD 3cx6qk7om59tsTR9G6qDRHaB7HF4WM69 on backend fcb9a487-a8ae-4745-b348-7427aed06584
2016-10-12 01:07:07 34300 +0200 - cmp01 - 2442/140064603072320 - lib/albanode - 5922 - DEBUG - Removing ASD 3cx6qk7om59tsTR9G6qDRHaB7HF4WM69 from disk ata-SAMSUNG_MZ7LM480HCHP-00003_S1YJNX0H801481
2016-10-12 01:07:37 37700 +0200 - cmp01 - 2442/140064603072320 - lib/scheduled tasks - 5923 - ERROR - Ensure single CHAINED mode - ID 1476227225_UsiPKZbEdw - Task albanode.remove_asd with params {'node_guid': '7a200087-26d7-4938-89
a3-c18ead436870', 'asd_id': u'3cx6qk7om59tsTR9G6qDRHaB7HF4WM69', 'expected_safety': {u'good': 0, u'critical': 0, u'lost': 1}} failed
Traceback (most recent call last):
  File "/opt/OpenvStorage/ovs/lib/helpers/decorators.py", line 305, in new_function
    output = function(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/albanodecontroller.py", line 228, in remove_asd
    result = node.client.delete_asd(disk_id, asd_id)
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 185, in delete_asd
    return self._call(requests.post, 'disks/{0}/asds/{1}/delete'.format(disk_id, asd_id), timeout=30)
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 80, in _call
    response = method(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 107, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 449, in send
    raise ReadTimeout(e, request=request)
ReadTimeout: HTTPSConnectionPool(host='172.19.5.30', port=8500): Read timed out. (read timeout=30)
2016-10-12 01:07:37 37900 +0200 - cmp01 - 2442/140064603072320 - lib/scheduled tasks - 5924 - INFO - Ensure single CHAINED mode - ID 1476227225_UsiPKZbEdw - Amount of jobs pending for key ovs_ensure_single_albanode.remove_asd: 0
2016-10-12 01:07:37 39300 +0200 - cmp01 - 15218/140064603072320 - celery/celery.worker.job - 5894 - ERROR - Task albanode.reset_asd[af195654-919c-4e3a-b41f-cdcfddf4b987] raised unexpected: ReadTimeout(ReadTimeoutError('None: None',),)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/albanodecontroller.py", line 264, in reset_asd
    disk_id = AlbaNodeController.remove_asd(node_guid, asd_id, expected_safety)
  File "/usr/local/lib/python2.7/dist-packages/celery/local.py", line 188, in __call__
    return self._get_current_object()(*a, **kw)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 439, in __protected_call__
    return orig(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/task.py", line 420, in __call__
    return self.run(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/helpers/decorators.py", line 305, in new_function
    output = function(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/albanodecontroller.py", line 228, in remove_asd
    result = node.client.delete_asd(disk_id, asd_id)
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 185, in delete_asd
    return self._call(requests.post, 'disks/{0}/asds/{1}/delete'.format(disk_id, asd_id), timeout=30)
  File "/opt/OpenvStorage/ovs/extensions/plugins/asdmanager.py", line 80, in _call
    response = method(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 107, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 449, in send
    raise ReadTimeout(e, request=request)
ReadTimeout: None: None
2016-10-12 01:07:40 74200 +0200 - cmp01 - 15218/140064603072320 - celery/celery.worker.strategy - 5895 - INFO - Received task: albanode.reset_asd[f91dee25-85fa-4f8b-ad69-fd13fdcd04c2]
2016-10-12 01:07:40 74200 +0200 - cmp01 - 15218/140064603072320 - celery/celery.worker.autoscale - 5896 - INFO - Scaling down -1 processes.
2016-10-12 01:07:40 74300 +0200 - cmp01 - 15218/140064603072320 - celery/celery.pool - 5897 - DEBUG - TaskPool: Apply <function _fast_trace_task at 0x7f634f74dc80> (args:('albanode.reset_asd', 'f91dee25-85fa-4f8b-ad69-fd13fdcd04c2', ('7a200087-26d7-4938-89a3-c18ead436870', u'RQugCIKuMtwQuDxsivEwcp13rYQEyWDX', u'(Failure "unknown osd 3cx6qk7om59tsTR9G6qDRHaB7HF4WM69")'), {}, {'utc': True, u'is_eager': False, 'chord': None, u'group': None, 'args': ('7a200087-26d7-4938-89a3-c18ead436870', u'RQugCIKuMtwQuDxsivEwcp13rYQEyWDX', u'(Failure "unknown osd 3cx6qk7om59tsTR9G6qDRHaB7HF4WM69")'), 'retries': 0, u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'generic.#', u'exchange': u'generic'}, 'expires': None, u'hostname': 'celery@cmp01', 'task': 'albanode.reset_asd', 'callbacks': None, u'correlation_id': u'f91dee25-85fa-4f8b-ad69-fd13fdcd04c2', 'errbacks': None, 'timelimit': (None, None), 'taskset': None, 'kwargs': {}, 'eta': None, u'reply_to': u'a8b9fcb3-5e22-30bf-b1a0-039bcabf037b', 'id': 'f91dee25-85fa-4f8b-ad69-fd13fdcd04c2', u'headers': {}}) kwargs:{})
2016-10-12 01:07:40 74300 +0200 - cmp01 - 15218/140064603072320 - celery/celery.worker.job - 5898 - DEBUG - Task accepted: albanode.reset_asd[f91dee25-85fa-4f8b-ad69-fd13fdcd04c2] pid:11545
2016-10-12 01:07:40 74800 +0200 - cmp01 - 11545/140064603072320 - lib/scheduled tasks - 5887 - INFO - Ensure single CHAINED mode - ID 1476227260_ynMRUOj73G - Amount of jobs pending for key ovs_ensure_single_albanode.remove_asd: 0
2016-10-12 01:07:40 75000 +0200 - cmp01 - 11545/140064603072320 - lib/scheduled tasks - 5888 - INFO - Ensure single CHAINED mode - ID 1476227260_ynMRUOj73G - New task albanode.remove_asd with params {'node_guid': '7a200087-26d7-4938-89a3-c18ead436870', 'asd_id': u'RQugCIKuMtwQuDxsivEwcp13rYQEyWDX', 'expected_safety': u'(Failure "unknown osd 3cx6qk7om59tsTR9G6qDRHaB7HF4WM69")'} scheduled for execution
2016-10-12 01:07:40 75100 +0200 - cmp01 - 11545/140064603072320 - lib/scheduled tasks - 5889 - INFO - Ensure single CHAINED mode - ID 1476227260_ynMRUOj73G - Amount of jobs pending for key ovs_ensure_single_albanode.remove_asd: 1
2016-10-12 01:07:40 75100 +0200 - cmp01 - 11545/140064603072320 - lib/scheduled tasks - 5890 - INFO - Ensure single CHAINED mode - ID 1476227260_ynMRUOj73G -   KWARGS: {'node_guid': '7a200087-26d7-4938-89a3-c18ead436870', 'asd_id': u'RQugCIKuMtwQuDxsivEwcp13rYQEyWDX', 'expected_safety': u'(Failure "unknown osd 3cx6qk7om59tsTR9G6qDRHaB7HF4WM69")'}
2016-10-12 01:07:40 75300 +0200 - cmp01 - 11545/140064603072320 - lib/albanode - 5891 - DEBUG - Removing ASD RQugCIKuMtwQuDxsivEwcp13rYQEyWDX at node 172.19.5.30
2016-10-12 01:07:41 91800 +0200 - cmp01 - 11545/140064603072320 - celery/celery.redirected - 5893 - WARNING - 2016-10-12 01:07:41 91800 +0200 - cmp01 - 11545/140064603072320 - extensions/asdmanagerclient - 5892 - INFO - Request "get_asds" took 1.15 seconds (internal duration 1.14 seconds)
2016-10-12 01:07:41 97600 +0200 - cmp01 - 11545/140064603072320 - celery/celery.redirected - 5895 - WARNING - 2016-10-12 01:07:41 97600 +0200 - cmp01 - 11545/140064603072320 - extensions/albacli - 5894 - ERROR - Error: (Failure "unknown osd 3cx6qk7om59tsTR9G6qDRHaB7HF4WM69")
Traceback (most recent call last):
  File "/opt/OpenvStorage/ovs/extensions/plugins/albacli.py", line 102, in run
    raise RuntimeError(output['error']['message'])
RuntimeError: (Failure "unknown osd 3cx6qk7om59tsTR9G6qDRHaB7HF4WM69")
2016-10-12 01:07:41 97600 +0200 - cmp01 - 11545/140064603072320 - celery/celery.redirected - 5897 - WARNING - 2016-10-12 01:07:41 97600 +0200 - cmp01 - 11545/140064603072320 - extensions/albacli - 5896 - DEBUG - Command: /usr/bin/alba get-disk-safety --config=arakoon://config/ovs/arakoon/mybackend-abm/config?ini=%2Fopt%2FOpenvStorage%2Fconfig%2Farakoon_cacc.ini --to-json --include-decommissioning-as-dead --long-id=RQugCIKuMtwQuDxsivEwcp13rYQEyWDX --long-id=ug6z1HwIXykMXLdP3tdliYjHfO3QhRH7 --long-id=3cx6qk7om59tsTR9G6qDRHaB7HF4WM69
2016-10-12 01:07:41 97700 +0200 - cmp01 - 11545/140064603072320 - celery/celery.redirected - 5899 - WARNING - 2016-10-12 01:07:41 97600 +0200 - cmp01 - 11545/140064603072320 - extensions/albacli - 5898 - DEBUG - stderr:
2016-10-12 01:07:41 97700 +0200 - cmp01 - 11545/140064603072320 - celery/celery.redirected - 5901 - WARNING - 2016-10-12 01:07:41 97700 +0200 - cmp01 - 11545/140064603072320 - extensions/albacli - 5900 - DEBUG - stdout: {"success":false,"error":{"message":"(Failure \"unknown osd 3cx6qk7om59tsTR9G6qDRHaB7HF4WM69\")","exception_type":"unknown","exception_code":0}}
2016-10-12 01:07:41 97700 +0200 - cmp01 - 11545/140064603072320 - lib/scheduled tasks - 5902 - ERROR - Ensure single CHAINED mode - ID 1476227260_ynMRUOj73G - Task albanode.remove_asd with params {'node_guid': '7a200087-26d7-4938-89a3-c18ead436870', 'asd_id': u'RQugCIKuMtwQuDxsivEwcp13rYQEyWDX', 'expected_safety': u'(Failure "unknown osd 3cx6qk7om59tsTR9G6qDRHaB7HF4WM69")'} failed
Traceback (most recent call last):
  File "/opt/OpenvStorage/ovs/lib/helpers/decorators.py", line 305, in new_function
    output = function(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/albanodecontroller.py", line 216, in remove_asd
    final_safety = AlbaController.calculate_safety(alba_backend.guid, [asd_id])
  File "/usr/local/lib/python2.7/dist-packages/celery/local.py", line 188, in __call__
    return self._get_current_object()(*a, **kw)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 439, in __protected_call__
    return orig(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/task.py", line 420, in __call__
    return self.run(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/albacontroller.py", line 1007, in calculate_safety
    safety_data = AlbaCLI.run(command='get-disk-safety', config=config, to_json=True, extra_params=extra_parameters)
  File "/opt/OpenvStorage/ovs/extensions/plugins/albacli.py", line 102, in run
    raise RuntimeError(output['error']['message'])
RuntimeError: (Failure "unknown osd 3cx6qk7om59tsTR9G6qDRHaB7HF4WM69")
2016-10-12 01:07:41 97800 +0200 - cmp01 - 11545/140064603072320 - lib/scheduled tasks - 5903 - INFO - Ensure single CHAINED mode - ID 1476227260_ynMRUOj73G - Amount of jobs pending for key ovs_ensure_single_albanode.remove_asd: 0
2016-10-12 01:07:41 99300 +0200 - cmp01 - 15218/140064603072320 - celery/celery.worker.job - 5899 - ERROR - Task albanode.reset_asd[f91dee25-85fa-4f8b-ad69-fd13fdcd04c2] raised unexpected: RuntimeError(u'(Failure "unknown osd 3cx6qk7om59tsTR9G6qDRHaB7HF4WM69")',)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/albanodecontroller.py", line 264, in reset_asd
    disk_id = AlbaNodeController.remove_asd(node_guid, asd_id, expected_safety)
  File "/usr/local/lib/python2.7/dist-packages/celery/local.py", line 188, in __call__
    return self._get_current_object()(*a, **kw)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 439, in __protected_call__
    return orig(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/task.py", line 420, in __call__
    return self.run(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/helpers/decorators.py", line 305, in new_function
    output = function(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/albanodecontroller.py", line 216, in remove_asd
    final_safety = AlbaController.calculate_safety(alba_backend.guid, [asd_id])
  File "/usr/local/lib/python2.7/dist-packages/celery/local.py", line 188, in __call__
    return self._get_current_object()(*a, **kw)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 439, in __protected_call__
    return orig(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/task.py", line 420, in __call__
    return self.run(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/albacontroller.py", line 1007, in calculate_safety
    safety_data = AlbaCLI.run(command='get-disk-safety', config=config, to_json=True, extra_params=extra_parameters)
  File "/opt/OpenvStorage/ovs/extensions/plugins/albacli.py", line 102, in run
    raise RuntimeError(output['error']['message'])
RuntimeError: (Failure "unknown osd 3cx6qk7om59tsTR9G6qDRHaB7HF4WM69")

Calculating the safety for any ASD is hindered due to this exception. selection_108

The disk has been that is giving the error is now 'faulted' on the GUI. The first few entries in the full traceback also state that the disk has been successfully purged from Alba and removed from the model.

Possible root of the problem

Possibly a caching of the value due to the error?

Additional information

Setup

Hyperconverged setup

Package information

openvstorage 2.7.4-rev.4115.c0dda6a-1 amd64 openvStorage
openvstorage-backend 1.7.4-rev.748.59bc937-1 amd64 openvStorage Backend plugin
openvstorage-backend-core 1.7.4-rev.748.59bc937-1 amd64 openvStorage Backend plugin core
openvstorage-backend-webapps 1.7.4-rev.748.59bc937-1 amd64 openvStorage Backend plugin Web Applications
openvstorage-cinder-plugin 1.2.2-rev.38.dcc3b76-1 amd64 OpenvStorage Cinder plugin for OpenStack
openvstorage-core 2.7.4-rev.4115.c0dda6a-1 amd64 openvStorage core
openvstorage-hc 1.7.4-rev.748.59bc937-1 amd64 openvStorage Backend plugin HyperConverged
openvstorage-sdm 1.6.4-rev.383.24dfaaa-1 amd64 Open vStorage Backend ASD Manager
openvstorage-webapps 2.7.4-rev.4115.c0dda6a-1 amd64 openvStorage Web Applications

JeffreyDevloo commented 7 years ago

Removed the similarity as the tickets are not related.

khenderick commented 7 years ago

Probably some error after purging, before removal that now causes issues. We should handle this error on more places

khenderick commented 7 years ago

Fixed by #245, packaged in openvstorage-backend-1.7.4-rev.770.ae31d34

JeffreyDevloo commented 7 years ago

Steps

Setup latest cluster
Configured latest cluster
Manually purged an asd

alba purge-osd --long-id dGvFDhFFa2vznXT7DinSsYOnNZawjSCW --config arakoon://config/ovs/arakoon/mybackend02-abm/config?ini=%2Fopt%2FOpenvStorage%2Fconfig%2Farakoon_cacc.ini

Removed the asd

Output

2016-11-18 16:15:42 20300 +0100 - ovs-node-1 - 18225/139622502233856 - extensions/albacli - 0 - ERROR - Error: (Failure "unknown osd pte1QL4eIRL2V6J0GNY2D7BTGpTmI0Nn")
Traceback (most recent call last):
  File "/opt/OpenvStorage/ovs/extensions/plugins/albacli.py", line 102, in run
    raise RuntimeError(output['error']['message'])
RuntimeError: (Failure "unknown osd pte1QL4eIRL2V6J0GNY2D7BTGpTmI0Nn")
2016-11-18 16:15:42 20400 +0100 - ovs-node-1 - 18225/139622502233856 - extensions/albacli - 1 - DEBUG - Command: /usr/bin/alba get-disk-safety --config=arakoon://config/ovs/arakoon/mybackend02-abm/config?ini=%2Fopt%2FOpenvStorage%2Fconfig%2Farakoon_cacc.ini --to-json --long-id=pte1QL4eIRL2V6J0GNY2D7BTGpTmI0Nn
2016-11-18 16:15:42 20400 +0100 - ovs-node-1 - 18225/139622502233856 - extensions/albacli - 2 - DEBUG - stderr: 
2016-11-18 16:15:42 20400 +0100 - ovs-node-1 - 18225/139622502233856 - extensions/albacli - 3 - DEBUG - stdout: {"success":false,"error":{"message":"(Failure \"unknown osd pte1QL4eIRL2V6J0GNY2D7BTGpTmI0Nn\")","exception_type":"unknown","exception_code":0}}

Should have failed with (Failure "unknown osd pte1QL4eIRL2V6J0GNY2D7BTGpTmI0Nn")
2016-11-18 16:15:42 20400 +0100 - ovs-node-1 - 18225/139622502233856 - remove/ci_backend_remover - 4 - INFO - Removing asd pte1QL4eIRL2V6J0GNY2D7BTGpTmI0Nn for disk dd734841-1943-484b-94c7-185a6966434f
{u'status': u'PENDING', u'successful': False, u'failed': False, u'result': None, u'ready': False, u'id': u'65ee6a81-759f-4a3e-910c-723f4406426f'}
{u'status': u'PENDING', u'successful': False, u'failed': False, u'result': None, u'ready': False, u'id': u'141477d2-a93f-4fc2-a40d-b494361b8280'}
{u'status': u'STARTED', u'successful': False, u'failed': False, u'result': u"{'hostname': 'celery@ovs-node-3', 'pid': 9416}", u'ready': False, u'id': u'141477d2-a93f-4fc2-a40d-b494361b8280'}
{u'status': u'STARTED', u'successful': False, u'failed': False, u'result': u"{'hostname': 'celery@ovs-node-3', 'pid': 9416}", u'ready': False, u'id': u'141477d2-a93f-4fc2-a40d-b494361b8280'}
{u'status': u'STARTED', u'successful': False, u'failed': False, u'result': u"{'hostname': 'celery@ovs-node-3', 'pid': 9416}", u'ready': False, u'id': u'141477d2-a93f-4fc2-a40d-b494361b8280'}
{u'status': u'STARTED', u'successful': False, u'failed': False, u'result': u"{'hostname': 'celery@ovs-node-3', 'pid': 9416}", u'ready': False, u'id': u'141477d2-a93f-4fc2-a40d-b494361b8280'}
{u'status': u'STARTED', u'successful': False, u'failed': False, u'result': u"{'hostname': 'celery@ovs-node-3', 'pid': 9416}", u'ready': False, u'id': u'141477d2-a93f-4fc2-a40d-b494361b8280'}
{u'status': u'STARTED', u'successful': False, u'failed': False, u'result': u"{'hostname': 'celery@ovs-node-3', 'pid': 9416}", u'ready': False, u'id': u'141477d2-a93f-4fc2-a40d-b494361b8280'}
{u'status': u'STARTED', u'successful': False, u'failed': False, u'result': u"{'hostname': 'celery@ovs-node-3', 'pid': 9416}", u'ready': False, u'id': u'141477d2-a93f-4fc2-a40d-b494361b8280'}
{u'status': u'STARTED', u'successful': False, u'failed': False, u'result': u"{'hostname': 'celery@ovs-node-3', 'pid': 9416}", u'ready': False, u'id': u'141477d2-a93f-4fc2-a40d-b494361b8280'}
{u'status': u'STARTED', u'successful': False, u'failed': False, u'result': u"{'hostname': 'celery@ovs-node-3', 'pid': 9416}", u'ready': False, u'id': u'141477d2-a93f-4fc2-a40d-b494361b8280'}

Test results

Test passed.

Packages

openvstorage 2.7.6-rev.4320.8c4fb99-1 amd64 openvStorage
openvstorage-backend 1.7.6-rev.817.91abb7c-1 amd64 openvStorage Backend plugin
openvstorage-backend-core 1.7.6-rev.817.91abb7c-1 amd64 openvStorage Backend plugin core
openvstorage-backend-webapps 1.7.6-rev.817.91abb7c-1 amd64 openvStorage Backend plugin Web Applications
openvstorage-core 2.7.6-rev.4320.8c4fb99-1 amd64 openvStorage core
openvstorage-hc 1.7.6-rev.817.91abb7c-1 amd64 openvStorage Backend plugin HyperConverged
openvstorage-health-check 3.1.2-rev.227.f50ebb6-1 amd64 Open vStorage HealthCheck
openvstorage-sdm 1.6.6-rev.445.4216f7e-1 amd64 Open vStorage Backend ASD Manager
openvstorage-webapps 2.7.6-rev.4320.8c4fb99-1 amd64 openvStorage Web Applications

openvstorage / framework-alba-plugin