Error when removing disk which is removed from the cluster

wimpers commented 8 years ago

On a working environment I physically removed a disk. Next I removed the disk from the ALBA backend throughthe GUI. The disk was removed but during the removal an error popped up. We should tackle this error and hide it.

wimpers commented 8 years ago

2016-08-04_10-50-36

khenderick commented 7 years ago

Should have been resolved by openvstorage/framework#901, packaged in openvstorage-2.7.3-rev.3978.4f93e02

JeffreyDevloo commented 7 years ago

Steps

Setup latest cluster
Configured cluster
Deleted asd disk
Tried to remove disk from backend

Output

Got selection_068

Logs:

ve_disk[11298937-4a10-47af-b7ae-420111ca2318] raised unexpected: RuntimeError('Disk /dev/disk/by-id/ata-QEMU_HARDDISK_2d2f4192-5d6c-11e6-8 not available on node 912f9ec0-b841-4c22-8957-947dcb2adbe2',)
Nov 22 13:31:33 ovs-node-1 celery[2109]: Traceback (most recent call last):
Nov 22 13:31:33 ovs-node-1 celery[2109]:   File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 240, in trace_task
Nov 22 13:31:33 ovs-node-1 celery[2109]:     R = retval = fun(*args, **kwargs)
Nov 22 13:31:33 ovs-node-1 celery[2109]:   File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 438, in __protected_call__
Nov 22 13:31:33 ovs-node-1 celery[2109]:     return self.run(*args, **kwargs)
Nov 22 13:31:33 ovs-node-1 celery[2109]:   File "/opt/OpenvStorage/ovs/lib/helpers/decorators.py", line 305, in new_function
Nov 22 13:31:33 ovs-node-1 celery[2109]:     output = function(*args, **kwargs)
Nov 22 13:31:33 ovs-node-1 celery[2109]:   File "/opt/OpenvStorage/ovs/lib/albanodecontroller.py", line 199, in remove_disk
Nov 22 13:31:33 ovs-node-1 celery[2109]:     raise RuntimeError('Disk {0} not available on node {1}'.format(device_alias, node.guid))
Nov 22 13:31:33 ovs-node-1 celery[2109]: RuntimeError: Disk /dev/disk/by-id/ata-QEMU_HARDDISK_2d2f4192-5d6c-11e6-8 not available on node 912f9ec0-b841-4c22-8957-947dcb2adbe2

Test result

Seems like the error is still present. You cannot delete a disk that was physically removed from a backend. Test failed.

Packages

openvstorage 2.7.6-rev.4323.a37c7f7-1 amd64 openvStorage
openvstorage-backend 1.7.6-rev.817.91abb7c-1 amd64 openvStorage Backend plugin
openvstorage-backend-core 1.7.6-rev.817.91abb7c-1 amd64 openvStorage Backend plugin core
openvstorage-backend-webapps 1.7.6-rev.817.91abb7c-1 amd64 openvStorage Backend plugin Web Applications
openvstorage-core 2.7.6-rev.4323.a37c7f7-1 amd64 openvStorage core
openvstorage-hc 1.7.6-rev.817.91abb7c-1 amd64 openvStorage Backend plugin HyperConverged
openvstorage-sdm 1.6.6-rev.449.77bd347-1 amd64 Open vStorage Backend ASD Manager
openvstorage-webapps 2.7.6-rev.4323.a37c7f7-1 amd64 openvStorage Web Applications

khenderick commented 7 years ago

After investigation:

Not the same issue as initially reported, but due to the delay in the verification, it seems this was more of a regression.
The issue is dat since the ASD Manager has no model, it cannot make the link anymore to a given partition (hosting the ASDs which can be discovered by reading fstab), and the disk device that is not longer present.

Issues solving this:

The ASD Manager can be changed to just not return any disk related data towards the OVS cluster
The OVS cluster can then detect based on it's model which disk is supposed to be present but is not reported anymore. It knows about the disk that is missing
The OVS cluster can then pass the disk information to the ASD Manager in a request to delete the Disk. However, the ASD Manager has no idea about what it needs to clean up, as it has no clue what partitions are supposed to be on that disk.

After a few tries making everything figure out what is going on, there's one thing I'd like to test further; that is whether the OVS cluster has enoug partition knowledge about the to-be-removed disk, and to see whether I can adapt both end that they also can handle partitions directly if disk information is not available.

If not working, I think the only solutions are:

Not cleaning up after ourself in case a disk is missing (so a mountpoint and invalid fstab entry are left)
Introducing a model on the ASD Manager side to keep track of disk/partition linkages. I think SQLite might be a solution here.

kvanhijf commented 7 years ago

https://github.com/openvstorage/alba-asdmanager/pull/188 --> openvstorage-sdm-1.6.8-rev.500.ffbd62d https://github.com/openvstorage/framework-alba-plugin/pull/305 --> openvstorage-backend-1.7.8-rev.879.74dc549

kvanhijf commented 7 years ago

@QA: ALBA Disks which have been physically pulled out, can now be deleted

JeffreyDevloo commented 7 years ago

Steps

Setup lastest cluster
Confiugred latest cluster
Removed disk from environment

Output

Dec 21 15:32:32 ovs-node-3 celery[1927]: 2016-12-21 15:32:32 80700 +0100 - ovs-node-3 - 1990/140334085199616 - lib/ensure single - 110 - INFO - Ensure single CHAINED mode - ID 1482330752_IvYKIlsQi8 - New task albanode.remove_disk w
ith params {'node_guid': '403880a4-0b70-4a40-a8d2-7915e2eabc45', 'device_alias': u'/dev/disk/by-id/ata-QEMU_HARDDISK_2d2f4192-5d6c-11e6-8'} scheduled for execution
Dec 21 15:32:32 ovs-node-3 celery[1927]: 2016-12-21 15:32:32 81000 +0100 - ovs-node-3 - 1990/140334085199616 - lib/ensure single - 111 - INFO - Ensure single CHAINED mode - ID 1482330752_IvYKIlsQi8 - Amount of jobs pending for key 
ovs_ensure_single_albanode.remove_disk: 1
Dec 21 15:32:32 ovs-node-3 celery[1927]: 2016-12-21 15:32:32 81100 +0100 - ovs-node-3 - 1990/140334085199616 - lib/ensure single - 112 - INFO - Ensure single CHAINED mode - ID 1482330752_IvYKIlsQi8 -   KWARGS: {'node_guid': '403880
a4-0b70-4a40-a8d2-7915e2eabc45', 'device_alias': u'/dev/disk/by-id/ata-QEMU_HARDDISK_2d2f4192-5d6c-11e6-8'}
Dec 21 15:32:33 ovs-node-3 celery[1927]: 2016-12-21 15:32:33 06300 +0100 - ovs-node-3 - 1990/140334085199616 - lib/ensure single - 113 - INFO - Ensure single CHAINED mode - ID 1482330753_rRXbztcPX6 - Amount of jobs pending for key 
ovs_ensure_single_ovs.disk.sync_with_reality: 0
Dec 21 15:32:33 ovs-node-3 celery[1927]: 2016-12-21 15:32:33 06700 +0100 - ovs-node-3 - 1990/140334085199616 - lib/ensure single - 114 - INFO - Ensure single CHAINED mode - ID 1482330753_rRXbztcPX6 - New task ovs.disk.sync_with_reality with params {'storagerouter_guid': u'aab8b444-b411-4cf6-9e71-13f9d5023c96'} scheduled for execution
Dec 21 15:32:33 ovs-node-3 celery[1927]: 2016-12-21 15:32:33 06900 +0100 - ovs-node-3 - 1990/140334085199616 - lib/ensure single - 115 - INFO - Ensure single CHAINED mode - ID 1482330753_rRXbztcPX6 - Amount of jobs pending for key ovs_ensure_single_ovs.disk.sync_with_reality: 1
Dec 21 15:32:33 ovs-node-3 celery[1927]: 2016-12-21 15:32:33 06900 +0100 - ovs-node-3 - 1990/140334085199616 - lib/ensure single - 116 - INFO - Ensure single CHAINED mode - ID 1482330753_rRXbztcPX6 -   KWARGS: {'storagerouter_guid': u'aab8b444-b411-4cf6-9e71-13f9d5023c96'}
Dec 21 15:32:39 ovs-node-3 celery[1927]: 2016-12-21 15:32:39 45700 +0100 - ovs-node-3 - 1990/140334085199616 - lib/disk - 117 - INFO - Disk vda - Found, updating
Dec 21 15:32:39 ovs-node-3 celery[1927]: 2016-12-21 15:32:39 48900 +0100 - ovs-node-3 - 1990/140334085199616 - lib/disk - 118 - INFO - Disk sde - Found, updating
Dec 21 15:32:39 ovs-node-3 celery[1927]: 2016-12-21 15:32:39 50600 +0100 - ovs-node-3 - 1990/140334085199616 - lib/disk - 119 - INFO - Disk sdb - Found, updating
Dec 21 15:32:39 ovs-node-3 celery[1927]: 2016-12-21 15:32:39 52200 +0100 - ovs-node-3 - 1990/140334085199616 - lib/disk - 120 - INFO - Disk sdd - Found, updating
Dec 21 15:32:39 ovs-node-3 celery[1927]: 2016-12-21 15:32:39 53800 +0100 - ovs-node-3 - 1990/140334085199616 - lib/disk - 121 - INFO - Disk sda - No longer found
Dec 21 15:32:39 ovs-node-3 celery[1927]: 2016-12-21 15:32:39 55500 +0100 - ovs-node-3 - 1990/140334085199616 - lib/disk - 122 - INFO - Disk sda - Deleted
Dec 21 15:32:39 ovs-node-3 celery[1927]: 2016-12-21 15:32:39 55600 +0100 - ovs-node-3 - 1990/140334085199616 - lib/disk - 123 - INFO - Disk sda - Found, updating
Dec 21 15:32:39 ovs-node-3 celery[1927]: 2016-12-21 15:32:39 57400 +0100 - ovs-node-3 - 1990/140334085199616 - lib/disk - 124 - INFO - Disk dm-0 - Found, updating
Dec 21 15:32:39 ovs-node-3 celery[1927]: 2016-12-21 15:32:39 59100 +0100 - ovs-node-3 - 1990/140334085199616 - lib/disk - 125 - INFO - Disk sdf - Found, updating
Dec 21 15:32:39 ovs-node-3 celery[1927]: 2016-12-21 15:32:39 60900 +0100 - ovs-node-3 - 1990/140334085199616 - lib/disk - 126 - INFO - Disk dm-1 - Found, updating
Dec 21 15:32:39 ovs-node-3 celery[1927]: 2016-12-21 15:32:39 62500 +0100 - ovs-node-3 - 1990/140334085199616 - lib/disk - 127 - INFO - Disk sdc - Found, updating
Dec 21 15:32:39 ovs-node-3 celery[1927]: 2016-12-21 15:32:39 65600 +0100 - ovs-node-3 - 1990/140334085199616 - lib/ensure single - 128 - INFO - Ensure single CHAINED mode - ID 1482330753_rRXbztcPX6 - Task ovs.disk.sync_with_reality finished successfully
Dec 21 15:32:39 ovs-node-3 celery[1927]: 2016-12-21 15:32:39 65800 +0100 - ovs-node-3 - 1990/140334085199616 - lib/ensure single - 129 - INFO - Ensure single CHAINED mode - ID 1482330753_rRXbztcPX6 - Amount of jobs pending for key ovs_ensure_single_ovs.disk.sync_with_reality: 0
Dec 21 15:32:39 ovs-node-3 celery[1927]: 2016-12-21 15:32:39 66200 +0100 - ovs-node-3 - 1990/140334085199616 - lib/ensure single - 130 - INFO - Ensure single CHAINED mode - ID 1482330752_IvYKIlsQi8 - Task albanode.remove_disk finished successfully
Dec 21 15:32:39 ovs-node-3 celery[1927]: 2016-12-21 15:32:39 66400 +0100 - ovs-node-3 - 1990/140334085199616 - lib/ensure single - 131 - INFO - Ensure single CHAINED mode - ID 1482330752_IvYKIlsQi8 - Amount of jobs pending for key ovs_ensure_single_albanode.remove_disk: 0
Dec 21 15:32:39 ovs-node-3 celery[1927]: 2016-12-21 15:32:39 67400 +0100 - ovs-node-3 - 1927/140334085199616 - celery/celery.worker.job - 709 - INFO - Task albanode.remove_disk[860eb4ae-c948-4694-8f97-e8044312da93] succeeded in 6.873141187s: None

Test result

Test passed.

Packages

ii  openvstorage                         2.7.9-rev.4463.0761b63-1        amd64        openvStorage
ii  openvstorage-backend                 1.7.9-rev.889.63f8d79-1         amd64        openvStorage Backend plugin
ii  openvstorage-backend-core            1.7.9-rev.889.63f8d79-1         amd64        openvStorage Backend plugin core
ii  openvstorage-backend-webapps         1.7.9-rev.889.63f8d79-1         amd64        openvStorage Backend plugin Web Applications
ii  openvstorage-core                    2.7.9-rev.4463.0761b63-1        amd64        openvStorage core
ii  openvstorage-hc                      1.7.9-rev.889.63f8d79-1         amd64        openvStorage Backend plugin HyperConverged
ii  openvstorage-sdm                     1.6.9-rev.508.9903a29-1         amd64        Open vStorage Backend ASD Manager
ii  openvstorage-webapps                 2.7.9-rev.4463.0761b63-1        amd64        openvStorage Web Applications

openvstorage / framework-alba-plugin