search

hpe-storage / python-lefthandclient

A Python REST Client to talk to the HPE StoreVirtual / LeftHand OS REST Service
Apache License 2.0
4 stars 4 forks source link

Error when deleting a snapshot that was used to create a volume #2

Open aorourkehp opened 8 years ago

aorourkehp commented 8 years ago

If you create a snapshot from a volume, then create a volume from that snapshot and try and delete the snapshot, you put the snapshot in error_deleting state in cinder and leave an orphaned snapshot in lefthand.

You would run into this when you create snapshots of an important volume. If something ever went wrong, you would want to be able to create a volume from one of those snapshots in order to recover the data. At this point, cinder allows you to delete either the snapshot or the newly created volume while lefthand doesn't. It seems like cinder thinks that both volumes are completely independent as well as the new volume and the snapshot are no longer related, while lefthand must maintain the snapshot. In lefthand, you must delete the newly created volume before you can delete the snapshot.

How to reproduce:

Create a lefthand volume: cinder create --volume-type lefthand 1

Create a snapshot of that volume: cinder snapshot-create <volume-id>

Create a volume from that snapshot: cinder create --snapshot-id <snapshot-id>

Delete the snapshot: cinder snapshot-delete <snapshot-id>

The snapshot goes into error_deleting state and we get the following error:

$ cinder snapshot-list
+--------------------------------------+--------------------------------------+----------------+------+------+
|                  ID                  |              Volume ID               |     Status     | Name | Size |
+--------------------------------------+--------------------------------------+----------------+------+------+
| 66ec2d8e-4ef2-452c-a4ef-ab9b1b2af00c | 5d97285a-51f5-4cdc-a5f1-ae3928daf874 | error_deleting |  -   |  1   |
+--------------------------------------+--------------------------------------+----------------+------+------+

2015-08-07 08:41:09.777 ERROR oslo_messaging.rpc.dispatcher [req-e31c99ee-718a-4b3e-bd9c-86ab676e8ac9 3e8bbfa9c7de4ddfa48904b3ca7f74d5 f620ce8af410457b808326f9ef6fdc7f] Exception during message handling: argument of type 'NoneType' is not iterable
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/cinder/cinder/volume/manager.py", line 185, in lso_inner1
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher     return lso_inner2(inst, context, snapshot, **kwargs)
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 252, in inner
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/cinder/cinder/volume/manager.py", line 184, in lso_inner2
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher     return f(*_args, **_kwargs)
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/cinder/cinder/volume/manager.py", line 729, in delete_snapshot
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher     snapshot.save()
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 119, in __exit__
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/cinder/cinder/volume/manager.py", line 719, in delete_snapshot
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher     self.driver.delete_snapshot(snapshot)
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/cinder/cinder/volume/drivers/san/hp/hp_lefthand_iscsi.py", line 163, in delete_snapshot
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher     self.proxy.delete_snapshot(snapshot)
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/cinder/cinder/volume/drivers/san/hp/hp_lefthand_rest_proxy.py", line 386, in delete_snapshot
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher     if in_use_msg in ex.get_description():
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher TypeError: argument of type 'NoneType' is not iterable
2015-08-07 08:41:09.777 TRACE oslo_messaging.rpc.dispatcher

The actual error in left hand states: The operation failed: The snapshot 'snapshot-66ec2d8e-4ef2-452c-a4ef-ab9b1b2af00c' cannot be deleted because it is a clone point. When multiple volumes depend on a snapshot, that snapshot is a clone point. To delete the clone point, you must first delete all but one of the volumes that depend on it.

aorourkehp commented 8 years ago

This is a client issue that was introduced in one of our newer patches (python requests support). The patch has been backed out and this issue is no loner relevant. Cinder driver catches the LeftHand exception as such:

except hpexceptions.HTTPServerError as ex:
    in_use_msg = 'cannot be deleted because it is a clone point'
    if in_use_msg in ex.get_description():
        raise exception.SnapshotIsBusy(ex)

This may not be the best practice to search for a specific string within the error description.