Found a problem with rebooting VMs and tracked it down to a key error in the unmap call:
Dec 1 15:41:14 nyx3 SM: [10012] BLKTAP2:<function _deactivate_locked at 0x2294938>: EXCEPTION <class 'XenAPI.Failure'>, ['XENAPI_PLUGIN_FAILURE', 'unmap', 'KeyError', "'sharable'"]
Dec 1 15:41:14 nyx3 SM: [10012] File "/opt/xensource/sm/blktap2.py", line 86, in wrapper
Dec 1 15:41:14 nyx3 SM: [10012] ret = op(self, args)
Dec 1 15:41:14 nyx3 SM: [10012] File "/opt/xensource/sm/blktap2.py", line 1687, in _deactivate_locked
Dec 1 15:41:14 nyx3 SM: [10012] self._detach(sr_uuid, vdi_uuid)
Dec 1 15:41:14 nyx3 SM: [10012] File "/opt/xensource/sm/blktap2.py", line 1744, in _detach
Dec 1 15:41:14 nyx3 SM: [10012] self.target.detach(sr_uuid, vdi_uuid)
Dec 1 15:41:14 nyx3 SM: [10012] File "/opt/xensource/sm/blktap2.py", line 1119, in detach
Dec 1 15:41:14 nyx3 SM: [10012] self.vdi.detach(sr_uuid, vdi_uuid)
Dec 1 15:41:14 nyx3 SM: [10012] File "/opt/xensource/sm/RBDSR.py", line 424, in detach
Dec 1 15:41:14 nyx3 SM: [10012] self._unmap_VHD(vdi_uuid)
Dec 1 15:41:14 nyx3 SM: [10012] File "/opt/xensource/sm/cephutils.py", line 423, in _unmap_VHD
Dec 1 15:41:14 nyx3 SM: [10012] self._call_plugin('unmap',args)
Dec 1 15:41:14 nyx3 SM: [10012] File "/opt/xensource/sm/cephutils.py", line 377, in _call_plugin
Dec 1 15:41:14 nyx3 SM: [10012] if not self.session.xenapi.host.call_plugin(host_ref, "ceph_plugin", op, args):
Dec 1 15:41:14 nyx3 SM: [10012] File "/usr/lib/python2.7/site-packages/XenAPI.py", line 248, in call
Dec 1 15:41:14 nyx3 SM: [10012] return self.send(self.name, args)
Dec 1 15:41:14 nyx3 SM: [10012] File "/usr/lib/python2.7/site-packages/XenAPI.py", line 150, in xenapi_request
Dec 1 15:41:14 nyx3 SM: [10012] result = _parse_result(getattr(self, methodname)(full_params))
Dec 1 15:41:14 nyx3 SM: [10012] File "/usr/lib/python2.7/site-packages/XenAPI.py", line 222, in _parse_result
Dec 1 15:41:14 nyx3 SM: [10012] raise Failure(result['ErrorDescription'])
Dec 1 15:41:14 nyx3 SM: [10012]
Dec 1 15:41:14 nyx3 SM: [10012] Raising exception [46, The VDI is not available [opterr=['XENAPI_PLUGIN_FAILURE', 'unmap', 'KeyError', "'sharable'"]]]
Dec 1 15:41:14 nyx3 SM: [10012] lock: released /var/lock/sm/784d503b-5a4e-45ef-8e1d-28c88bbdb6ae/vdi
Dec 1 15:41:14 nyx3 SM: [10012] generic exception: vdi_deactivate: EXCEPTION <class 'SR.SROSError'>, The VDI is not available [opterr=['XENAPI_PLUGIN_FAILURE', 'unmap', 'KeyError', "'sharable'"]]
Hi Brian,
There was a error in ceph_plugin.py in _unmap function where it try to get sharable attribute in spite it was not passed to _unmap function.
Last commit should fix this error.
Hi there,
Found a problem with rebooting VMs and tracked it down to a key error in the unmap call:
Dec 1 15:41:14 nyx3 SM: [10012] BLKTAP2:<function _deactivate_locked at 0x2294938>: EXCEPTION <class 'XenAPI.Failure'>, ['XENAPI_PLUGIN_FAILURE', 'unmap', 'KeyError', "'sharable'"] Dec 1 15:41:14 nyx3 SM: [10012] File "/opt/xensource/sm/blktap2.py", line 86, in wrapper Dec 1 15:41:14 nyx3 SM: [10012] ret = op(self, args) Dec 1 15:41:14 nyx3 SM: [10012] File "/opt/xensource/sm/blktap2.py", line 1687, in _deactivate_locked Dec 1 15:41:14 nyx3 SM: [10012] self._detach(sr_uuid, vdi_uuid) Dec 1 15:41:14 nyx3 SM: [10012] File "/opt/xensource/sm/blktap2.py", line 1744, in _detach Dec 1 15:41:14 nyx3 SM: [10012] self.target.detach(sr_uuid, vdi_uuid) Dec 1 15:41:14 nyx3 SM: [10012] File "/opt/xensource/sm/blktap2.py", line 1119, in detach Dec 1 15:41:14 nyx3 SM: [10012] self.vdi.detach(sr_uuid, vdi_uuid) Dec 1 15:41:14 nyx3 SM: [10012] File "/opt/xensource/sm/RBDSR.py", line 424, in detach Dec 1 15:41:14 nyx3 SM: [10012] self._unmap_VHD(vdi_uuid) Dec 1 15:41:14 nyx3 SM: [10012] File "/opt/xensource/sm/cephutils.py", line 423, in _unmap_VHD Dec 1 15:41:14 nyx3 SM: [10012] self._call_plugin('unmap',args) Dec 1 15:41:14 nyx3 SM: [10012] File "/opt/xensource/sm/cephutils.py", line 377, in _call_plugin Dec 1 15:41:14 nyx3 SM: [10012] if not self.session.xenapi.host.call_plugin(host_ref, "ceph_plugin", op, args): Dec 1 15:41:14 nyx3 SM: [10012] File "/usr/lib/python2.7/site-packages/XenAPI.py", line 248, in call Dec 1 15:41:14 nyx3 SM: [10012] return self.send(self.name, args) Dec 1 15:41:14 nyx3 SM: [10012] File "/usr/lib/python2.7/site-packages/XenAPI.py", line 150, in xenapi_request Dec 1 15:41:14 nyx3 SM: [10012] result = _parse_result(getattr(self, methodname)(full_params)) Dec 1 15:41:14 nyx3 SM: [10012] File "/usr/lib/python2.7/site-packages/XenAPI.py", line 222, in _parse_result Dec 1 15:41:14 nyx3 SM: [10012] raise Failure(result['ErrorDescription']) Dec 1 15:41:14 nyx3 SM: [10012] Dec 1 15:41:14 nyx3 SM: [10012] Raising exception [46, The VDI is not available [opterr=['XENAPI_PLUGIN_FAILURE', 'unmap', 'KeyError', "'sharable'"]]] Dec 1 15:41:14 nyx3 SM: [10012] lock: released /var/lock/sm/784d503b-5a4e-45ef-8e1d-28c88bbdb6ae/vdi Dec 1 15:41:14 nyx3 SM: [10012] generic exception: vdi_deactivate: EXCEPTION <class 'SR.SROSError'>, The VDI is not available [opterr=['XENAPI_PLUGIN_FAILURE', 'unmap', 'KeyError', "'sharable'"]]
It seems that attribute isn't being passed in:
Dec 1 15:41:14 nyx3 xapi: [debug|nyx3|1082 UNIX /var/lib/xcp/xapi|host.call_plugin R:0eaa9abb212c|audit] Host.call_plugin host = '07ecd211-7e63-44e9-95ea-2fd3da5b5eee (nyx3)'; plugin = 'ceph_plugin'; fn = 'unmap'; args = [ CEPH_USER: client.admin; dev_name: /dev/nbd/RBD_XenStorage-3e115378-9791-4cf4-b973-0c228d1c1f64/VHD-784d503b-5a4e-45ef-8e1d-28c88bbdb6ae; vdi_name: VHD-784d503b-5a4e-45ef-8e1d-28c88bbdb6ae; mode: nbd; NBDS_MAX: 64; vdi_uuid: 784d503b-5a4e-45ef-8e1d-28c88bbdb6ae; CEPH_POOL_NAME: RBD_XenStorage-3e115378-9791-4cf4-b973-0c228d1c1f64 ] Dec 1 15:41:14 nyx3 xapi: [error|nyx3|1082 UNIX /var/lib/xcp/xapi|dispatch:host.call_plugin D:e31d770f82e7|backtrace] host.call_plugin R:0eaa9abb212c failed with exception Server_error(XENAPI_PLUGIN_FAILURE, [ unmap; KeyError; 'sharable' ]) Dec 1 15:41:14 nyx3 xapi: [error|nyx3|1082 UNIX /var/lib/xcp/xapi|dispatch:host.call_plugin D:e31d770f82e7|backtrace] Raised Server_error(XENAPI_PLUGIN_FAILURE, [ unmap; KeyError; 'sharable' ])
I'm able to work around it for now by simply removing the assignment in the unmap function.
Cheers, Brian