Closed mhoffmann75 closed 8 years ago
Interesting enough after rebooting Host XenServer i still cannot start the VM. Even after deleting snapshot. But this time i get error message about MAP_DUPLICATE_KEY:
Aug 1 17:24:45 pns-xen06 SM: [12820] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-ff12160f-ff09-40bb-a874-1366ad907f44'] Aug 1 17:24:45 pns-xen06 SM: [12820] preit SUCCESS Aug 1 17:24:45 pns-xen06 SM: [12820] ['ceph', 'df', '-f', 'json'] Aug 1 17:24:46 pns-xen06 SM: [12820] preit SUCCESS Aug 1 17:24:46 pns-xen06 SM: [12820] vdi_epoch_begin {'sr_uuid': 'ff12160f-ff09-40bb-a874-1366ad907f44', 'subtask_of': 'DummyRef:|eae0510e-1a25-03ad-69c2-076f8ef7899d|VDI.epoch_begin', 'vdi_ref': 'OpaqueRef:f25db949-ec0e-e414-8f87-7fa393c228fe', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': 'bb086701-2e94-4dad-91b0-3e9e0bd56a5d', 'host_ref': 'OpaqueRef:4eac2447-bf4e-c909-560b-ff67a467dd29', 'session_ref': 'OpaqueRef:412a6490-cf5a-ad5d-38b7-9ee293b4fb77', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_epoch_begin', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:6d76056b-1874-c808-a9b9-ee9abaa31513', 'vdi_uuid': 'bb086701-2e94-4dad-91b0-3e9e0bd56a5d'} Aug 1 17:24:46 pns-xen06 SM: [12896] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-ff12160f-ff09-40bb-a874-1366ad907f44'] Aug 1 17:24:46 pns-xen06 SM: [12896] preit SUCCESS Aug 1 17:24:46 pns-xen06 SM: [12896] ['ceph', 'df', '-f', 'json'] Aug 1 17:24:47 pns-xen06 SM: [12896] preit SUCCESS Aug 1 17:24:47 pns-xen06 SM: [12896] vdi_attach {'sr_uuid': 'ff12160f-ff09-40bb-a874-1366ad907f44', 'subtask_of': 'DummyRef:|9553b5a5-5abf-4b71-cfac-92c7612b2cf4|VDI.attach', 'vdi_ref': 'OpaqueRef:f25db949-ec0e-e414-8f87-7fa393c228fe', 'vdi_on_boot': 'persist', 'args': ['true'], 'vdi_location': 'bb086701-2e94-4dad-91b0-3e9e0bd56a5d', 'host_ref': 'OpaqueRef:4eac2447-bf4e-c909-560b-ff67a467dd29', 'session_ref': 'OpaqueRef:4b84018a-2673-2b39-d437-10862c591127', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_attach', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:6d76056b-1874-c808-a9b9-ee9abaa31513', 'vdi_uuid': 'bb086701-2e94-4dad-91b0-3e9e0bd56a5d'} Aug 1 17:24:47 pns-xen06 SM: [12896] lock: opening lock file /var/lock/sm/bb086701-2e94-4dad-91b0-3e9e0bd56a5d/vdi Aug 1 17:24:47 pns-xen06 SM: [12896] result: {'o_direct_reason': 'LICENSE_RESTRICTION', 'params': '/dev/sm/backend/ff12160f-ff09-40bb-a874-1366ad907f44/bb086701-2e94-4dad-91b0-3e9e0bd56a5d', 'o_direct': True, 'xenstore_data': {'scsi/0x12/0x80': 'AIAAEmJiMDg2NzAxLTJlOTQtNGQgIA==', 'scsi/0x12/0x83': 'AIMAMQIBAC1YRU5TUkMgIGJiMDg2NzAxLTJlOTQtNGRhZC05MWIwLTNlOWUwYmQ1NmE1ZCA=', 'vdi-uuid': 'bb086701-2e94-4dad-91b0-3e9e0bd56a5d', 'mem-pool': 'ff12160f-ff09-40bb-a874-1366ad907f44'}} Aug 1 17:24:47 pns-xen06 SM: [12896] lock: closed /var/lock/sm/bb086701-2e94-4dad-91b0-3e9e0bd56a5d/vdi Aug 1 17:24:47 pns-xen06 SM: [12965] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-ff12160f-ff09-40bb-a874-1366ad907f44'] Aug 1 17:24:47 pns-xen06 SM: [12965] preit SUCCESS Aug 1 17:24:47 pns-xen06 SM: [12965] ['ceph', 'df', '-f', 'json'] Aug 1 17:24:47 pns-xen06 SM: [12965] preit SUCCESS Aug 1 17:24:47 pns-xen06 SM: [12965] vdi_activate {'sr_uuid': 'ff12160f-ff09-40bb-a874-1366ad907f44', 'subtask_of': 'DummyRef:|9afed3e7-c9b7-af20-f615-43912b792d1e|VDI.activate', 'vdi_ref': 'OpaqueRef:f25db949-ec0e-e414-8f87-7fa393c228fe', 'vdi_on_boot': 'persist', 'args': ['true'], 'vdi_location': 'bb086701-2e94-4dad-91b0-3e9e0bd56a5d', 'host_ref': 'OpaqueRef:4eac2447-bf4e-c909-560b-ff67a467dd29', 'session_ref': 'OpaqueRef:7fa61944-bf78-90c0-338d-c70f193984a9', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_activate', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:6d76056b-1874-c808-a9b9-ee9abaa31513', 'vdi_uuid': 'bb086701-2e94-4dad-91b0-3e9e0bd56a5d'} Aug 1 17:24:47 pns-xen06 SM: [12965] lock: opening lock file /var/lock/sm/bb086701-2e94-4dad-91b0-3e9e0bd56a5d/vdi Aug 1 17:24:47 pns-xen06 SM: [12965] blktap2.activate Aug 1 17:24:47 pns-xen06 SM: [12965] lock: acquired /var/lock/sm/bb086701-2e94-4dad-91b0-3e9e0bd56a5d/vdi Aug 1 17:24:47 pns-xen06 SM: [12965] Adding tag to: bb086701-2e94-4dad-91b0-3e9e0bd56a5d Aug 1 17:24:47 pns-xen06 SM: [12965] Activate lock succeeded Aug 1 17:24:47 pns-xen06 SM: [12965] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-ff12160f-ff09-40bb-a874-1366ad907f44'] Aug 1 17:24:48 pns-xen06 SM: [12965] preit SUCCESS Aug 1 17:24:48 pns-xen06 SM: [12965] ['ceph', 'df', '-f', 'json'] Aug 1 17:24:48 pns-xen06 SM: [12965] preit SUCCESS Aug 1 17:24:48 pns-xen06 SM: [12965] RBDVDI.attach for bb086701-2e94-4dad-91b0-3e9e0bd56a5d Aug 1 17:24:48 pns-xen06 SM: [12965] Exception in activate/attach Aug 1 17:24:48 pns-xen06 SM: [12965] Removed host key host_OpaqueRef:4eac2447-bf4e-c909-560b-ff67a467dd29 for bb086701-2e94-4dad-91b0-3e9e0bd56a5d Aug 1 17:24:48 pns-xen06 SM: [12965] * BLKTAP2:<function _activate_locked at 0x1967578>: EXCEPTION <class 'XenAPI.Failure'>, ['MAP_DUPLICATE_KEY', 'VDI', 'sm_config', 'OpaqueRef:f25db949-ec0e-e414-8f87-7fa393c228fe', 'attached'] Aug 1 17:24:48 pns-xen06 SM: [12965] File "/opt/xensource/sm/blktap2.py", line 86, in wrapper Aug 1 17:24:48 pns-xen06 SM: [12965] ret = op(self, _args) Aug 1 17:24:48 pns-xen06 SM: [12965] File "/opt/xensource/sm/blktap2.py", line 1593, in _activate_locked Aug 1 17:24:48 pns-xen06 SM: [12965] self._attach(sr_uuid, vdi_uuid) Aug 1 17:24:48 pns-xen06 SM: [12965] File "/opt/xensource/sm/blktap2.py", line 1658, in _attach Aug 1 17:24:48 pns-xen06 SM: [12965] attach_info = xmlrpclib.loads(self.target.attach(sr_uuid, vdi_uuid))[0][0] Aug 1 17:24:48 pns-xen06 SM: [12965] File "/opt/xensource/sm/blktap2.py", line 1115, in attach Aug 1 17:24:48 pns-xen06 SM: [12965] return self.vdi.attach(sr_uuid, vdi_uuid) Aug 1 17:24:48 pns-xen06 SM: [12965] File "/opt/xensource/sm/RBDSR.py", line 329, in attach Aug 1 17:24:48 pns-xen06 SM: [12965] self.session.xenapi.VDI.add_to_sm_config(vdi_ref, 'attached', 'true') Aug 1 17:24:48 pns-xen06 SM: [12965] File "/usr/lib/python2.7/site-packages/XenAPI.py", line 248, in call Aug 1 17:24:48 pns-xen06 SM: [12965] return self.send(self.name, args) Aug 1 17:24:48 pns-xen06 SM: [12965] File "/usr/lib/python2.7/site-packages/XenAPI.py", line 150, in xenapi_request Aug 1 17:24:48 pns-xen06 SM: [12965] result = _parse_result(getattr(self, methodname)(_full_params)) Aug 1 17:24:48 pns-xen06 SM: [12965] File "/usr/lib/python2.7/site-packages/XenAPI.py", line 222, in _parse_result Aug 1 17:24:48 pns-xen06 SM: [12965] raise Failure(result['ErrorDescription']) Aug 1 17:24:48 pns-xen06 SM: [12965] Aug 1 17:24:48 pns-xen06 SM: [12965] Raising exception [46, The VDI is not available [opterr=['MAP_DUPLICATE_KEY', 'VDI', 'sm_config', 'OpaqueRef:f25db949-ec0e-e414-8f87-7fa393c228fe', 'attached']]] Aug 1 17:24:48 pns-xen06 SM: [12965] lock: released /var/lock/sm/bb086701-2e94-4dad-91b0-3e9e0bd56a5d/vdi Aug 1 17:24:48 pns-xen06 SM: [12965] * generic exception: vdi_activate: EXCEPTION <class 'SR.SROSError'>, The VDI is not available [opterr=['MAP_DUPLICATE_KEY', 'VDI', 'sm_config', 'OpaqueRef:f25db949-ec0e-e414-8f87-7fa393c228fe', 'attached']] Aug 1 17:24:48 pns-xen06 SM: [12965] File "/opt/xensource/sm/SRCommand.py", line 110, in run Aug 1 17:24:48 pns-xen06 SM: [12965] return self._run_locked(sr) Aug 1 17:24:48 pns-xen06 SM: [12965] File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked Aug 1 17:24:48 pns-xen06 SM: [12965] rv = self._run(sr, target) Aug 1 17:24:48 pns-xen06 SM: [12965] File "/opt/xensource/sm/SRCommand.py", line 264, in _run Aug 1 17:24:48 pns-xen06 SM: [12965] writable, caching_params) Aug 1 17:24:48 pns-xen06 SM: [12965] File "/opt/xensource/sm/blktap2.py", line 1560, in activate Aug 1 17:24:48 pns-xen06 SM: [12965] if self._activate_locked(sr_uuid, vdi_uuid, options): Aug 1 17:24:48 pns-xen06 SM: [12965] File "/opt/xensource/sm/blktap2.py", line 94, in wrapper Aug 1 17:24:48 pns-xen06 SM: [12965] raise xs_errors.XenError(excType, opterr=msg) Aug 1 17:24:48 pns-xen06 SM: [12965] File "/opt/xensource/sm/xs_errors.py", line 52, in init Aug 1 17:24:48 pns-xen06 SM: [12965] raise SR.SROSError(errorcode, errormessage) Aug 1 17:24:48 pns-xen06 SM: [12965] Aug 1 17:24:48 pns-xen06 SM: [12965] ***\ RBD: EXCEPTION <class 'SR.SROSError'>, The VDI is not available [opterr=['MAP_DUPLICATE_KEY', 'VDI', 'sm_config', 'OpaqueRef:f25db949-ec0e-e414-8f87-7fa393c228fe', 'attached']] Aug 1 17:24:48 pns-xen06 SM: [12965] File "/opt/xensource/sm/SRCommand.py", line 352, in run Aug 1 17:24:48 pns-xen06 SM: [12965] ret = cmd.run(sr) Aug 1 17:24:48 pns-xen06 SM: [12965] File "/opt/xensource/sm/SRCommand.py", line 110, in run Aug 1 17:24:48 pns-xen06 SM: [12965] return self._run_locked(sr) Aug 1 17:24:48 pns-xen06 SM: [12965] File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked Aug 1 17:24:48 pns-xen06 SM: [12965] rv = self._run(sr, target) Aug 1 17:24:48 pns-xen06 SM: [12965] File "/opt/xensource/sm/SRCommand.py", line 264, in _run Aug 1 17:24:48 pns-xen06 SM: [12965] writable, caching_params) Aug 1 17:24:48 pns-xen06 SM: [12965] File "/opt/xensource/sm/blktap2.py", line 1560, in activate Aug 1 17:24:48 pns-xen06 SM: [12965] if self._activate_locked(sr_uuid, vdi_uuid, options): Aug 1 17:24:48 pns-xen06 SM: [12965] File "/opt/xensource/sm/blktap2.py", line 94, in wrapper Aug 1 17:24:48 pns-xen06 SM: [12965] raise xs_errors.XenError(excType, opterr=msg) Aug 1 17:24:48 pns-xen06 SM: [12965] File "/opt/xensource/sm/xs_errors.py", line 52, in init Aug 1 17:24:48 pns-xen06 SM: [12965] raise SR.SROSError(errorcode, errormessage) Aug 1 17:24:48 pns-xen06 SM: [12965] Aug 1 17:24:48 pns-xen06 SM: [12965] lock: closed /var/lock/sm/bb086701-2e94-4dad-91b0-3e9e0bd56a5d/vdi Aug 1 17:24:48 pns-xen06 SM: [13120] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-ff12160f-ff09-40bb-a874-1366ad907f44'] Aug 1 17:24:48 pns-xen06 SM: [13120] preit SUCCESS Aug 1 17:24:48 pns-xen06 SM: [13120] ['ceph', 'df', '-f', 'json'] Aug 1 17:24:49 pns-xen06 SM: [13120] preit SUCCESS Aug 1 17:24:49 pns-xen06 SM: [13120] vdi_detach {'sr_uuid': 'ff12160f-ff09-40bb-a874-1366ad907f44', 'subtask_of': 'DummyRef:|ed5e66fb-4281-3ea8-83bd-2c3088b0dc31|VDI.detach', 'vdi_ref': 'OpaqueRef:f25db949-ec0e-e414-8f87-7fa393c228fe', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': 'bb086701-2e94-4dad-91b0-3e9e0bd56a5d', 'host_ref': 'OpaqueRef:4eac2447-bf4e-c909-560b-ff67a467dd29', 'session_ref': 'OpaqueRef:e8351bd0-d039-92cc-7cee-7b3dfb639242', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_detach', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:6d76056b-1874-c808-a9b9-ee9abaa31513', 'vdi_uuid': 'bb086701-2e94-4dad-91b0-3e9e0bd56a5d'} Aug 1 17:24:49 pns-xen06 SM: [13120] lock: opening lock file /var/lock/sm/bb086701-2e94-4dad-91b0-3e9e0bd56a5d/vdi Aug 1 17:24:49 pns-xen06 SM: [13120] lock: closed /var/lock/sm/bb086701-2e94-4dad-91b0-3e9e0bd56a5d/vdi
Maybe this helps understanding whats going on?
Hi Martin,
You can resolve the issue when you get the message about MAP_DUPLICATE_KEY doing this:
# xe vdi-list name-label=name of vdi here.
name of vdi is usualy something like name of vm 0
.
you need uuid of vdi
after that you can tell xenserver to forget vdi.
# xe vdi-forget uuid= uuid from output of command above
Then in XenCenter click rescan
on your SR
after rescan reattach the vdi to your VM in XenCenter.
As for the crash of the VM I will investigate the problem and let you know.
Hi Martin, Issue with the vm crash has been fixed. It just requires to unmap nbd device before snapshot and remap after.
Please check.
Thanks for the fix! I have installed your latest fix and found snapshot not crashing VM now! Good. However after creating snapshot and deleting it on running VM the VM cannot be rebooted anymore. It again gets stuck. Eventually hanging XenServer Host aswell: Scanning SR takes forever :-(
Also it seems to me that creating a new VM with new VDI refuses to reboot and ends up in the MAP_DUPLICATE_KEY error and adding a second VDI to an existing VM also doesn't work. But both of these issues need more investigation since they might be caused by an already stuck Host. When i can find more time i will try to isolate it further ...
Hi Martin, I have made new fix. Please check.
Seems to work now! Great work. One minor issue that remains is that adding a second disk to a running vm, xencenter tells me that i need to reboot. But rebooting does not make the disk show up. Only powering down the vm and starting it up again. It seems to me that this issue only appears in RBDSR nbd mode, although i'm not 100% sure.
Which OS is on VM? I have added second disk to running VM with Win7. It just required to format disk so that the system could see it I will check it with linux.
I tested it on VM with CentOS 7. It works too.
Is xen-tools installed on VM?
You're right: No Xen Tools installed. So my fault. VM on local storage just behaves the same. So no RBDSR problem :-) Sorry to bother you. Just for completeness: Was a default install of ubuntu 16.04 server without guest tools.
Ok. Thanks for your tests. Can I close the issue? :-)
Yes please :-)
We have evaluated RBDSR in combination with a ceph jewel cluster (ubuntu 14.04) and XenServer 7. From our testing, it seems that every mode of RBDSR has it's pros/cons. Our results: kernel mode +really fast -forces ceph cluster into compatibility mode +most features like VDI creation, snapshot, etc work
fuse mode -quite slow (kernel seems like 5 times faster) +full jewel compatibility +most features like VDI creation, snapshot, etc work
nbd mode +almost (?) as fast as kernel mode +full jewel compatibility -snapshot of a running VM hangs VM and XenServer :-(
Thats why for us the most interesting mode is nbd-Mode. Other ideas welcome :-)
However the crash of the VM you run a snapshot on and the corresponding XenServer Host is a real show stopper. Any idea about that?
The SMLog doesn't seem to reveal any errors:
It seems that the VM cannot access its disk anymore - running OS of VM complains that disk does not response. However VM cannot be shut down and even Host Server (XenServer7) refuses to force stop VM. Hard reboot via console is necessary.