Closed Emmenemoi closed 7 years ago
SR_PROBE is present in caps
Now it supports HA but it's not tested.
Hello! Thanks for adding support for HA. I just cloned master branch and tried creating a HA in my Xenserver pool but unfortunately it did't work. It detected that the SR had support for HA but when completing the guide in XenCenter o got an error:
"Failed","Enabling HA on pool 'Xenserver 7 Pool' This operation cannot be performed because VDI OpaqueRef:0cf4946e-9ac9-f4ec-74c2-4ef1b9023943 could not be properly attached to the VM. Time: 00:00:04","Xenserver 7 Pool","Aug 14, 2016 12:42 AM"
The interesting part of SMlog is here. Aug 14 00:42:36 arb-xen3 SM: [4409] vdi_create {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'DummyRef:|4f4bf0ae-5aeb-d922-a868-d33346bdc16a|VDI.create', 'vdi_type': 'ha_statefile', 'args': ['266240', 'Statefile for HA', 'Used for storage heartbeating', '', 'false', '19700101T00:00:00Z', '', 'false'], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:bda94161-4075-fb8d-ea57-041741125c00', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_create', 'sr_ref': 'OpaqueRef:ca45453a-eeb5-1da1-837c-607133a54114', 'vdi_sm_config': {'type': 'raw'}} Aug 14 00:42:36 arb-xen3 SM: [4409] RBDVDI.create for 1a20d597-4cc2-4da8-8424-d0ae33df4134 Aug 14 00:42:36 arb-xen3 SM: [4409] ['rbd', 'create', 'VHD-1a20d597-4cc2-4da8-8424-d0ae33df4134', '--size', '0', '--object-size', '2097152', '--image-format', '2', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Aug 14 00:42:36 arb-xen3 SM: [4409] pread SUCCESS Aug 14 00:42:36 arb-xen3 SM: [4409] ['rbd', 'image-meta', 'set', 'VHD-1a20d597-4cc2-4da8-8424-d0ae33df4134', 'VDI_LABEL', 'Statefile for HA', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Aug 14 00:42:37 arb-xen3 SM: [4409] pread SUCCESS Aug 14 00:42:37 arb-xen3 SM: [4409] ['rbd', 'image-meta', 'set', 'VHD-1a20d597-4cc2-4da8-8424-d0ae33df4134', 'VDI_DESCRIPTION', 'Used for storage heartbeating', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Aug 14 00:42:37 arb-xen3 SM: [4409] pread SUCCESS Aug 14 00:42:37 arb-xen3 SM: [4530] ['uuidgen', '-r'] Aug 14 00:42:37 arb-xen3 SM: [4530] pread SUCCESS Aug 14 00:42:37 arb-xen3 SM: [4530] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Aug 14 00:42:37 arb-xen3 SM: [4530] pread SUCCESS Aug 14 00:42:37 arb-xen3 SM: [4530] vdi_create {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'DummyRef:|9a6b0752-ce82-048e-92e9-9a5ca9c2944b|VDI.create', 'vdi_type': 'redo_log', 'args': ['268435456', 'Metadata for HA', 'Used for master failover', '', 'false', '19700101T00:00:00Z', '', 'false'], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:c7ddd03e-f779-0aa3-dfd9-852fa730c34b', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_create', 'sr_ref': 'OpaqueRef:ca45453a-eeb5-1da1-837c-607133a54114', 'vdi_sm_config': {'type': 'raw'}} Aug 14 00:42:37 arb-xen3 SM: [4530] RBDVDI.create for c981e941-b2f6-44b2-9369-9fcc166a202c Aug 14 00:42:37 arb-xen3 SM: [4530] ['rbd', 'create', 'VHD-c981e941-b2f6-44b2-9369-9fcc166a202c', '--size', '256', '--object-size', '2097152', '--image-format', '2', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Aug 14 00:42:37 arb-xen3 SM: [4530] pread SUCCESS Aug 14 00:42:37 arb-xen3 SM: [4530] ['rbd', 'image-meta', 'set', 'VHD-c981e941-b2f6-44b2-9369-9fcc166a202c', 'VDI_LABEL', 'Metadata for HA', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Aug 14 00:42:38 arb-xen3 SM: [4530] pread SUCCESS Aug 14 00:42:38 arb-xen3 SM: [4530] ['rbd', 'image-meta', 'set', 'VHD-c981e941-b2f6-44b2-9369-9fcc166a202c', 'VDI_DESCRIPTION', 'Used for master failover', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Aug 14 00:42:38 arb-xen3 SM: [4530] pread SUCCESS Aug 14 00:42:38 arb-xen3 SM: [4672] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Aug 14 00:42:39 arb-xen3 SM: [4672] pread SUCCESS Aug 14 00:42:39 arb-xen3 SM: [4672] vdi_generate_config {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:bcdc23a7-ee2f-b781-ed21-ea3c508c1068', 'vdi_ref': 'OpaqueRef:0cf4946e-9ac9-f4ec-74c2-4ef1b9023943', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '1a20d597-4cc2-4da8-8424-d0ae33df4134', 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:1ee8c8d6-1ce1-f449-9314-886f5ef2ca6f', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_generate_config', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:ca45453a-eeb5-1da1-837c-607133a54114', 'vdi_uuid': '1a20d597-4cc2-4da8-8424-d0ae33df4134'} Aug 14 00:42:39 arb-xen3 SM: [4672] RBDVDI.generate_config Aug 14 00:42:39 arb-xen3 SM: [4672] * generic exception: vdi_generate_config: EXCEPTION <type 'exceptions.TypeError'>, coercing to Unicode: need string or buffer, NoneType found Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/SRCommand.py", line 110, in run Aug 14 00:42:39 arb-xen3 SM: [4672] return self._run_locked(sr) Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked Aug 14 00:42:39 arb-xen3 SM: [4672] rv = self._run(sr, target) Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/SRCommand.py", line 282, in _run Aug 14 00:42:39 arb-xen3 SM: [4672] return target.generate_config(self.params['sr_uuid'], self.vdi_uuid) Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/RBDSR", line 654, in generate_config Aug 14 00:42:39 arb-xen3 SM: [4672] if not util.pathexists(self.path): Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/util.py", line 400, in pathexists Aug 14 00:42:39 arb-xen3 SM: [4672] os.stat(path) Aug 14 00:42:39 arb-xen3 SM: [4672] Aug 14 00:42:39 arb-xen3 SM: [4672] * RBD: EXCEPTION <type 'exceptions.TypeError'>, coercing to Unicode: need string or buffer, NoneType found Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/SRCommand.py", line 352, in run Aug 14 00:42:39 arb-xen3 SM: [4672] ret = cmd.run(sr) Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/SRCommand.py", line 110, in run Aug 14 00:42:39 arb-xen3 SM: [4672] return self._run_locked(sr) Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked Aug 14 00:42:39 arb-xen3 SM: [4672] rv = self._run(sr, target) Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/SRCommand.py", line 282, in _run Aug 14 00:42:39 arb-xen3 SM: [4672] return target.generate_config(self.params['sr_uuid'], self.vdi_uuid) Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/RBDSR", line 654, in generate_config Aug 14 00:42:39 arb-xen3 SM: [4672] if not util.pathexists(self.path): Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/util.py", line 400, in pathexists Aug 14 00:42:39 arb-xen3 SM: [4672] os.stat(path) Aug 14 00:42:39 arb-xen3 SM: [4672]
If any more information is needed please just tell me what and i will try to extract it.
Hi, Fixed. Please check.
Thank you for a really quick fix. Unfortunately the same kind of error in XenCenter when activating ha.
"Failed","Enabling HA on pool 'Xenserver 7 Pool' This operation cannot be performed because VDI OpaqueRef:524feed2-f5e4-ce90-dacb-84b807139caa could not be properly attached to the VM. Time: 00:00:05","Xenserver 7 Pool","Aug 14, 2016 10:29 AM"
This is the SMlog, a new type of error there.
Aug 14 10:29:23 arb-xen3 SM: [8581] sr_scan {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|6a2105c1-68f1-d82b-5e9d-22355fb36b35|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:f0ba5026-762e-d8d5-8bf2-c61297294664', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'}
Aug 14 10:29:23 arb-xen3 SM: [8596] sr_update {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|69dc7dd0-5f3a-99a6-6ca4-6503d6098556|SR.stat', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:28c7cade-a113-9c3c-988a-51de34d47241', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_update', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'}
Aug 14 10:29:33 arb-xen3 SM: [8648] ['uuidgen', '-r']
Aug 14 10:29:33 arb-xen3 SM: [8648] pread SUCCESS
Aug 14 10:29:33 arb-xen3 SM: [8648] ['ceph', 'df', '--format', 'json', '--name', 'client.admin']
Aug 14 10:29:33 arb-xen3 SM: [8648] pread SUCCESS
Aug 14 10:29:33 arb-xen3 SM: [8648] vdi_create {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'DummyRef:|1de2660f-67d4-9511-36d8-91a1f7512429|VDI.create', 'vdi_type': 'ha_statefile', 'args': ['266240', 'Statefile for HA', 'Used for storage heartbeating', '', 'false', '19700101T00:00:00Z', '', 'false'], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:ebcd9c4d-6525-0b91-ffe7-54c11af79797', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_create', 'sr_ref': 'OpaqueRef:ca45453a-eeb5-1da1-837c-607133a54114', 'vdi_sm_config': {'type': 'raw'}}
Aug 14 10:29:33 arb-xen3 SM: [8648] RBDVDI.create for fc8df789-83c0-4ad7-b128-3e88e12b3c53
Aug 14 10:29:33 arb-xen3 SM: [8648] ['rbd', 'create', 'VHD-fc8df789-83c0-4ad7-b128-3e88e12b3c53', '--size', '0', '--object-size', '2097152', '--image-format', '2', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Aug 14 10:29:34 arb-xen3 SM: [8648] pread SUCCESS
Aug 14 10:29:34 arb-xen3 SM: [8648] ['rbd', 'image-meta', 'set', 'VHD-fc8df789-83c0-4ad7-b128-3e88e12b3c53', 'VDI_LABEL', 'Statefile for HA', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Aug 14 10:29:34 arb-xen3 SM: [8648] pread SUCCESS
Aug 14 10:29:34 arb-xen3 SM: [8648] ['rbd', 'image-meta', 'set', 'VHD-fc8df789-83c0-4ad7-b128-3e88e12b3c53', 'VDI_DESCRIPTION', 'Used for storage heartbeating', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Aug 14 10:29:34 arb-xen3 SM: [8648] pread SUCCESS
Aug 14 10:29:34 arb-xen3 SM: [8783] ['uuidgen', '-r']
Aug 14 10:29:34 arb-xen3 SM: [8783] pread SUCCESS
Aug 14 10:29:34 arb-xen3 SM: [8783] ['ceph', 'df', '--format', 'json', '--name', 'client.admin']
Aug 14 10:29:34 arb-xen3 SM: [8783] pread SUCCESS
Aug 14 10:29:34 arb-xen3 SM: [8783] vdi_create {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'DummyRef:|51afc66f-0528-a4be-be5b-220edddd3402|VDI.create', 'vdi_type': 'redo_log', 'args': ['268435456', 'Metadata for HA', 'Used for master failover', '', 'false', '19700101T00:00:00Z', '', 'false'], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:8e01113f-b68c-cfc8-ef7f-4d806e15b602', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_create', 'sr_ref': 'OpaqueRef:ca45453a-eeb5-1da1-837c-607133a54114', 'vdi_sm_config': {'type': 'raw'}}
Aug 14 10:29:34 arb-xen3 SM: [8783] RBDVDI.create for aa386559-0f70-474a-8e84-1e309f6f2156
Aug 14 10:29:34 arb-xen3 SM: [8783] ['rbd', 'create', 'VHD-aa386559-0f70-474a-8e84-1e309f6f2156', '--size', '256', '--object-size', '2097152', '--image-format', '2', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Aug 14 10:29:35 arb-xen3 SM: [8783] pread SUCCESS
Aug 14 10:29:35 arb-xen3 SM: [8783] ['rbd', 'image-meta', 'set', 'VHD-aa386559-0f70-474a-8e84-1e309f6f2156', 'VDI_LABEL', 'Metadata for HA', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Aug 14 10:29:35 arb-xen3 SM: [8783] pread SUCCESS
Aug 14 10:29:35 arb-xen3 SM: [8783] ['rbd', 'image-meta', 'set', 'VHD-aa386559-0f70-474a-8e84-1e309f6f2156', 'VDI_DESCRIPTION', 'Used for master failover', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Aug 14 10:29:35 arb-xen3 SM: [8783] pread SUCCESS
Aug 14 10:29:35 arb-xen3 SM: [8921] ['ceph', 'df', '--format', 'json', '--name', 'client.admin']
Aug 14 10:29:36 arb-xen3 SM: [8921] pread SUCCESS
Aug 14 10:29:36 arb-xen3 SM: [8921] vdi_generate_config {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:27f868d0-e195-2f92-7dea-5294e6d86180', 'vdi_ref': 'OpaqueRef:524feed2-f5e4-ce90-dacb-84b807139caa', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': 'fc8df789-83c0-4ad7-b128-3e88e12b3c53', 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:31298192-e6d0-a3a1-9cca-985307626720', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_generate_config', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:ca45453a-eeb5-1da1-837c-607133a54114', 'vdi_uuid': 'fc8df789-83c0-4ad7-b128-3e88e12b3c53'}
Aug 14 10:29:36 arb-xen3 SM: [8921] RBDVDI.generate_config
Aug 14 10:29:36 arb-xen3 SM: [8952] Failed to parse commandline; exception = 'device_config' argv = ['/opt/xensource/sm/RBDSR', "<?xml version='1.0'?>\n
One more fix
Hello again!
Great! Now the guide to enable HA completes without any errors. Also if any protected VM is shutdown it's restarted automatically. So far so good. What i'm not able to test is is to kill one Xenserver and make sure protected VM:s on the killed Xenserver is booted on another one in the pool. This is because i only have one server in my Xenserver 7 Pool and i'm on holidays this week. I will test this as soon as i get back so i can install another server.
Hi there!
Having problems with HA via ceph too. 3 nodes, no VMs, trying to enable HA. Got this in SMlog:
Sep 19 14:51:11 alahd-xen2 SM: [22201] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Sep 19 14:51:12 alahd-xen2 SM: [22201] pread SUCCESS Sep 19 14:51:12 alahd-xen2 SM: [22201] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Sep 19 14:51:12 alahd-xen2 SM: [22201] pread SUCCESS Sep 19 14:51:12 alahd-xen2 SM: [22201] vdi_generate_config {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:f29e6641-87fa-d985-9268-107822105265', 'vdi_ref': 'OpaqueRef:26516ae1-cae0-4319-ccb1-aaff2a2af419', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '78edbe81-3530-4cc7-8834-dca1c385f99f', 'host_ref': 'OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01', 'session_ref': 'OpaqueRef:d0e98f1a-3e25-f983-af2d-9f5e907b3234', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_generate_config', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:224a0edd-4c3e-4e0d-4091-d3fd8b9c6ad9', 'vdi_uuid': '78edbe81-3530-4cc7-8834-dca1c385f99f'} Sep 19 14:51:12 alahd-xen2 SM: [22201] RBDVDI.generate_config Sep 19 14:51:12 alahd-xen2 SM: [22254] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Sep 19 14:51:12 alahd-xen2 SM: [22254] pread SUCCESS Sep 19 14:51:12 alahd-xen2 SM: [22254] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Sep 19 14:51:12 alahd-xen2 SM: [22254] pread SUCCESS Sep 19 14:51:12 alahd-xen2 SM: [22254] vdi_attach_from_config {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_attach_from_config', 'vdi_uuid': '78edbe81-3530-4cc7-8834-dca1c385f99f'} Sep 19 14:51:12 alahd-xen2 SM: [22254] RBDVDI.attach_from_config Sep 19 14:51:12 alahd-xen2 SM: [22254] RBDSR.attach for 4ceb0f8a-1539-40a4-bee2-450a025b04e1 Sep 19 14:51:12 alahd-xen2 SM: [22254] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Sep 19 14:51:13 alahd-xen2 SM: [22254] pread SUCCESS Sep 19 14:51:13 alahd-xen2 SM: [22254] ['mkdir', '-p', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1'] Sep 19 14:51:13 alahd-xen2 SM: [22254] pread SUCCESS Sep 19 14:51:13 alahd-xen2 SM: [22254] ['rbd-nbd', '--nbds_max', '64', 'map', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-78edbe81-3530-4cc7-8834-dca1c385f99f', '--name', 'client.admin'] Sep 19 14:51:13 alahd-xen2 SM: [22254] pread SUCCESS Sep 19 14:51:13 alahd-xen2 SM: [22254] ['ln', '-s', '/dev/nbd0', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-78edbe81-3530-4cc7-8834-dca1c385f99f'] Sep 19 14:51:13 alahd-xen2 SM: [22254] pread SUCCESS Sep 19 14:51:13 alahd-xen2 SM: [22379] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Sep 19 14:51:14 alahd-xen2 SM: [22379] pread SUCCESS Sep 19 14:51:14 alahd-xen2 SM: [22379] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Sep 19 14:51:14 alahd-xen2 SM: [22379] pread SUCCESS Sep 19 14:51:14 alahd-xen2 SM: [22379] vdi_generate_config {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:eb2dcbc5-786f-c551-b8d8-b24f7d12c684', 'vdi_ref': 'OpaqueRef:3f3a5442-2870-755e-df3f-be4db94df5a5', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1', 'host_ref': 'OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01', 'session_ref': 'OpaqueRef:c6d7a2e4-6d1c-a156-299e-52c363f93cdc', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_generate_config', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:224a0edd-4c3e-4e0d-4091-d3fd8b9c6ad9', 'vdi_uuid': '3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1'} Sep 19 14:51:14 alahd-xen2 SM: [22379] RBDVDI.generate_config Sep 19 14:51:14 alahd-xen2 SM: [22432] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Sep 19 14:51:14 alahd-xen2 SM: [22432] pread SUCCESS Sep 19 14:51:14 alahd-xen2 SM: [22432] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Sep 19 14:51:14 alahd-xen2 SM: [22432] pread SUCCESS Sep 19 14:51:14 alahd-xen2 SM: [22432] vdi_attach_from_config {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_attach_from_config', 'vdi_uuid': '3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1'} Sep 19 14:51:14 alahd-xen2 SM: [22432] RBDVDI.attach_from_config Sep 19 14:51:14 alahd-xen2 SM: [22432] RBDSR.attach for 4ceb0f8a-1539-40a4-bee2-450a025b04e1 Sep 19 14:51:14 alahd-xen2 SM: [22432] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Sep 19 14:51:15 alahd-xen2 SM: [22432] pread SUCCESS Sep 19 14:51:15 alahd-xen2 SM: [22432] ['mkdir', '-p', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1'] Sep 19 14:51:15 alahd-xen2 SM: [22432] pread SUCCESS Sep 19 14:51:15 alahd-xen2 SM: [22432] ['rbd-nbd', '--nbds_max', '64', 'map', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1', '--name', 'client.admin'] Sep 19 14:51:15 alahd-xen2 SM: [22432] pread SUCCESS Sep 19 14:51:15 alahd-xen2 SM: [22432] ['ln', '-s', '/dev/nbd1', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1'] Sep 19 14:51:15 alahd-xen2 SM: [22432] pread SUCCESS
Then freeze for ~2 minutes, in xensource.log got:
Sep 19 14:53:19 alahd-xen2 xapi: [error|alahd-xen2|579 INET :::80|host.ha_join_liveset R:194ce241ab08|xapi] HA enable failed with BOOTJOIN_TIMEOUT
And then in SMlog:
Sep 19 14:53:20 alahd-xen2 SM: [23049] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Sep 19 14:53:20 alahd-xen2 SM: [23049] pread SUCCESS Sep 19 14:53:20 alahd-xen2 SM: [23049] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Sep 19 14:53:20 alahd-xen2 SM: [23049] pread SUCCESS Sep 19 14:53:20 alahd-xen2 SM: [23049] vdi_deactivate {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:e3037904-79a3-c69a-e038-da1b5dd16e71', 'vdi_ref': 'OpaqueRef:26516ae1-cae0-4319-ccb1-aaff2a2af419', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '78edbe81-3530-4cc7-8834-dca1c385f99f', 'host_ref': 'OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01', 'session_ref': 'OpaqueRef:eb43896d-b93f-8308-8d17-2d98d3147028', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_deactivate', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:224a0edd-4c3e-4e0d-4091-d3fd8b9c6ad9', 'vdi_uuid': '78edbe81-3530-4cc7-8834-dca1c385f99f'} Sep 19 14:53:20 alahd-xen2 SM: [23049] lock: opening lock file /var/lock/sm/78edbe81-3530-4cc7-8834-dca1c385f99f/vdi Sep 19 14:53:20 alahd-xen2 SM: [23049] blktap2.deactivate Sep 19 14:53:20 alahd-xen2 SM: [23049] lock: acquired /var/lock/sm/78edbe81-3530-4cc7-8834-dca1c385f99f/vdi Sep 19 14:53:20 alahd-xen2 SM: [23049] Backend path /dev/sm/backend/4ceb0f8a-1539-40a4-bee2-450a025b04e1/78edbe81-3530-4cc7-8834-dca1c385f99f does not exist Sep 19 14:53:20 alahd-xen2 SM: [23049] Calling _unmap_VHD Sep 19 14:53:20 alahd-xen2 SM: [23049] Calling ceph_plugin Sep 19 14:53:20 alahd-xen2 SM: [23049] Calling rbd/nbd map on localhost OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01 Sep 19 14:53:20 alahd-xen2 SM: [23114] ['realpath', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-78edbe81-3530-4cc7-8834-dca1c385f99f'] Sep 19 14:53:20 alahd-xen2 SM: [23114] pread SUCCESS Sep 19 14:53:20 alahd-xen2 SM: [23114] ['unlink', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-78edbe81-3530-4cc7-8834-dca1c385f99f'] Sep 19 14:53:20 alahd-xen2 SM: [23114] pread SUCCESS Sep 19 14:53:20 alahd-xen2 SM: [23114] ['rbd-nbd', 'unmap', '/dev/nbd0', '--name', 'client.admin'] Sep 19 14:53:20 alahd-xen2 SM: [23114] pread SUCCESS Sep 19 14:53:20 alahd-xen2 SM: [23049] WARNING: host key host_OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01 not found! Sep 19 14:53:20 alahd-xen2 SM: [23049] lock: released /var/lock/sm/78edbe81-3530-4cc7-8834-dca1c385f99f/vdi Sep 19 14:53:20 alahd-xen2 SM: [23049] lock: closed /var/lock/sm/78edbe81-3530-4cc7-8834-dca1c385f99f/vdi Sep 19 14:53:20 alahd-xen2 SM: [23126] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Sep 19 14:53:21 alahd-xen2 SM: [23126] pread SUCCESS Sep 19 14:53:21 alahd-xen2 SM: [23126] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Sep 19 14:53:21 alahd-xen2 SM: [23126] pread SUCCESS Sep 19 14:53:21 alahd-xen2 SM: [23126] vdi_detach {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:e3037904-79a3-c69a-e038-da1b5dd16e71', 'vdi_ref': 'OpaqueRef:26516ae1-cae0-4319-ccb1-aaff2a2af419', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '78edbe81-3530-4cc7-8834-dca1c385f99f', 'host_ref': 'OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01', 'session_ref': 'OpaqueRef:1001a030-6a52-fcb6-95af-fbafb348d29f', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_detach', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:224a0edd-4c3e-4e0d-4091-d3fd8b9c6ad9', 'vdi_uuid': '78edbe81-3530-4cc7-8834-dca1c385f99f'} Sep 19 14:53:21 alahd-xen2 SM: [23126] lock: opening lock file /var/lock/sm/78edbe81-3530-4cc7-8834-dca1c385f99f/vdi Sep 19 14:53:21 alahd-xen2 SM: [23126] lock: closed /var/lock/sm/78edbe81-3530-4cc7-8834-dca1c385f99f/vdi Sep 19 14:53:21 alahd-xen2 SM: [23182] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Sep 19 14:53:21 alahd-xen2 SM: [23182] pread SUCCESS Sep 19 14:53:21 alahd-xen2 SM: [23182] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Sep 19 14:53:21 alahd-xen2 SM: [23182] pread SUCCESS Sep 19 14:53:21 alahd-xen2 SM: [23182] vdi_deactivate {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:e3037904-79a3-c69a-e038-da1b5dd16e71', 'vdi_ref': 'OpaqueRef:3f3a5442-2870-755e-df3f-be4db94df5a5', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1', 'host_ref': 'OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01', 'session_ref': 'OpaqueRef:8690e36e-1b69-5633-7fa3-cfb87b3f3e07', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_deactivate', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:224a0edd-4c3e-4e0d-4091-d3fd8b9c6ad9', 'vdi_uuid': '3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1'} Sep 19 14:53:21 alahd-xen2 SM: [23182] lock: opening lock file /var/lock/sm/3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1/vdi Sep 19 14:53:21 alahd-xen2 SM: [23182] blktap2.deactivate Sep 19 14:53:21 alahd-xen2 SM: [23182] lock: acquired /var/lock/sm/3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1/vdi Sep 19 14:53:21 alahd-xen2 SM: [23182] Backend path /dev/sm/backend/4ceb0f8a-1539-40a4-bee2-450a025b04e1/3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1 does not exist Sep 19 14:53:21 alahd-xen2 SM: [23182] Calling _unmap_VHD Sep 19 14:53:21 alahd-xen2 SM: [23182] Calling ceph_plugin Sep 19 14:53:21 alahd-xen2 SM: [23182] Calling rbd/nbd map on localhost OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01 Sep 19 14:53:22 alahd-xen2 SM: [23240] ['realpath', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1'] Sep 19 14:53:22 alahd-xen2 SM: [23240] pread SUCCESS Sep 19 14:53:22 alahd-xen2 SM: [23240] ['unlink', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1'] Sep 19 14:53:22 alahd-xen2 SM: [23240] pread SUCCESS Sep 19 14:53:22 alahd-xen2 SM: [23240] ['rbd-nbd', 'unmap', '/dev/nbd1', '--name', 'client.admin'] Sep 19 14:53:22 alahd-xen2 SM: [23240] pread SUCCESS Sep 19 14:53:22 alahd-xen2 SM: [23182] WARNING: host key host_OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01 not found! Sep 19 14:53:22 alahd-xen2 SM: [23182] lock: released /var/lock/sm/3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1/vdi Sep 19 14:53:22 alahd-xen2 SM: [23182] lock: closed /var/lock/sm/3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1/vdi Sep 19 14:53:22 alahd-xen2 SM: [23253] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Sep 19 14:53:22 alahd-xen2 SM: [23253] pread SUCCESS Sep 19 14:53:22 alahd-xen2 SM: [23253] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Sep 19 14:53:22 alahd-xen2 SM: [23253] pread SUCCESS Sep 19 14:53:22 alahd-xen2 SM: [23253] vdi_detach {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:e3037904-79a3-c69a-e038-da1b5dd16e71', 'vdi_ref': 'OpaqueRef:3f3a5442-2870-755e-df3f-be4db94df5a5', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1', 'host_ref': 'OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01', 'session_ref': 'OpaqueRef:411c2241-37e0-d09d-f14d-472f51b5fe18', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_detach', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:224a0edd-4c3e-4e0d-4091-d3fd8b9c6ad9', 'vdi_uuid': '3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1'} Sep 19 14:53:22 alahd-xen2 SM: [23253] lock: opening lock file /var/lock/sm/3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1/vdi Sep 19 14:53:22 alahd-xen2 SM: [23253] lock: closed /var/lock/sm/3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1/vdi
In dmesg got:
[ 9163.277373] block nbd0: NBD_DISCONNECT [ 9163.277484] block nbd0: Receive control failed (result -32) [ 9163.277534] block nbd0: queue cleared [ 9164.664371] block nbd1: NBD_DISCONNECT [ 9164.664455] block nbd1: Receive control failed (result -32) [ 9164.664504] block nbd1: queue cleared
Also tried to use kernel and fuse modes, but failed with different errors. Tried to deploy few VMs with ceph - all works well, including live migration.
Hi, By default in Jewel release and higher the librbd caching is enabled by default. This is was the reason why HA daemon couldn't start. The problem can be addressed by disabling librbd caching by adding to ceph.conf
[client]
rbd cache = false
But it's not good idea to disable caching completely. The disabling librbd caching makes sense only for shared VDIs or for HA VDIs So in the last fix the modified ceph.conf is made and rbd-nbd is invoked with this modified ceph.conf only for shared and HA VDIs. For all other VDIs rbd-nbd is invoked as usual.
Don't know why.
If someone has a clue (SR_PROBE missing in caps?) I could work on implementation.