search

rposudnevskiy / RBDSR

RBDSR - XenServer/XCP-ng Storage Manager plugin for CEPH
GNU Lesser General Public License v2.1
58 stars 23 forks source link

Can't use RBDSR as heartbeat-sr-uuid #13

Closed Emmenemoi closed 7 years ago

Emmenemoi commented 8 years ago

Don't know why.

If someone has a clue (SR_PROBE missing in caps?) I could work on implementation.

rposudnevskiy commented 8 years ago

SR_PROBE is present in caps

rposudnevskiy commented 8 years ago

Now it supports HA but it's not tested.

djmoonshine commented 8 years ago

Hello! Thanks for adding support for HA. I just cloned master branch and tried creating a HA in my Xenserver pool but unfortunately it did't work. It detected that the SR had support for HA but when completing the guide in XenCenter o got an error: "Failed","Enabling HA on pool 'Xenserver 7 Pool' This operation cannot be performed because VDI OpaqueRef:0cf4946e-9ac9-f4ec-74c2-4ef1b9023943 could not be properly attached to the VM. Time: 00:00:04","Xenserver 7 Pool","Aug 14, 2016 12:42 AM"

The interesting part of SMlog is here. Aug 14 00:42:36 arb-xen3 SM: [4409] vdi_create {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'DummyRef:|4f4bf0ae-5aeb-d922-a868-d33346bdc16a|VDI.create', 'vdi_type': 'ha_statefile', 'args': ['266240', 'Statefile for HA', 'Used for storage heartbeating', '', 'false', '19700101T00:00:00Z', '', 'false'], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:bda94161-4075-fb8d-ea57-041741125c00', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_create', 'sr_ref': 'OpaqueRef:ca45453a-eeb5-1da1-837c-607133a54114', 'vdi_sm_config': {'type': 'raw'}} Aug 14 00:42:36 arb-xen3 SM: [4409] RBDVDI.create for 1a20d597-4cc2-4da8-8424-d0ae33df4134 Aug 14 00:42:36 arb-xen3 SM: [4409] ['rbd', 'create', 'VHD-1a20d597-4cc2-4da8-8424-d0ae33df4134', '--size', '0', '--object-size', '2097152', '--image-format', '2', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Aug 14 00:42:36 arb-xen3 SM: [4409] pread SUCCESS Aug 14 00:42:36 arb-xen3 SM: [4409] ['rbd', 'image-meta', 'set', 'VHD-1a20d597-4cc2-4da8-8424-d0ae33df4134', 'VDI_LABEL', 'Statefile for HA', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Aug 14 00:42:37 arb-xen3 SM: [4409] pread SUCCESS Aug 14 00:42:37 arb-xen3 SM: [4409] ['rbd', 'image-meta', 'set', 'VHD-1a20d597-4cc2-4da8-8424-d0ae33df4134', 'VDI_DESCRIPTION', 'Used for storage heartbeating', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Aug 14 00:42:37 arb-xen3 SM: [4409] pread SUCCESS Aug 14 00:42:37 arb-xen3 SM: [4530] ['uuidgen', '-r'] Aug 14 00:42:37 arb-xen3 SM: [4530] pread SUCCESS Aug 14 00:42:37 arb-xen3 SM: [4530] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Aug 14 00:42:37 arb-xen3 SM: [4530] pread SUCCESS Aug 14 00:42:37 arb-xen3 SM: [4530] vdi_create {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'DummyRef:|9a6b0752-ce82-048e-92e9-9a5ca9c2944b|VDI.create', 'vdi_type': 'redo_log', 'args': ['268435456', 'Metadata for HA', 'Used for master failover', '', 'false', '19700101T00:00:00Z', '', 'false'], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:c7ddd03e-f779-0aa3-dfd9-852fa730c34b', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_create', 'sr_ref': 'OpaqueRef:ca45453a-eeb5-1da1-837c-607133a54114', 'vdi_sm_config': {'type': 'raw'}} Aug 14 00:42:37 arb-xen3 SM: [4530] RBDVDI.create for c981e941-b2f6-44b2-9369-9fcc166a202c Aug 14 00:42:37 arb-xen3 SM: [4530] ['rbd', 'create', 'VHD-c981e941-b2f6-44b2-9369-9fcc166a202c', '--size', '256', '--object-size', '2097152', '--image-format', '2', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Aug 14 00:42:37 arb-xen3 SM: [4530] pread SUCCESS Aug 14 00:42:37 arb-xen3 SM: [4530] ['rbd', 'image-meta', 'set', 'VHD-c981e941-b2f6-44b2-9369-9fcc166a202c', 'VDI_LABEL', 'Metadata for HA', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Aug 14 00:42:38 arb-xen3 SM: [4530] pread SUCCESS Aug 14 00:42:38 arb-xen3 SM: [4530] ['rbd', 'image-meta', 'set', 'VHD-c981e941-b2f6-44b2-9369-9fcc166a202c', 'VDI_DESCRIPTION', 'Used for master failover', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Aug 14 00:42:38 arb-xen3 SM: [4530] pread SUCCESS Aug 14 00:42:38 arb-xen3 SM: [4672] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Aug 14 00:42:39 arb-xen3 SM: [4672] pread SUCCESS Aug 14 00:42:39 arb-xen3 SM: [4672] vdi_generate_config {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:bcdc23a7-ee2f-b781-ed21-ea3c508c1068', 'vdi_ref': 'OpaqueRef:0cf4946e-9ac9-f4ec-74c2-4ef1b9023943', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '1a20d597-4cc2-4da8-8424-d0ae33df4134', 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:1ee8c8d6-1ce1-f449-9314-886f5ef2ca6f', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_generate_config', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:ca45453a-eeb5-1da1-837c-607133a54114', 'vdi_uuid': '1a20d597-4cc2-4da8-8424-d0ae33df4134'} Aug 14 00:42:39 arb-xen3 SM: [4672] RBDVDI.generate_config Aug 14 00:42:39 arb-xen3 SM: [4672] * generic exception: vdi_generate_config: EXCEPTION <type 'exceptions.TypeError'>, coercing to Unicode: need string or buffer, NoneType found Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/SRCommand.py", line 110, in run Aug 14 00:42:39 arb-xen3 SM: [4672] return self._run_locked(sr) Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked Aug 14 00:42:39 arb-xen3 SM: [4672] rv = self._run(sr, target) Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/SRCommand.py", line 282, in _run Aug 14 00:42:39 arb-xen3 SM: [4672] return target.generate_config(self.params['sr_uuid'], self.vdi_uuid) Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/RBDSR", line 654, in generate_config Aug 14 00:42:39 arb-xen3 SM: [4672] if not util.pathexists(self.path): Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/util.py", line 400, in pathexists Aug 14 00:42:39 arb-xen3 SM: [4672] os.stat(path) Aug 14 00:42:39 arb-xen3 SM: [4672] Aug 14 00:42:39 arb-xen3 SM: [4672] * RBD: EXCEPTION <type 'exceptions.TypeError'>, coercing to Unicode: need string or buffer, NoneType found Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/SRCommand.py", line 352, in run Aug 14 00:42:39 arb-xen3 SM: [4672] ret = cmd.run(sr) Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/SRCommand.py", line 110, in run Aug 14 00:42:39 arb-xen3 SM: [4672] return self._run_locked(sr) Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked Aug 14 00:42:39 arb-xen3 SM: [4672] rv = self._run(sr, target) Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/SRCommand.py", line 282, in _run Aug 14 00:42:39 arb-xen3 SM: [4672] return target.generate_config(self.params['sr_uuid'], self.vdi_uuid) Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/RBDSR", line 654, in generate_config Aug 14 00:42:39 arb-xen3 SM: [4672] if not util.pathexists(self.path): Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/util.py", line 400, in pathexists Aug 14 00:42:39 arb-xen3 SM: [4672] os.stat(path) Aug 14 00:42:39 arb-xen3 SM: [4672]

If any more information is needed please just tell me what and i will try to extract it.

rposudnevskiy commented 8 years ago

Hi, Fixed. Please check.

djmoonshine commented 8 years ago

Thank you for a really quick fix. Unfortunately the same kind of error in XenCenter when activating ha.

"Failed","Enabling HA on pool 'Xenserver 7 Pool' This operation cannot be performed because VDI OpaqueRef:524feed2-f5e4-ce90-dacb-84b807139caa could not be properly attached to the VM. Time: 00:00:05","Xenserver 7 Pool","Aug 14, 2016 10:29 AM"

This is the SMlog, a new type of error there.

Aug 14 10:29:23 arb-xen3 SM: [8581] sr_scan {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|6a2105c1-68f1-d82b-5e9d-22355fb36b35|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:f0ba5026-762e-d8d5-8bf2-c61297294664', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'} Aug 14 10:29:23 arb-xen3 SM: [8596] sr_update {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|69dc7dd0-5f3a-99a6-6ca4-6503d6098556|SR.stat', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:28c7cade-a113-9c3c-988a-51de34d47241', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_update', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'} Aug 14 10:29:33 arb-xen3 SM: [8648] ['uuidgen', '-r'] Aug 14 10:29:33 arb-xen3 SM: [8648] pread SUCCESS Aug 14 10:29:33 arb-xen3 SM: [8648] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Aug 14 10:29:33 arb-xen3 SM: [8648] pread SUCCESS Aug 14 10:29:33 arb-xen3 SM: [8648] vdi_create {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'DummyRef:|1de2660f-67d4-9511-36d8-91a1f7512429|VDI.create', 'vdi_type': 'ha_statefile', 'args': ['266240', 'Statefile for HA', 'Used for storage heartbeating', '', 'false', '19700101T00:00:00Z', '', 'false'], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:ebcd9c4d-6525-0b91-ffe7-54c11af79797', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_create', 'sr_ref': 'OpaqueRef:ca45453a-eeb5-1da1-837c-607133a54114', 'vdi_sm_config': {'type': 'raw'}} Aug 14 10:29:33 arb-xen3 SM: [8648] RBDVDI.create for fc8df789-83c0-4ad7-b128-3e88e12b3c53 Aug 14 10:29:33 arb-xen3 SM: [8648] ['rbd', 'create', 'VHD-fc8df789-83c0-4ad7-b128-3e88e12b3c53', '--size', '0', '--object-size', '2097152', '--image-format', '2', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Aug 14 10:29:34 arb-xen3 SM: [8648] pread SUCCESS Aug 14 10:29:34 arb-xen3 SM: [8648] ['rbd', 'image-meta', 'set', 'VHD-fc8df789-83c0-4ad7-b128-3e88e12b3c53', 'VDI_LABEL', 'Statefile for HA', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Aug 14 10:29:34 arb-xen3 SM: [8648] pread SUCCESS Aug 14 10:29:34 arb-xen3 SM: [8648] ['rbd', 'image-meta', 'set', 'VHD-fc8df789-83c0-4ad7-b128-3e88e12b3c53', 'VDI_DESCRIPTION', 'Used for storage heartbeating', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Aug 14 10:29:34 arb-xen3 SM: [8648] pread SUCCESS Aug 14 10:29:34 arb-xen3 SM: [8783] ['uuidgen', '-r'] Aug 14 10:29:34 arb-xen3 SM: [8783] pread SUCCESS Aug 14 10:29:34 arb-xen3 SM: [8783] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Aug 14 10:29:34 arb-xen3 SM: [8783] pread SUCCESS Aug 14 10:29:34 arb-xen3 SM: [8783] vdi_create {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'DummyRef:|51afc66f-0528-a4be-be5b-220edddd3402|VDI.create', 'vdi_type': 'redo_log', 'args': ['268435456', 'Metadata for HA', 'Used for master failover', '', 'false', '19700101T00:00:00Z', '', 'false'], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:8e01113f-b68c-cfc8-ef7f-4d806e15b602', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_create', 'sr_ref': 'OpaqueRef:ca45453a-eeb5-1da1-837c-607133a54114', 'vdi_sm_config': {'type': 'raw'}} Aug 14 10:29:34 arb-xen3 SM: [8783] RBDVDI.create for aa386559-0f70-474a-8e84-1e309f6f2156 Aug 14 10:29:34 arb-xen3 SM: [8783] ['rbd', 'create', 'VHD-aa386559-0f70-474a-8e84-1e309f6f2156', '--size', '256', '--object-size', '2097152', '--image-format', '2', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Aug 14 10:29:35 arb-xen3 SM: [8783] pread SUCCESS Aug 14 10:29:35 arb-xen3 SM: [8783] ['rbd', 'image-meta', 'set', 'VHD-aa386559-0f70-474a-8e84-1e309f6f2156', 'VDI_LABEL', 'Metadata for HA', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Aug 14 10:29:35 arb-xen3 SM: [8783] pread SUCCESS Aug 14 10:29:35 arb-xen3 SM: [8783] ['rbd', 'image-meta', 'set', 'VHD-aa386559-0f70-474a-8e84-1e309f6f2156', 'VDI_DESCRIPTION', 'Used for master failover', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Aug 14 10:29:35 arb-xen3 SM: [8783] pread SUCCESS Aug 14 10:29:35 arb-xen3 SM: [8921] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Aug 14 10:29:36 arb-xen3 SM: [8921] pread SUCCESS Aug 14 10:29:36 arb-xen3 SM: [8921] vdi_generate_config {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:27f868d0-e195-2f92-7dea-5294e6d86180', 'vdi_ref': 'OpaqueRef:524feed2-f5e4-ce90-dacb-84b807139caa', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': 'fc8df789-83c0-4ad7-b128-3e88e12b3c53', 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:31298192-e6d0-a3a1-9cca-985307626720', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_generate_config', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:ca45453a-eeb5-1da1-837c-607133a54114', 'vdi_uuid': 'fc8df789-83c0-4ad7-b128-3e88e12b3c53'} Aug 14 10:29:36 arb-xen3 SM: [8921] RBDVDI.generate_config Aug 14 10:29:36 arb-xen3 SM: [8952] Failed to parse commandline; exception = 'device_config' argv = ['/opt/xensource/sm/RBDSR', "<?xml version='1.0'?>\n\nvdi_attach_from_config\n\n\n\n\nsr_uuid\n4ceb0f8a-1539-40a4-bee2-450a025b04e1\n\n\ncommand\nvdi_attach_from_config\n\n\nvdi_uuid\nfc8df789-83c0-4ad7-b128-3e88e12b3c53\n\n\n\n\n"] Aug 14 10:29:36 arb-xen3 SM: [8952] Raising exception [143, Failed to parse the request] Aug 14 10:29:36 arb-xen3 SM: [8952] *\ RBD: EXCEPTION <class 'SR.SROSError'>, Failed to parse the request Aug 14 10:29:36 arb-xen3 SM: [8952] File "/opt/xensource/sm/SRCommand.py", line 348, in run Aug 14 10:29:36 arb-xen3 SM: [8952] cmd.parse() Aug 14 10:29:36 arb-xen3 SM: [8952] File "/opt/xensource/sm/SRCommand.py", line 101, in parse Aug 14 10:29:36 arb-xen3 SM: [8952] raise xs_errors.XenError('BadRequest') Aug 14 10:29:36 arb-xen3 SM: [8952] File "/opt/xensource/sm/xs_errors.py", line 52, in init* Aug 14 10:29:36 arb-xen3 SM: [8952] raise SR.SROSError(errorcode, errormessage) Aug 14 10:29:36 arb-xen3 SM: [8952] Aug 14 10:29:36 arb-xen3 SM: [8956] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Aug 14 10:29:37 arb-xen3 SM: [8956] pread SUCCESS Aug 14 10:29:37 arb-xen3 SM: [8956] vdi_detach {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:677b70f6-73f8-8912-53f3-e78f7a3351a9', 'vdi_ref': 'OpaqueRef:524feed2-f5e4-ce90-dacb-84b807139caa', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': 'fc8df789-83c0-4ad7-b128-3e88e12b3c53', 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:025bba04-b2d0-4ad0-80d8-16eb29cbb35b', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_detach', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:ca45453a-eeb5-1da1-837c-607133a54114', 'vdi_uuid': 'fc8df789-83c0-4ad7-b128-3e88e12b3c53'} Aug 14 10:29:37 arb-xen3 SM: [8956] lock: opening lock file /var/lock/sm/fc8df789-83c0-4ad7-b128-3e88e12b3c53/vdi Aug 14 10:29:37 arb-xen3 SM: [8956] lock: closed /var/lock/sm/fc8df789-83c0-4ad7-b128-3e88e12b3c53/vdi Aug 14 10:29:53 arb-xen3 SM: [9061] sr_scan {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|fbe92242-3794-8377-e674-0648bc28fdd8|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:7999bf48-7189-6fe9-64ec-8b7f810c4ff9', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'} Aug 14 10:29:53 arb-xen3 SM: [9076] sr_update {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|797fa15d-6698-ebb8-2e10-752a08dbb368|SR.stat', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:e36b391c-0a8b-5758-d424-8a5be0f149fe', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_update', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'} Aug 14 10:30:23 arb-xen3 SM: [9212] sr_scan {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|3e27d6ef-36cc-32cf-afbd-74acd87b8298|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:b455db94-1f5a-de2e-bc02-05a899fd6b41', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'} Aug 14 10:30:23 arb-xen3 SM: [9227] sr_update {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|2d5999c7-2a1f-382e-d4d0-e01d37defbc4|SR.stat', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:e5da20c4-47da-7e6d-8555-81a93b3c9f4d', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_update', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'} Aug 14 10:30:53 arb-xen3 SM: [9357] sr_scan {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|c7d32ae8-fda3-5cea-0f78-8f475ab802a2|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:a916e1d9-3130-3f33-e1a7-65a5760eeff5', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'} Aug 14 10:30:53 arb-xen3 SM: [9372] sr_update {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|6a828226-379b-90a3-1e6b-0c9e2c919f1a|SR.stat', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:a3157a1e-1302-176d-fa8b-2d0f0db77095', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_update', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'} Aug 14 10:31:23 arb-xen3 SM: [9502] sr_scan {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|7e4fcf49-bfba-b6e2-d2dd-92863aec0dad|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:9dcee133-6bd6-8116-cf9e-fa9b76ea7646', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'} Aug 14 10:31:23 arb-xen3 SM: [9518] sr_update {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|73277515-dec7-0ed0-8fdb-4b2b391b97ab|SR.stat', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:1085dfe4-7d8c-659b-7bf9-4fd812f3da05', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_update', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'} Aug 14 10:31:24 arb-xen3 SMGC: [6342] GC active, quiet period ended Aug 14 10:31:24 arb-xen3 SM: [6342] lock: tried lock /var/lock/sm/17d872f0-20f8-2e87-de98-5110566873b8/sr, acquired: True (exists: True) Aug 14 10:31:24 arb-xen3 SM: [6342] LVMCache: refreshing Aug 14 10:31:24 arb-xen3 SM: [6342] ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-17d872f0-20f8-2e87-de98-5110566873b8'] Aug 14 10:31:24 arb-xen3 SM: [6342] pread SUCCESS Aug 14 10:31:24 arb-xen3 SM: [6342] ['/usr/bin/vhd-util', 'scan', '-f', '-c', '-m', 'VHD-', '-l', 'VG_XenStorage-17d872f0-20f8-2e87-de98-5110566873b8'] Aug 14 10:31:24 arb-xen3 SM: [6342] pread SUCCESS Aug 14 10:31:24 arb-xen3 SMGC: [6342] SR 17d8 ('local storage') (1 VDIs in 1 VHD trees): Aug 14 10:31:24 arb-xen3 SMGC: [6342] cb45e34cVHD Aug 14 10:31:24 arb-xen3 SMGC: [6342] Aug 14 10:31:24 arb-xen3 SM: [6342] lock: released /var/lock/sm/17d872f0-20f8-2e87-de98-5110566873b8/sr Aug 14 10:31:24 arb-xen3 SM: [6342] ['/sbin/vgs', '--noheadings', '--nosuffix', '--units', 'b', 'VG_XenStorage-17d872f0-20f8-2e87-de98-5110566873b8'] Aug 14 10:31:24 arb-xen3 SM: [6342] pread SUCCESS Aug 14 10:31:24 arb-xen3 SM: [6342] ['/sbin/vgs', '--noheadings', '--nosuffix', '--units', 'b', 'VG_XenStorage-17d872f0-20f8-2e87-de98-5110566873b8'] Aug 14 10:31:24 arb-xen3 SM: [6342] pread SUCCESS Aug 14 10:31:24 arb-xen3 SMGC: [6342] No work, exiting Aug 14 10:31:24 arb-xen3 SMGC: [6342] GC process exiting, no work left Aug 14 10:31:24 arb-xen3 SM: [6342] lock: released /var/lock/sm/17d872f0-20f8-2e87-de98-5110566873b8/gc_active Aug 14 10:31:24 arb-xen3 SMGC: [6342] SR 17d8 ('local storage') (1 VDIs in 1 VHD trees): no changes Aug 14 10:31:53 arb-xen3 SM: [9656] sr_scan {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|15d87e62-fcd6-5eb1-f35f-7186c6a92035|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:2fe38590-9266-29ee-d6b3-98130bb34cd8', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'} Aug 14 10:31:53 arb-xen3 SM: [9675] sr_update {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|4643c98d-4a1a-90f4-c4dd-738a84732036|SR.stat', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:13fff3d7-85c4-ee68-c34e-cdeebe276237', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_update', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'}

rposudnevskiy commented 8 years ago

One more fix

djmoonshine commented 8 years ago

Hello again!

Great! Now the guide to enable HA completes without any errors. Also if any protected VM is shutdown it's restarted automatically. So far so good. What i'm not able to test is is to kill one Xenserver and make sure protected VM:s on the killed Xenserver is booted on another one in the pool. This is because i only have one server in my Xenserver 7 Pool and i'm on holidays this week. I will test this as soon as i get back so i can install another server.

zigmund commented 7 years ago

Hi there!

Having problems with HA via ceph too. 3 nodes, no VMs, trying to enable HA. Got this in SMlog:

Sep 19 14:51:11 alahd-xen2 SM: [22201] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Sep 19 14:51:12 alahd-xen2 SM: [22201] pread SUCCESS Sep 19 14:51:12 alahd-xen2 SM: [22201] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Sep 19 14:51:12 alahd-xen2 SM: [22201] pread SUCCESS Sep 19 14:51:12 alahd-xen2 SM: [22201] vdi_generate_config {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:f29e6641-87fa-d985-9268-107822105265', 'vdi_ref': 'OpaqueRef:26516ae1-cae0-4319-ccb1-aaff2a2af419', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '78edbe81-3530-4cc7-8834-dca1c385f99f', 'host_ref': 'OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01', 'session_ref': 'OpaqueRef:d0e98f1a-3e25-f983-af2d-9f5e907b3234', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_generate_config', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:224a0edd-4c3e-4e0d-4091-d3fd8b9c6ad9', 'vdi_uuid': '78edbe81-3530-4cc7-8834-dca1c385f99f'} Sep 19 14:51:12 alahd-xen2 SM: [22201] RBDVDI.generate_config Sep 19 14:51:12 alahd-xen2 SM: [22254] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Sep 19 14:51:12 alahd-xen2 SM: [22254] pread SUCCESS Sep 19 14:51:12 alahd-xen2 SM: [22254] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Sep 19 14:51:12 alahd-xen2 SM: [22254] pread SUCCESS Sep 19 14:51:12 alahd-xen2 SM: [22254] vdi_attach_from_config {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_attach_from_config', 'vdi_uuid': '78edbe81-3530-4cc7-8834-dca1c385f99f'} Sep 19 14:51:12 alahd-xen2 SM: [22254] RBDVDI.attach_from_config Sep 19 14:51:12 alahd-xen2 SM: [22254] RBDSR.attach for 4ceb0f8a-1539-40a4-bee2-450a025b04e1 Sep 19 14:51:12 alahd-xen2 SM: [22254] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Sep 19 14:51:13 alahd-xen2 SM: [22254] pread SUCCESS Sep 19 14:51:13 alahd-xen2 SM: [22254] ['mkdir', '-p', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1'] Sep 19 14:51:13 alahd-xen2 SM: [22254] pread SUCCESS Sep 19 14:51:13 alahd-xen2 SM: [22254] ['rbd-nbd', '--nbds_max', '64', 'map', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-78edbe81-3530-4cc7-8834-dca1c385f99f', '--name', 'client.admin'] Sep 19 14:51:13 alahd-xen2 SM: [22254] pread SUCCESS Sep 19 14:51:13 alahd-xen2 SM: [22254] ['ln', '-s', '/dev/nbd0', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-78edbe81-3530-4cc7-8834-dca1c385f99f'] Sep 19 14:51:13 alahd-xen2 SM: [22254] pread SUCCESS Sep 19 14:51:13 alahd-xen2 SM: [22379] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Sep 19 14:51:14 alahd-xen2 SM: [22379] pread SUCCESS Sep 19 14:51:14 alahd-xen2 SM: [22379] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Sep 19 14:51:14 alahd-xen2 SM: [22379] pread SUCCESS Sep 19 14:51:14 alahd-xen2 SM: [22379] vdi_generate_config {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:eb2dcbc5-786f-c551-b8d8-b24f7d12c684', 'vdi_ref': 'OpaqueRef:3f3a5442-2870-755e-df3f-be4db94df5a5', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1', 'host_ref': 'OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01', 'session_ref': 'OpaqueRef:c6d7a2e4-6d1c-a156-299e-52c363f93cdc', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_generate_config', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:224a0edd-4c3e-4e0d-4091-d3fd8b9c6ad9', 'vdi_uuid': '3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1'} Sep 19 14:51:14 alahd-xen2 SM: [22379] RBDVDI.generate_config Sep 19 14:51:14 alahd-xen2 SM: [22432] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Sep 19 14:51:14 alahd-xen2 SM: [22432] pread SUCCESS Sep 19 14:51:14 alahd-xen2 SM: [22432] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Sep 19 14:51:14 alahd-xen2 SM: [22432] pread SUCCESS Sep 19 14:51:14 alahd-xen2 SM: [22432] vdi_attach_from_config {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_attach_from_config', 'vdi_uuid': '3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1'} Sep 19 14:51:14 alahd-xen2 SM: [22432] RBDVDI.attach_from_config Sep 19 14:51:14 alahd-xen2 SM: [22432] RBDSR.attach for 4ceb0f8a-1539-40a4-bee2-450a025b04e1 Sep 19 14:51:14 alahd-xen2 SM: [22432] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Sep 19 14:51:15 alahd-xen2 SM: [22432] pread SUCCESS Sep 19 14:51:15 alahd-xen2 SM: [22432] ['mkdir', '-p', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1'] Sep 19 14:51:15 alahd-xen2 SM: [22432] pread SUCCESS Sep 19 14:51:15 alahd-xen2 SM: [22432] ['rbd-nbd', '--nbds_max', '64', 'map', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1', '--name', 'client.admin'] Sep 19 14:51:15 alahd-xen2 SM: [22432] pread SUCCESS Sep 19 14:51:15 alahd-xen2 SM: [22432] ['ln', '-s', '/dev/nbd1', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1'] Sep 19 14:51:15 alahd-xen2 SM: [22432] pread SUCCESS

Then freeze for ~2 minutes, in xensource.log got:

Sep 19 14:53:19 alahd-xen2 xapi: [error|alahd-xen2|579 INET :::80|host.ha_join_liveset R:194ce241ab08|xapi] HA enable failed with BOOTJOIN_TIMEOUT

And then in SMlog:

Sep 19 14:53:20 alahd-xen2 SM: [23049] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Sep 19 14:53:20 alahd-xen2 SM: [23049] pread SUCCESS Sep 19 14:53:20 alahd-xen2 SM: [23049] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Sep 19 14:53:20 alahd-xen2 SM: [23049] pread SUCCESS Sep 19 14:53:20 alahd-xen2 SM: [23049] vdi_deactivate {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:e3037904-79a3-c69a-e038-da1b5dd16e71', 'vdi_ref': 'OpaqueRef:26516ae1-cae0-4319-ccb1-aaff2a2af419', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '78edbe81-3530-4cc7-8834-dca1c385f99f', 'host_ref': 'OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01', 'session_ref': 'OpaqueRef:eb43896d-b93f-8308-8d17-2d98d3147028', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_deactivate', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:224a0edd-4c3e-4e0d-4091-d3fd8b9c6ad9', 'vdi_uuid': '78edbe81-3530-4cc7-8834-dca1c385f99f'} Sep 19 14:53:20 alahd-xen2 SM: [23049] lock: opening lock file /var/lock/sm/78edbe81-3530-4cc7-8834-dca1c385f99f/vdi Sep 19 14:53:20 alahd-xen2 SM: [23049] blktap2.deactivate Sep 19 14:53:20 alahd-xen2 SM: [23049] lock: acquired /var/lock/sm/78edbe81-3530-4cc7-8834-dca1c385f99f/vdi Sep 19 14:53:20 alahd-xen2 SM: [23049] Backend path /dev/sm/backend/4ceb0f8a-1539-40a4-bee2-450a025b04e1/78edbe81-3530-4cc7-8834-dca1c385f99f does not exist Sep 19 14:53:20 alahd-xen2 SM: [23049] Calling _unmap_VHD Sep 19 14:53:20 alahd-xen2 SM: [23049] Calling ceph_plugin Sep 19 14:53:20 alahd-xen2 SM: [23049] Calling rbd/nbd map on localhost OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01 Sep 19 14:53:20 alahd-xen2 SM: [23114] ['realpath', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-78edbe81-3530-4cc7-8834-dca1c385f99f'] Sep 19 14:53:20 alahd-xen2 SM: [23114] pread SUCCESS Sep 19 14:53:20 alahd-xen2 SM: [23114] ['unlink', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-78edbe81-3530-4cc7-8834-dca1c385f99f'] Sep 19 14:53:20 alahd-xen2 SM: [23114] pread SUCCESS Sep 19 14:53:20 alahd-xen2 SM: [23114] ['rbd-nbd', 'unmap', '/dev/nbd0', '--name', 'client.admin'] Sep 19 14:53:20 alahd-xen2 SM: [23114] pread SUCCESS Sep 19 14:53:20 alahd-xen2 SM: [23049] WARNING: host key host_OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01 not found! Sep 19 14:53:20 alahd-xen2 SM: [23049] lock: released /var/lock/sm/78edbe81-3530-4cc7-8834-dca1c385f99f/vdi Sep 19 14:53:20 alahd-xen2 SM: [23049] lock: closed /var/lock/sm/78edbe81-3530-4cc7-8834-dca1c385f99f/vdi Sep 19 14:53:20 alahd-xen2 SM: [23126] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Sep 19 14:53:21 alahd-xen2 SM: [23126] pread SUCCESS Sep 19 14:53:21 alahd-xen2 SM: [23126] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Sep 19 14:53:21 alahd-xen2 SM: [23126] pread SUCCESS Sep 19 14:53:21 alahd-xen2 SM: [23126] vdi_detach {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:e3037904-79a3-c69a-e038-da1b5dd16e71', 'vdi_ref': 'OpaqueRef:26516ae1-cae0-4319-ccb1-aaff2a2af419', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '78edbe81-3530-4cc7-8834-dca1c385f99f', 'host_ref': 'OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01', 'session_ref': 'OpaqueRef:1001a030-6a52-fcb6-95af-fbafb348d29f', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_detach', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:224a0edd-4c3e-4e0d-4091-d3fd8b9c6ad9', 'vdi_uuid': '78edbe81-3530-4cc7-8834-dca1c385f99f'} Sep 19 14:53:21 alahd-xen2 SM: [23126] lock: opening lock file /var/lock/sm/78edbe81-3530-4cc7-8834-dca1c385f99f/vdi Sep 19 14:53:21 alahd-xen2 SM: [23126] lock: closed /var/lock/sm/78edbe81-3530-4cc7-8834-dca1c385f99f/vdi Sep 19 14:53:21 alahd-xen2 SM: [23182] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Sep 19 14:53:21 alahd-xen2 SM: [23182] pread SUCCESS Sep 19 14:53:21 alahd-xen2 SM: [23182] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Sep 19 14:53:21 alahd-xen2 SM: [23182] pread SUCCESS Sep 19 14:53:21 alahd-xen2 SM: [23182] vdi_deactivate {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:e3037904-79a3-c69a-e038-da1b5dd16e71', 'vdi_ref': 'OpaqueRef:3f3a5442-2870-755e-df3f-be4db94df5a5', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1', 'host_ref': 'OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01', 'session_ref': 'OpaqueRef:8690e36e-1b69-5633-7fa3-cfb87b3f3e07', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_deactivate', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:224a0edd-4c3e-4e0d-4091-d3fd8b9c6ad9', 'vdi_uuid': '3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1'} Sep 19 14:53:21 alahd-xen2 SM: [23182] lock: opening lock file /var/lock/sm/3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1/vdi Sep 19 14:53:21 alahd-xen2 SM: [23182] blktap2.deactivate Sep 19 14:53:21 alahd-xen2 SM: [23182] lock: acquired /var/lock/sm/3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1/vdi Sep 19 14:53:21 alahd-xen2 SM: [23182] Backend path /dev/sm/backend/4ceb0f8a-1539-40a4-bee2-450a025b04e1/3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1 does not exist Sep 19 14:53:21 alahd-xen2 SM: [23182] Calling _unmap_VHD Sep 19 14:53:21 alahd-xen2 SM: [23182] Calling ceph_plugin Sep 19 14:53:21 alahd-xen2 SM: [23182] Calling rbd/nbd map on localhost OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01 Sep 19 14:53:22 alahd-xen2 SM: [23240] ['realpath', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1'] Sep 19 14:53:22 alahd-xen2 SM: [23240] pread SUCCESS Sep 19 14:53:22 alahd-xen2 SM: [23240] ['unlink', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1'] Sep 19 14:53:22 alahd-xen2 SM: [23240] pread SUCCESS Sep 19 14:53:22 alahd-xen2 SM: [23240] ['rbd-nbd', 'unmap', '/dev/nbd1', '--name', 'client.admin'] Sep 19 14:53:22 alahd-xen2 SM: [23240] pread SUCCESS Sep 19 14:53:22 alahd-xen2 SM: [23182] WARNING: host key host_OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01 not found! Sep 19 14:53:22 alahd-xen2 SM: [23182] lock: released /var/lock/sm/3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1/vdi Sep 19 14:53:22 alahd-xen2 SM: [23182] lock: closed /var/lock/sm/3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1/vdi Sep 19 14:53:22 alahd-xen2 SM: [23253] ['ceph', 'df', '--format', 'json', '--name', 'client.admin'] Sep 19 14:53:22 alahd-xen2 SM: [23253] pread SUCCESS Sep 19 14:53:22 alahd-xen2 SM: [23253] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin'] Sep 19 14:53:22 alahd-xen2 SM: [23253] pread SUCCESS Sep 19 14:53:22 alahd-xen2 SM: [23253] vdi_detach {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:e3037904-79a3-c69a-e038-da1b5dd16e71', 'vdi_ref': 'OpaqueRef:3f3a5442-2870-755e-df3f-be4db94df5a5', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1', 'host_ref': 'OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01', 'session_ref': 'OpaqueRef:411c2241-37e0-d09d-f14d-472f51b5fe18', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_detach', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:224a0edd-4c3e-4e0d-4091-d3fd8b9c6ad9', 'vdi_uuid': '3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1'} Sep 19 14:53:22 alahd-xen2 SM: [23253] lock: opening lock file /var/lock/sm/3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1/vdi Sep 19 14:53:22 alahd-xen2 SM: [23253] lock: closed /var/lock/sm/3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1/vdi

In dmesg got:

[ 9163.277373] block nbd0: NBD_DISCONNECT [ 9163.277484] block nbd0: Receive control failed (result -32) [ 9163.277534] block nbd0: queue cleared [ 9164.664371] block nbd1: NBD_DISCONNECT [ 9164.664455] block nbd1: Receive control failed (result -32) [ 9164.664504] block nbd1: queue cleared

Also tried to use kernel and fuse modes, but failed with different errors. Tried to deploy few VMs with ceph - all works well, including live migration.

rposudnevskiy commented 7 years ago

Hi, By default in Jewel release and higher the librbd caching is enabled by default. This is was the reason why HA daemon couldn't start. The problem can be addressed by disabling librbd caching by adding to ceph.conf

[client] rbd cache = false

But it's not good idea to disable caching completely. The disabling librbd caching makes sense only for shared VDIs or for HA VDIs So in the last fix the modified ceph.conf is made and rbd-nbd is invoked with this modified ceph.conf only for shared and HA VDIs. For all other VDIs rbd-nbd is invoked as usual.