Open dlux95 opened 7 years ago
Is XenMotion possible now? Need to reinstall a complete Pool because of misbehaving XenServer Hosts after applying RBDSR. I don't wan't to reinstall them twice in case it doesn't work yet.
Hi Daniel, I tested the latest version (commit d7b62cf) on my test environment with migration of image from local storage to Ceph and back. It worked for me, but I can't give you any warranty that it will work in your case. So you are welcome to test it and give feedback.
Thank you, Roman
Hi Roman, I did not expect any warranty. Corner cases are not the main issue but i wanted to know if anyone tested basic functionality before im going to invest time settung up my entire pool.
Thanks, Daniel
Hi again,
Can you explain your test environment a little bit? I tried to clean out my hosts from the previous installation but all operations seem instable (sometimes they work, sometimes not). Reinstalled all hosts again and trying RBDSR again on clean hosts next week.
Daniel
Hi Daniel,
My test environment consist of 5 VM under VirtualBox. 3 VMs - Ceph cluster 2 VMs - XenServer hosts in pool
Roman
Hi,
just tested rbdsr on my pool. The first storage migratet flawlessly, the next did not work:
May 11 22:54:32 application1 xapi: [ info|application1|28965 UNIX /var/lib/xcp/xapi|dispatch:VDI.db_forget D:d092f413ed2b|taskhelper] task VDI.db_forget R:df0dffa454c6 (uuid:e9e35823-e4a8-89f0-95db-397454084a77) created (trackid=40e3a82c4ba003d6d126d72f4bd0f7cd) by task D:b26e4b9263a7
May 11 22:54:32 application1 xapi: [debug|application1|28965 UNIX /var/lib/xcp/xapi|VDI.db_forget R:df0dffa454c6|xapi] db_forget uuid=f6cfc3a7-e9ec-4b81-848b-0d8769d193d6
May 11 22:54:32 application1 xapi: [ info|application1|28905 |sm_exec D:04220a49c9aa|xapi] Session.destroy trackid=40e3a82c4ba003d6d126d72f4bd0f7cd
May 11 22:54:32 application1 xapi: [debug|application1|28902 INET :::80||mscgen] xapi=>xapi [label="session.logout"];
May 11 22:54:32 application1 xapi: [ info|application1|28966 INET :::80|session.logout D:a3cf5172c9ad|xapi] Session.destroy trackid=f2d72fab9153978c6ceac39491e1e0e1
May 11 22:54:32 application1 xapi: [debug|application1|28825 INET :::80|Querying services D:8df19933b652|mscgen] smapiv2=>smapiv2 [label="VDI.destroy"];
May 11 22:54:32 application1 xapi: [debug|application1|28967 INET :::80||mscgen] xapi=>xapi [label="session.slave_login"];
May 11 22:54:32 application1 xapi: [ info|application1|28968 INET :::80|session.slave_login D:8a24310da781|xapi] Session.create trackid=80290e9a7969f03b5f3deb3522d769c1 pool=true uname= originator=xapi is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
May 11 22:54:32 application1 xapi: [debug|application1|28968 INET :::80|session.slave_login D:8a24310da781|mscgen] xapi=>xapi [label="pool.get_all"];
May 11 22:54:32 application1 xapi: [debug|application1|28969 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:pool.get_all D:7d766405d7b7 created by task D:8a24310da781
May 11 22:54:32 application1 xapi: [ info|application1|28967 INET :::80|Querying services D:0451640cc270|storage_impl] VDI.destroy dbg:Async.VDI.pool_migrate R:5fa10ad791e5 sr:4ceb0f8a-1539-40a4-bee2-450a025b04e1 vdi:c39691f3-5798-464d-97aa-250c68c83a52
May 11 22:54:32 application1 xapi: [debug|application1|28970 ||dummytaskhelper] task VDI.destroy D:b11e179c5af3 created by task R:pool_migate
May 11 22:54:32 application1 xapi: [error|application1|28970 ||backtrace] VDI.destroy D:b11e179c5af3 failed with exception Storage_access.No_VDI
May 11 22:54:32 application1 xapi: [error|application1|28970 ||backtrace] Raised Storage_access.No_VDI
May 11 22:54:32 application1 xapi: [error|application1|28970 ||backtrace] 1/1 xapi @ application1 Raised at file (Thread 28970 has no backtrace table. Was with_backtraces called?, line 0
May 11 22:54:32 application1 xapi: [error|application1|28970 ||backtrace]
May 11 22:54:32 application1 xapi: [debug|application1|28967 INET :::80||mscgen] xapi=>xapi [label="session.logout"];
May 11 22:54:32 application1 xapi: [ info|application1|28971 INET :::80|session.logout D:2e75e945f0fa|xapi] Session.destroy trackid=80290e9a7969f03b5f3deb3522d769c1
May 11 22:54:32 application1 xapi: [debug|application1|28825 INET :::80|Querying services D:8df19933b652|storage_migrate] Got failure: checking for redirect
May 11 22:54:32 application1 xapi: [debug|application1|28825 INET :::80|Querying services D:8df19933b652|storage_migrate] Call was: -> VDI.destroy({dbg:S(Async.VDI.pool_migrate R:5fa10ad791e5);sr:S(4ceb0f8a-1539-40a4-bee2-450a025b04e1);vdi:S(c39691f3-5798-464d-97aa-250c68c83a52)})
May 11 22:54:32 application1 xapi: [debug|application1|28825 INET :::80|Querying services D:8df19933b652|storage_migrate] result.contents: ["Vdi_does_not_exist", "c39691f3-5798-464d-97aa-250c68c83a52"]
May 11 22:54:32 application1 xapi: [debug|application1|28825 INET :::80|Querying services D:8df19933b652|storage_migrate] Not a redirect
May 11 22:54:32 application1 xapi: [debug|application1|28825 INET :::80|Querying services D:8df19933b652|storage_migrate] Called from file \"lib/debug.ml\", line 285, characters 6-10\n
May 11 22:54:32 application1 xapi: [debug|application1|28707 INET :::80||mscgen] xapi=>xapi [label="session.logout"];
May 11 22:54:32 application1 xapi: [ info|application1|28972 INET :::80|session.logout D:c8da7aee4f90|xapi] Session.destroy trackid=a61df35043090aa6e0fbe683bc93e108
May 11 22:54:32 application1 xapi: [debug|application1|24853 |Async.VDI.pool_migrate R:5fa10ad791e5|mscgen] xapi=>smapiv2 [label="DATA.MIRROR.stat"];
May 11 22:54:32 application1 xapi: [debug|application1|28973 INET :::80||mscgen] xapi=>xapi [label="session.slave_login"];
May 11 22:54:32 application1 xapi: [ info|application1|28974 INET :::80|session.slave_login D:79dc4b771411|xapi] Session.create trackid=8a6b13cb50b2bbb6b6c12ab160e57b5a pool=true uname= originator=xapi is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
May 11 22:54:32 application1 xapi: [debug|application1|28974 INET :::80|session.slave_login D:79dc4b771411|mscgen] xapi=>xapi [label="pool.get_all"];
May 11 22:54:32 application1 xapi: [debug|application1|28975 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:pool.get_all D:57fe2390afb9 created by task D:79dc4b771411
May 11 22:54:32 application1 xapi: [ info|application1|28973 INET :::80|Querying services D:3e91e40b521a|storage_impl] DATA.MIRROR.stat dbg:Async.VDI.pool_migrate R:5fa10ad791e5 id:21a611e8-4262-3a6b-fe12-dce02cbccd8d/345f1bff-85e4-48e6-9d2e-dbca5ea0f767
May 11 22:54:32 application1 xapi: [debug|application1|28973 INET :::80||mscgen] xapi=>xapi [label="session.logout"];
May 11 22:54:32 application1 xapi: [ info|application1|28976 INET :::80|session.logout D:4871be303312|xapi] Session.destroy trackid=8a6b13cb50b2bbb6b6c12ab160e57b5a
May 11 22:54:32 application1 xapi: [debug|application1|24853 |Async.VDI.pool_migrate R:5fa10ad791e5|storage_migrate] Got failure: checking for redirect
May 11 22:54:32 application1 xapi: [debug|application1|24853 |Async.VDI.pool_migrate R:5fa10ad791e5|storage_migrate] Call was: -> DATA.MIRROR.stat({dbg:S(Async.VDI.pool_migrate R:5fa10ad791e5);id:S(21a611e8-4262-3a6b-fe12-dce02cbccd8d/345f1bff-85e4-48e6-9d2e-dbca5ea0f767)})
May 11 22:54:32 application1 xapi: [debug|application1|24853 |Async.VDI.pool_migrate R:5fa10ad791e5|storage_migrate] result.contents: ["Does_not_exist", ["mirror", "21a611e8-4262-3a6b-fe12-dce02cbccd8d\/345f1bff-85e4-48e6-9d2e-dbca5ea0f767"]]
May 11 22:54:32 application1 xapi: [debug|application1|24853 |Async.VDI.pool_migrate R:5fa10ad791e5|storage_migrate] Not a redirect
May 11 22:54:32 application1 xapi: [debug|application1|24853 |Async.VDI.pool_migrate R:5fa10ad791e5|mscgen] xapi=>remote_xapi [label="VDI.destroy"];
May 11 22:54:32 application1 xapi: [debug|application1|24853 |Async.VDI.pool_migrate R:5fa10ad791e5|stunnel] stunnel start
May 11 22:54:32 application1 xapi: [debug|application1|24853 |Async.VDI.pool_migrate R:5fa10ad791e5|xmlrpc_client] stunnel pid: 31566 (cached = false) connected to 192.XX.XX.XX:443
May 11 22:54:32 application1 xapi: [debug|application1|24853 |Async.VDI.pool_migrate R:5fa10ad791e5|xmlrpc_client] with_recorded_stunnelpid task_opt=None s_pid=31566
May 11 22:54:32 application1 xapi: [debug|application1|28977 INET :::80|VDI.destroy R:750c2ff071cc|audit] VDI.destroy: VDI = 'invalid'
May 11 22:54:32 application1 xapi: [error|application1|28977 INET :::80|dispatch:VDI.destroy D:2c2af536dfb1|backtrace] VDI.destroy R:750c2ff071cc failed with exception Db_exn.DBCache_NotFound("missing row", "VDI", "OpaqueRef:2590fd1a-56b7-4026-b085-b47fe48edb34")
May 11 22:54:32 application1 xapi: [error|application1|28977 INET :::80|dispatch:VDI.destroy D:2c2af536dfb1|backtrace] Raised Db_exn.DBCache_NotFound("missing row", "VDI", "OpaqueRef:2590fd1a-56b7-4026-b085-b47fe48edb34")
May 11 22:54:32 application1 xapi: [error|application1|28977 INET :::80|dispatch:VDI.destroy D:2c2af536dfb1|backtrace] 1/6 xapi @ application1 Raised at file rbac.ml, line 236
May 11 22:54:32 application1 xapi: [error|application1|28977 INET :::80|dispatch:VDI.destroy D:2c2af536dfb1|backtrace] 2/6 xapi @ application1 Called from file server_helpers.ml, line 76
May 11 22:54:32 application1 xapi: [error|application1|28977 INET :::80|dispatch:VDI.destroy D:2c2af536dfb1|backtrace] 3/6 xapi @ application1 Called from file server_helpers.ml, line 91
May 11 22:54:32 application1 xapi: [error|application1|28977 INET :::80|dispatch:VDI.destroy D:2c2af536dfb1|backtrace] 4/6 xapi @ application1 Called from file lib/pervasiveext.ml, line 22
May 11 22:54:32 application1 xapi: [error|application1|28977 INET :::80|dispatch:VDI.destroy D:2c2af536dfb1|backtrace] 5/6 xapi @ application1 Called from file lib/pervasiveext.ml, line 26
May 11 22:54:32 application1 xapi: [error|application1|28977 INET :::80|dispatch:VDI.destroy D:2c2af536dfb1|backtrace] 6/6 xapi @ application1 Called from file lib/backtrace.ml, line 176
May 11 22:54:32 application1 xapi: [error|application1|28977 INET :::80|dispatch:VDI.destroy D:2c2af536dfb1|backtrace]
May 11 22:54:32 application1 xapi: [error|application1|24853 |Async.VDI.pool_migrate R:5fa10ad791e5|xapi] Failed to destroy remote VDI
May 11 22:54:32 application1 xapi: [debug|application1|24853 |Async.VDI.pool_migrate R:5fa10ad791e5|mscgen] xapi=>smapiv2 [label="DP.destroy"];
May 11 22:54:32 application1 xapi: [debug|application1|28978 INET :::80||mscgen] xapi=>xapi [label="session.slave_login"];
May 11 22:54:32 application1 xapi: [ info|application1|28979 INET :::80|session.slave_login D:f3d5a0e75b80|xapi] Session.create trackid=6ae5fa7a76889f99b63249077c00fe0d pool=true uname= originator=xapi is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
May 11 22:54:32 application1 xapi: [debug|application1|28979 INET :::80|session.slave_login D:f3d5a0e75b80|mscgen] xapi=>xapi [label="pool.get_all"];
May 11 22:54:32 application1 xapi: [debug|application1|28980 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:pool.get_all D:59a0910acace created by task D:f3d5a0e75b80
May 11 22:54:32 application1 xapi: [ info|application1|28978 INET :::80|Querying services D:e817df74e530|storage_impl] DP.destroy dbg:Async.VDI.pool_migrate R:5fa10ad791e5 dp:mirror_vbd/53/xvda allow_leak:false
May 11 22:54:32 application1 xapi: [debug|application1|28978 INET :::80|Querying services D:e817df74e530|storage_impl] dbg:Async.VDI.pool_migrate R:5fa10ad791e5 dp:mirror_vbd/53/xvda sr:21a611e8-4262-3a6b-fe12-dce02cbccd8d vdi:345f1bff-85e4-48e6-9d2e-dbca5ea0f767 superstate:activated RW
May 11 22:54:32 application1 xapi: [debug|application1|28978 INET :::80|Querying services D:e817df74e530|storage_impl] dbg:Async.VDI.pool_migrate R:5fa10ad791e5 dp:mirror_vbd/53/xvda sr:21a611e8-4262-3a6b-fe12-dce02cbccd8d vdi:345f1bff-85e4-48e6-9d2e-dbca5ea0f767 superstate:activated RW
May 11 22:54:32 application1 xapi: [debug|application1|28978 INET :::80||mscgen] xapi=>xapi [label="session.logout"];
May 11 22:54:32 application1 xapi: [error|application1|24853 |Async.VDI.pool_migrate R:5fa10ad791e5|xapi] Caught Storage_interface.Does_not_exist(_): cleaning up
May 11 22:54:32 application1 xapi: [debug|application1|24853 |Async.VDI.pool_migrate R:5fa10ad791e5|xenops] suppressing xenops events on VM: 1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3
May 11 22:54:32 application1 xapi: [ info|application1|28981 INET :::80|session.logout D:21ee0f8ad714|xapi] Session.destroy trackid=6ae5fa7a76889f99b63249077c00fe0d
May 11 22:54:32 application1 xenopsd-xc: [debug|application1|5650 |Async.VDI.pool_migrate R:5fa10ad791e5|xenops_server] VM.stat 1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3
May 11 22:54:32 application1 xapi: [debug|application1|24853 |Async.VDI.pool_migrate R:5fa10ad791e5|xenops] re-enabled xenops events on VM: 1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3; refreshing VM
May 11 22:54:32 application1 xenopsd-xc: [debug|application1|5651 |Async.VDI.pool_migrate R:5fa10ad791e5|xenops_server] UPDATES.refresh_vm 1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3
May 11 22:54:32 application1 xenopsd-xc: [debug|application1|5651 |Async.VDI.pool_migrate R:5fa10ad791e5|xenops_server] VM_DB.signal 1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3
May 11 22:54:32 application1 xenopsd-xc: [debug|application1|5651 |Async.VDI.pool_migrate R:5fa10ad791e5|xenops_server] VBD_DB.signal 1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3.xvda
May 11 22:54:32 application1 xenopsd-xc: [debug|application1|5651 |Async.VDI.pool_migrate R:5fa10ad791e5|xenops_server] VBD_DB.signal 1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3.xvdd
May 11 22:54:32 application1 xenopsd-xc: [debug|application1|5651 |Async.VDI.pool_migrate R:5fa10ad791e5|xenops_server] VIF_DB.signal 1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3.1
May 11 22:54:32 application1 xapi: [debug|application1|117 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops] Processing event: ["Vm", "1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3"]
May 11 22:54:32 application1 xapi: [debug|application1|117 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops] xenops event on VM 1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3
May 11 22:54:32 application1 xenopsd-xc: [debug|application1|5652 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops_server] VM.stat 1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3
May 11 22:54:32 application1 xapi: [debug|application1|24853 |Async.VDI.pool_migrate R:5fa10ad791e5|xenops] Client.UPDATES.inject_barrier 20
May 11 22:54:32 application1 xenopsd-xc: [debug|application1|5653 |Async.VDI.pool_migrate R:5fa10ad791e5|xenops_server] UPDATES.inject_barrier 1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3 20
May 11 22:54:32 application1 xapi: [debug|application1|117 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops] xenopsd event: ignoring event for VM 1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3: metadata has not changed
May 11 22:54:32 application1 xapi: [debug|application1|117 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops] Processing barrier 20
May 11 22:54:32 application1 xapi: [debug|application1|117 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops] Processing event: ["Vbd", ["1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3", "xvda"]]
May 11 22:54:32 application1 xapi: [debug|application1|117 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops] xenops event on VBD 1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3.xvda
May 11 22:54:32 application1 xenopsd-xc: [debug|application1|5655 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops_server] VBD.stat 1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3.xvda
May 11 22:54:32 application1 xenopsd-xc: [debug|application1|5655 |org.xen.xapi.xenops.classic events D:23b896eda772|memory] /usr/bin/ionice -p28499
May 11 22:54:32 application1 xenopsd-xc: [ warn|application1|5655 |org.xen.xapi.xenops.classic events D:23b896eda772|memory] Failed to parse ionice result: unknown: prio 4
May 11 22:54:32 application1 xapi: [debug|application1|117 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops] xenopsd event: ignoring event for VBD 1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3.xvda: metadata has not changed
May 11 22:54:32 application1 xapi: [debug|application1|117 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops] Processing event: ["Vbd", ["1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3", "xvdd"]]
May 11 22:54:32 application1 xapi: [debug|application1|117 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops] xenops event on VBD 1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3.xvdd
May 11 22:54:32 application1 xenopsd-xc: [debug|application1|5656 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops_server] VBD.stat 1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3.xvdd
May 11 22:54:32 application1 xapi: [debug|application1|117 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops] xenopsd event: ignoring event for VBD 1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3.xvdd: metadata has not changed
May 11 22:54:32 application1 xapi: [debug|application1|117 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops] Processing event: ["Vif", ["1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3", "1"]]
May 11 22:54:32 application1 xapi: [debug|application1|117 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops] xenops event on VIF 1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3.1
May 11 22:54:32 application1 xenopsd-xc: [debug|application1|5657 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops_server] VIF.stat 1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3.1
May 11 22:54:32 application1 xapi: [debug|application1|117 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops] xenopsd event: ignoring event for VIF 1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3.1: metadata has not changed
May 11 22:54:32 application1 xenopsd-xc: [debug|application1|5658 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops_server] UPDATES.remove_barrier 20
May 11 22:54:32 application1 xapi: [debug|application1|117 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops] Processing event: ["Vbd", ["1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3", "xvda"]]
May 11 22:54:32 application1 xapi: [debug|application1|24853 |Async.VDI.pool_migrate R:5fa10ad791e5|xapi] Mirror failed for VDI: 345f1bff-85e4-48e6-9d2e-dbca5ea0f767
May 11 22:54:32 application1 xapi: [debug|application1|117 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops] Skipping (already processed this round)
May 11 22:54:32 application1 xapi: [debug|application1|24853 |Async.VDI.pool_migrate R:5fa10ad791e5|mscgen] xapi=>xapi [label="session.logout"];
May 11 22:54:32 application1 xapi: [debug|application1|117 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops] Processing event: ["Vbd", ["1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3", "xvdd"]]
May 11 22:54:32 application1 xapi: [debug|application1|117 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops] Skipping (already processed this round)
May 11 22:54:32 application1 xapi: [debug|application1|117 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops] Processing event: ["Vif", ["1998f2bd-48ed-2e6c-53dc-6ffd2629f9c3", "1"]]
May 11 22:54:32 application1 xapi: [debug|application1|117 |org.xen.xapi.xenops.classic events D:23b896eda772|xenops] Skipping (already processed this round)
May 11 22:54:32 application1 xapi: [debug|application1|28982 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:session.logout D:7cde6341778d created by task R:5fa10ad791e5
May 11 22:54:32 application1 xapi: [ info|application1|28982 UNIX /var/lib/xcp/xapi|session.logout D:b0aec6769fa4|xapi] Session.destroy trackid=97447580dded8fe8e3ae48fd291b6a21
May 11 22:54:32 application1 xapi: [error|application1|24853 ||backtrace] Async.VDI.pool_migrate R:5fa10ad791e5 failed with exception Server_error(MIRROR_FAILED, [ OpaqueRef:f91439c7-ccb7-6c10-02f9-e910a889966b ])
May 11 22:54:32 application1 xapi: [error|application1|24853 ||backtrace] Raised Server_error(MIRROR_FAILED, [ OpaqueRef:f91439c7-ccb7-6c10-02f9-e910a889966b ])
May 11 22:54:32 application1 xapi: [error|application1|24853 ||backtrace] 1/1 xapi @ application1 Raised at file (Thread 24853 has no backtrace table. Was with_backtraces called?, line 0
May 11 22:54:32 application1 xapi: [error|application1|24853 ||backtrace]
Hi Daniel, Could you please give more details. What type of storages did you use? Did you migrate from Ceph or to Ceph? From which type of storage? It would be greate if you can provide the output if these commands: xe sr-list xe vdi-list
Thank you. Best regards. Roman
Hi,
I migrated from rbdsr storage to an nfs rbd ceph brigde (host mounting rbd and exporting xfs filesystem on it with nfs). I will post some stripped versions of these outputs with only the storages in it that are relevant to the migration. This morning i shut down the vm, detached the volume, migrated it to nfs rbd brigde from rbdsr, attached again and started the vm again. That worked on the second try. I also migrated from the nfs rbd brigde to rbdsr with an error but i have no log from that event.
Thank you for your help, Daniel
Am 12. Mai 2017 15:12:24 MESZ schrieb Roman Posudnevskiy notifications@github.com:
Hi Daniel, Could you please give more details. What type of storages did you use? Did you migrate from Ceph or to Ceph? From which type of storage? It would be greate if you can provide the output if these commands: xe sr-list xe vdi-list
Thank you. Best regards. Roman
-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/rposudnevskiy/RBDSR/issues/37#issuecomment-301072615
xe sr-list:
uuid ( RO) : 94f9658c-19cf-b5fb-0582-1c8b637c5a13
name-label ( RW): Removable storage
name-description ( RW):
host ( RO): application1
type ( RO): udev
content-type ( RO): disk
uuid ( RO) : f8dbc814-e66b-7a56-8cfd-67d676501fe8
name-label ( RW): Removable storage
name-description ( RW):
host ( RO): application2
type ( RO): udev
content-type ( RO): disk
uuid ( RO) : 4eab7cee-8ecf-2c41-a00c-291a375c4cc8
name-label ( RW): Local storage
name-description ( RW):
host ( RO): <not in database>
type ( RO): lvm
content-type ( RO): user
uuid ( RO) : 5a816343-ea27-6971-ddfb-3d38b752cdb6
name-label ( RW): DVD drives
name-description ( RW): Physical DVD drives
host ( RO): <not in database>
type ( RO): udev
content-type ( RO): iso
uuid ( RO) : 343f6c57-c2f2-8581-a7c5-484f71808b77
name-label ( RW): DVD drives
name-description ( RW): Physical DVD drives
host ( RO): application1
type ( RO): udev
content-type ( RO): iso
uuid ( RO) : 12e6931a-a94c-2a85-a0f7-94ebacce0486
name-label ( RW): NFS ISO library
name-description ( RW): NFS ISO Library [192.168.0.150:/nfs/xenserver-iso]
host ( RO): <shared>
type ( RO): iso
content-type ( RO): iso
uuid ( RO) : 21a611e8-4262-3a6b-fe12-dce02cbccd8d
name-label ( RW): NFS Ceph Storage
name-description ( RW): NFS SR [192.168.0.150:/nfs/xenserver-storage] 192.168.0.150:/dev/rbd/rbd/xenserver-storage
host ( RO): <shared>
type ( RO): nfs
content-type ( RO):
uuid ( RO) : cc289972-70e5-df7d-0f92-3e71b93b06c5
name-label ( RW): DVD drives
name-description ( RW): Physical DVD drives
host ( RO): application2
type ( RO): udev
content-type ( RO): iso
uuid ( RO) : c490155d-1148-1adf-7d68-dac77d97410b
name-label ( RW): Local storage
name-description ( RW):
host ( RO): application1
type ( RO): lvm
content-type ( RO): user
uuid ( RO) : 4ceb0f8a-1539-40a4-bee2-450a025b04e1
name-label ( RW): CEPH RBD Storage
name-description ( RW):
host ( RO): <shared>
type ( RO): rbd
content-type ( RO): user
uuid ( RO) : 30a582f1-e740-dfcd-b2e8-a5c834daec95
name-label ( RW): XenServer Tools
name-description ( RW): XenServer Tools ISOs
host ( RO): <shared>
type ( RO): iso
content-type ( RO): iso
uuid ( RO) : a92e32c0-c6fe-dfa3-0a0c-14db9bc2122d
name-label ( RW): Removable storage
name-description ( RW):
host ( RO): <not in database>
type ( RO): udev
content-type ( RO): disk
uuid ( RO) : 835a3ac0-7427-8c19-d7d8-bf96e4e8b7a5
name-label ( RW): Local storage
name-description ( RW):
host ( RO): application2
type ( RO): lvm
content-type ( RO): user
xe vdi-list:
uuid ( RO) : a23f9376-81e5-4c11-82fd-899cd5ef6ca4
name-label ( RW): gateway-disk
name-description ( RW):
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 2147483648
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : 345f1bff-85e4-48e6-9d2e-dbca5ea0f767
name-label ( RW): monitoring-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 10737418240
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : 5efd59b9-0723-4f36-84e4-a91f3e76dfe6
name-label ( RW): Metadata for HA
name-description ( RW): Used for master failover
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 268435456
sharable ( RO): true
read-only ( RO): false
uuid ( RO) : 8b06f295-275c-46d2-9aac-bb418214103f
name-label ( RW): etherpad-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 10737418240
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : 53085e54-05b0-4a7c-a409-65a4323bc2be
name-label ( RW): XS71E006.iso
name-description ( RW): Temporary virtual disk used for the installation of an update
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 4194304
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : 1ec61219-647a-475b-8880-a8f25fe5773a
name-label ( RW): base copy
name-description ( RW):
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 3221225472
sharable ( RO): false
read-only ( RO): true
uuid ( RO) : 87bd6230-27d2-49b2-98b5-eda23c95d513
name-label ( RW): guest-tools.iso
name-description ( RW):
sr-uuid ( RO): 30a582f1-e740-dfcd-b2e8-a5c834daec95
virtual-size ( RO): 71301120
sharable ( RO): false
read-only ( RO): true
uuid ( RO) : 6264dcf9-2eae-40c8-a74b-c1c4c1f694e0
name-label ( RW): Pool Metadata Backup
name-description ( RW):
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 262144000
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : d06d882e-2b48-4c1a-b818-303b8b4afa9b
name-label ( RW): log-diskr
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 3221225472
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : 08bb4abb-f5af-432d-b313-42479228c294
name-label ( RW): pfSense-CE-2.3.4-RELEASE-amd64.iso
name-description ( RW):
sr-uuid ( RO): 12e6931a-a94c-2a85-a0f7-94ebacce0486
virtual-size ( RO): 657231872
sharable ( RO): false
read-only ( RO): true
uuid ( RO) : 4aa5563c-f75b-461b-847f-f2f4e379058f
name-label ( RW): log-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 3221225472
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : 9ec964bb-78b7-43ff-b6aa-6ce968f75354
name-label ( RW): base copy
name-description ( RW):
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 10737418240
sharable ( RO): false
read-only ( RO): true
uuid ( RO) : 680dfa4e-724f-4a6c-ac1e-c85c3d018e3c
name-label ( RW): haproxy-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 10737418240
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : 03c8f6f5-47b3-4aa0-ab46-def8650c773f
name-label ( RW): netxcloud-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 10737418240
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : a50fe45a-bf4d-4f2b-bd12-acc87e3e8801
name-label ( RW): radius-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 10737418240
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : c97b8b03-ce7f-4c39-b0c9-cfe1281031d7
name-label ( RW): haproxy-backup
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 3221225472
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : bf867836-838a-4319-8526-1f770d6f8cbc
name-label ( RW): minimal-server-base
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 3221225472
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : bbfe8a00-bfb1-418e-bdde-a8261660be64
name-label ( RW): roundcube-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 10737418240
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : 13d458f5-7e97-486b-abdf-91d693f3f59c
name-label ( RW): log-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 3221225472
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : bf10b3a2-915d-43df-9f22-ab536ea968f7
name-label ( RW): minimal-server-base
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 3221225472
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : b995fa8a-f003-4ff3-af5c-a52ca88e8ee2
name-label ( RW): SCSI 0:0:0:0
name-description ( RW): HL-DT-ST model RW/DVD GCC-C10N rev 2.00 type 5
sr-uuid ( RO): 343f6c57-c2f2-8581-a7c5-484f71808b77
virtual-size ( RO): 1073741312
sharable ( RO): false
read-only ( RO): true
uuid ( RO) : 0329d355-1e08-42e1-a036-30e1610e08f0
name-label ( RW): print-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 3221225472
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : a2cdccb4-6e36-424a-ba24-ca31a4fbbc4c
name-label ( RW): log-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 3221225472
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : e27bb973-0aa1-4fc8-b8c4-e01743211627
name-label ( RW): SCSI 0:0:0:0
name-description ( RW): SONY model DVD-ROM DDU810A rev KD38 type 5
sr-uuid ( RO): 5a816343-ea27-6971-ddfb-3d38b752cdb6
virtual-size ( RO): 1073741312
sharable ( RO): false
read-only ( RO): true
uuid ( RO) : 38e25708-d3aa-4118-8c98-13d4f4c55eeb
name-label ( RW): minimal-server-base
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 3221225472
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : 4b72f119-6655-4786-9d7f-678c5a637807
name-label ( RW): de_windows_7_professional_with_sp1_x64_dvd_u_676919.iso
name-description ( RW):
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 3269459968
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : cbefd7d7-9aba-4b04-ba3e-408bc5ee9310
name-label ( RW): ubuntu-start-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 3221225472
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : eea77b87-5074-4418-92bf-7c0608ba3f68
name-label ( RW): Statefile for HA
name-description ( RW): Used for storage heartbeating
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 266240
sharable ( RO): true
read-only ( RO): false
uuid ( RO) : 5e53cbc7-b64c-4734-b8d8-cbc792997b95
name-label ( RW): netxcloud-diskr
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 4ceb0f8a-1539-40a4-bee2-450a025b04e1
virtual-size ( RO): 10737418240
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : 5e2678c4-84f3-4202-a991-3c3dda278fea
name-label ( RW): base copy
name-description ( RW):
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 3221225472
sharable ( RO): false
read-only ( RO): true
uuid ( RO) : c891d5b3-db9e-4391-bd57-ee9d5f381532
name-label ( RW): netxcloud-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 10737418240
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : 6d797c7c-fc6e-4907-b2f5-88e9669dbd72
name-label ( RW): ntp-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 3221225472
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : 4388b28b-0de9-4d10-8b40-8f88d21f1656
name-label ( RW): redis-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 10737418240
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : a78c21bf-dfed-4e0a-8787-5a59fbb99760
name-label ( RW): mysql-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 10737418240
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : e44fb8b1-0c73-4bd9-a117-a7ba099fdc50
name-label ( RW): log-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 3221225472
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : 9de94464-7af4-418c-8579-f6969b8d45ea
name-label ( RW): minimal-server-base
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 3221225472
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : 9c09c8e5-31bf-4f76-8637-ea949e61e70a
name-label ( RW): mail-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 10737418240
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : e82df494-6cb0-4624-91df-69a21bb43c11
name-label ( RW): log-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 3221225472
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : a7f6f38a-2197-4348-a4d0-aa66f7431ce2
name-label ( RW): XenCenter.iso
name-description ( RW):
sr-uuid ( RO): 30a582f1-e740-dfcd-b2e8-a5c834daec95
virtual-size ( RO): 57546752
sharable ( RO): false
read-only ( RO): true
uuid ( RO) : b45b8d02-c560-4496-a48b-8ca4a35d9794
name-label ( RW): mini.iso
name-description ( RW):
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 60817408
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : e47c4413-d494-48b2-9481-41b9d61fcfd0
name-label ( RW): management-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 25769803776
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : 5b401621-5aa4-4069-8060-0872133bbdc5
name-label ( RW): ldap-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 10737418240
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : e7f03314-62a5-43e7-83ac-4c06592b0e93
name-label ( RW): ubuntu-16.04.1-server-amd64.iso
name-description ( RW):
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 700448768
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : 9ab01b51-8197-402f-ac18-ca191da84c3d
name-label ( RW): XS71E006.iso
name-description ( RW): Temporary virtual disk used for the installation of an update
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 4194304
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : bda79526-8b3b-4798-9e23-2ee97459ad16
name-label ( RW): netxcloud-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 10737418240
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : 8b0dd42a-9197-466e-93cb-5d380562df51
name-label ( RW): dnsmasq-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 10737418240
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : 6e7e0fe7-972b-4414-926b-481c95d4230b
name-label ( RW): minimal-server-base
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 3221225472
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : 23da20e1-f539-4251-a527-08651e89df0d
name-label ( RW): log-disk
name-description ( RW): Created by template provisioner
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 3221225472
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : adc9320a-a751-4d2e-a2fa-31628d89cf7f
name-label ( RW): base copy
name-description ( RW):
sr-uuid ( RO): 21a611e8-4262-3a6b-fe12-dce02cbccd8d
virtual-size ( RO): 3221225472
sharable ( RO): false
read-only ( RO): true
uuid ( RO) : cf5251bb-c63e-4bc0-a5f9-0a4323680e0e
name-label ( RW): Transfer VM system disk
name-description ( RW):
sr-uuid ( RO): 835a3ac0-7427-8c19-d7d8-bf96e4e8b7a5
virtual-size ( RO): 8388608
sharable ( RO): false
read-only ( RO): false
uuid ( RO) : 6d205a81-23da-4eef-980b-ac9775de6ed9
name-label ( RW): SCSI 0:0:0:0
name-description ( RW): HL-DT-ST model RW/DVD GCC-C10N rev 2.00 type 5
sr-uuid ( RO): cc289972-70e5-df7d-0f92-3e71b93b06c5
virtual-size ( RO): 1073741312
sharable ( RO): false
read-only ( RO): true
Hi Daniel, I couldn't reproduce the error in my test environment. I checked live migration from RBDSR to NFS and back and it migrated flawlessly I didn't use Ceph as backend for NFS storage but I think it should't be a reason of problem. Could you test it again and if problem appears again please send me the /var/log/SMlog and /var/log/xensource.log files from your XenServer host
Thank you. Best regards. Roman
Hi,
Could it be that the vdi's or sr's are polluted with broken meta information from previous versions? If that could be the case, how do i cleam them?
Sincerely, Daniel
Hi, Yes, the meta information for vdi could be broken. You can clean it using these command:
xe vdi-forget uuid=here put uuid of vdi that you tried to migrate and received error
xe sr-scan uuid=here put uuid of RBDSR
Hi Daniel, Do you still have problem with Live Migration? You can check this update e8f51e9 where some errors related with Live Migration have been fixed.
The Readme states that migration is supported. Trying to do that in XenServer 7.1 with Ceph jewel is not working. XenCenter reports: You attempted to migrate a VDI to or from an SR that doesn't support migration I don't see any errors in SMlog or xensource.log Tracking that issue down to the xapi script that handles migration its about the capabilities listed in RBDSR.py Missing capabilities compared to NFS: SR_CACHING, VDI_GENERATE_CONFIG, VDI_MIRROR and VDI_RESET_ON_BOOT/2 but because it is not missing any capabilities compared to ISCSISR that can not be the reason. Hopefully you can resolve that issue,
Thank you in advance
Daniel