Closed mhoffmann75 closed 8 years ago
Yes, this is the issue I was talking about and it puts VM in bad state. Thanks for documenting. I'll create a xenserver7 branch because (my first impression from yesterday) it seems SR+RBDSR are both adding the paused state which crash the snapshot process. I'm not going to test on xenserver<7 and let other people check if the xs7 branch will be compatible with xs<7.
Hi, This error occurs because we unmap image before making snapshot due to problem, described in issue #4. If we have only one host it works fine. But if we have more then one host the error occurs as the image mapped on host2 but snapshot operation is executed by pool master i.e. host1 and try to unmap image that already unmapped on host1. We could do not unmap the image before making snapshot but if we use nbd and try to make a snapshot the nbd device hang. So we should find a means how to unmap the image on the right host.
As workaround you can use FUSE mode as it doesn't use map/unmap
Hi, I was not aware that pool master is actually taking the snapshots - but sounds logical. Do you know if it's necessary to map/unmap on host2 (VM running on host2) when host1 is pool master and creating the snapshot? If not necessary then it would be enough for RDBSR to check for existence of the /dev/nbd/xxx device before unmapping. If it's also hanging the nbd driver on host2 then we need a mechanism for RBDSR on pool master to contact RBDSR on the pool member(s). Makes it quite complex i guess.
Same situation for VDI cloning?
BTW: fuse mode is broken in latest build, too:
['rbd-fuse', '-p', 'RBD_XenStorage-ff12160f-ff09-40bb-a874-1366ad907f44', '/run/sr-mount/ff12160f-ff09-40bb-a874-1366ad907f44', '--name', 'client.admin'] FAILED in util.pread: (rc 1) stdout: '', stderr: 'fuse: invalid argument `client.admin'
Seems as if rdb-fuse doesn't like: --name client.admin
Pull in from my fork to fix.
I added the fallback to kernel mode if fuse mode and cephx is used (commit 6e5ed2d7da5a470a1aea8dad7f33596eb00c9408)
By the way: why unmap + remap if blktap is paused + unpaused? Isn't it double "paused" of the FS ?
because mapped nbd device hangs if you try to make snapshot of the image i don't know why it hangs but if you unmap before snapshot it doesn't hang
Has anyone actually tested if nbd does hang too on a mapped volume if snapshot is done on another host where the volume ist NOT mapped?
Issue has been fixed.
copy ceph_plugin.py to /etc/xapi.d/plugins on each host in pool
rename to ceph_plugin and make it executable
It now calls rbd-nbd map/unmap
on the right host
Please test it
Just tested it - look very promising. VM Snapshot and snapshot deletion always worked. Regardless of host running on or moving to another host. So i guess you really fixed this. Well done!
Ok. Thank you Martin. I'm going to close the issue.
Thanks for all the fixes but on stress testing RBDSR with XenServer 7 i think i found just another misbeaviour. There seems to be a bug when creating/deleting snapshots close after moving a VM from one host to another (XenServer pool with RBDSR Shared Storage)
This way you can reproduce the problem (latest commit 31):
Logs from host1 (The VM is already running on host2 !!!)
After that VM hangs and cannot be shutdown - host2 endlessly repeating:
Similar problem seem to occur on snapshot removal:
Most of the time VM hangs and also host must be rebooted in order to fix this situation.
If you need i will be able to provide logs while problem occurs.
So whats going on here? Why does the old host try to do SR stuff and what can be done in RBDSR to fix this?