Request to add multipath support

[fabiand]

Currently libiscsi enables multipath support by allowing multiple connections (in the sense of initiators) to the same portal/lun. In the case of qemu this means that a OS within a VM could reassemble the LUNs back to a single LUN. But this depends on the OS support.

If libiscsi was able to aggregate multiple connections into a single LUN again, and provide this LUN to the application, then the multipath support would be transparent to (in the qemu case) a OS.

The benefit of such a change would be that a VM could bypass the host's iscsi stack - and would even have mpath support.

[sahlberg]

I was considering adding this at one point, in old branch epoll, but decided against it.

One of the problems with adding multi-path support is that it would require that we would suddenly start listening for multiple sockets. The only way to do this, without breaking the get_fds/which_events_service API, would be to use something like epoll under linux and the equivalent of it on the other platforms. But that would make libiscsi a lot less portable.

I also do not want to make libiscsi too intelligent. I want to keep it as close to being just a simple pipe rather than a full blown initiator.

For the case of VMs, I think a better solution would be to just set up two separate libiscsi connections and present them to the guest kernel. Then let the guest kernel manage the multipathing.

On Mon, May 8, 2017 at 5:22 AM, Fabian Deutsch notifications@github.com wrote:

Currently libiscsi enables multipath support by allowing multiple connections (in the sense of initiators) to the same portal/lun. In the case of qemu this means that a OS within a VM could reassemble the LUNs back to a single LUN. But this depends on the OS support.

If libiscsi was able to aggregate multiple connections into a single LUN again, and provide this LUN to the application, then the multipath support would be transparent to (in the qemu case) a OS.

The benefit of such a change would be that a VM could bypass the host's iscsi stack - and would even have mpath support.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/sahlberg/libiscsi/issues/246, or mute the thread https://github.com/notifications/unsubscribe-auth/AAeNkE2V7oBrwNR8VNN9JWt3angpU32Bks5r3wj2gaJpZM4NT34w .

[fabiand]

Thanks for the reply.

Yes, the two connections and let the guest OS handle it is actually something I got on my list to try. But this goes and falls with the guest OS support. Having the multipath on the hypervisor side would make it transparent to the os - IWO it would not require OS support for multipath.

I can understand your concerns on portability - and got nothing to reply on that :) To some degree however I'd see libiscsi as the right place. But maybe then it's rather qemu to solve this.

[sitsofe]

@fabiand Perhaps you could build a new project called multipath-libiscsi library that would take care of the multipathing using different libiscsi connections? The problem is multipathing is a big job in and of itself because you have to control whether you really have more than one path to the same device, when you timeout, when you fail, how you cope with resets, what you do when multiple paths are available (e.g. do you read ALUA information or use another means to get path weightings?), how you failover, whether you failback, what to do when you have no paths etc.

[sahlberg]

On Thu, May 18, 2017 at 1:19 PM, Sitsofe Wheeler notifications@github.com wrote:

@fabiand https://github.com/fabiand Perhaps you could build a new project called multipath-libiscsi library that would take care of the multipathing using different libiscsi connections? The problem is multipathing is a big job in and of itself because you have to control whether you really have more than one path to the same device, when you timeout, when you fail, how you cope with resets, what you do when multiple paths are available (e.g. do you read ALUA information or use another means to get path weightings?), how you failover, whether you failback, what to do when you have no paths etc.

multipath-libiscsi would only solve it for libiscsi devices. If you can do multipathing for libiscsi, I imagine that at some stage you would also want to do the same for libnfs, then for libsmb2 and all other remote storage backends that qemu support.

Because of this I think for QEMU the better options would be either to 1, just expose multiple devices to the guest kernel and have it handle multipathing, or 2, add multipath support to the qemu block layer itself and expose this as a single device to the guest.

1, should work already today for iscsi. For NFS I think you might need some small changes to be able to convert a nfs inode/devicenumber into something that resembles a scsi unit serial number so that the guest kernel can identify which devices are mulipath components. 2, would require another layer in the block stack that would sit above the iscsi/nfs/... drivers but it would add multipath support for every backend.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/sahlberg/libiscsi/issues/246#issuecomment-302529571, or mute the thread https://github.com/notifications/unsubscribe-auth/AAeNkLyFxqYl9YSFIqo9iXJ-JupN8STbks5r7Ke-gaJpZM4NT34w .

[fabiand]

Ins't multipathing - or multiple IPs per session - a iSCSI specific thing?

[plieven]

Its more a SCSI specific thing. Its quite common for FC devices as well.

Let me think some days about how complicated it is to build it into the Qemu iSCSI driver.

[fabiand]

Yes - I also see that the implementation might be pretty redundant to what multipathd does - i actually wonder if multipathd could be reused to some degree, after all it's pretty much in user-space.

OTOH I've never looked at the code ..

[plieven]

multipathd is designed to work with kernel block devices. it more or less reprogramms the device mapper. i don't think that it might be of great use.

as for qemu iscsi code most of the code was done by ronnie and me, so thats not the problem. the biggest issue i see at the moment is to get the right PDUs of the queues when we decide that a path has failed and put it onto another connection. and as for the iscsi standard there are a few things that might have to be implemented when we have more than one connection to the same target from the same initiator. if i remember correctly there are some async messages to control that.

Peter

Am 19.05.2017 um 11:29 schrieb Fabian Deutsch:

Yes - I also see that the implementation might be pretty redundant to what multipathd does - i actually wonder if multipathd could be reused to some degree, after all it's pretty much in user-space.

OTOH I've never looked at the code ..

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/sahlberg/libiscsi/issues/246#issuecomment-302655498, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlQnWmkJ0Uo1rwo1X3Z0qTL_chAzemyks5r7WERgaJpZM4NT34w.

[fabiand]

Alright, I see - thanks for the clarification.