search

pkoutoupis / rapiddisk

An Advanced Linux RAM Drive and Caching kernel modules. Dynamically allocate RAM as block devices. Use them as stand alone drives or even map them as caching nodes to slower local disk drives. Access those volumes locally or export them across an NVMe Target network. Manage it all from a web API.
http://www.rapiddisk.org
GNU General Public License v2.0
296 stars 49 forks source link

Separate NVMe initiator host management from target management #148

Open pkoutoupis opened 1 year ago

pkoutoupis commented 1 year ago

@matteotenca writes in https://github.com/pkoutoupis/rapiddisk/pull/145:

In my very humble opinion, the exporting/unexporting process should be handled apart from the process of choosing allowed hosts for an export. I don't know if it is mandatory to specify the allowed hosts upon exporting, though.

I say this because the unexport function is quite tricky just because it has to manage the allowed host list too: when more than one host is present, the function behaviour switches to a bare handling of the access list, since it cannot unexport until the allowed hosts list is empty.

My opinion is the -e and -x should just trigger exporting and unexporting, and that other two switches should be added to choose allowed hosts for an exported target, like this fictional example:

$ sudo rapiddisk -e -b rd0 -P 1
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

Block device rd0 has been mapped through port 1 as nqn.2021-06.org.rapiddisk:ubuserver-rd0.
$ sudo rapiddisk -A nqn.2021-06.org.rapiddisk:ubuserver-rd0 -H hostone
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

Host 'hostone' is now allowed to access nqn.2021-06.org.rapiddisk:ubuserver-rd0.
$ sudo rapiddisk -D nqn.2021-06.org.rapiddisk:ubuserver-rd0 -H hostone
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

Host 'hostone' is no more allowed to access nqn.2021-06.org.rapiddisk:ubuserver-rd0.
$
pkoutoupis commented 1 year ago

Additional notes

@matteotenca wrote:

there is an issue with RDMA ports. When a RDMA port is created, the exporting operations fail, and the environment becomes "dirty". Since exporting and unexporting fails in different steps, the kernel filesystem situation ends up to be messed up. I believe that export/unexport functions should be modified so that they can undo the operations already carried on upon failure... :-\

Look at this sequence of commands:

shub@ubuserver:~/rapiddisk/src$ sudo ./rapiddisk_debug -n
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

NVMe Target Exports

        None.

Exported NVMe Ports

        None.

shub@ubuserver:~/rapiddisk/src$ sudo ./rapiddisk_debug -N
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

Exported NVMe Ports

        None.

shub@ubuserver:~/rapiddisk/src$ sudo ./rapiddisk_debug -i enp0s3 -P 1 -t rdma
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

Successfully created port 1 set to rdma for interface enp0s3 (with IP address 192.168.20.40).
shub@ubuserver:~/rapiddisk/src$ sudo ./rapiddisk_debug -N
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

Exported NVMe Ports

        1: Port: 1 - 192.168.20.40 (rdma)
shub@ubuserver:~/rapiddisk/src$ sudo ./rapiddisk_debug -n
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

NVMe Target Exports

        None.

Exported NVMe Ports

        None.

shub@ubuserver:~/rapiddisk/src$ sudo ./rapiddisk_debug -e -b rd0 -P 1 -H test
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

Error. Unable to create link of NQN to port. nvmet_export_volume: symlink: No such device
shub@ubuserver:~/rapiddisk/src$ sudo ./rapiddisk_debug -n
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

NVMe Target Exports

        1: NQN: nqn.2021-06.org.rapiddisk:ubuserver-rd0         Namespace: 1    Device: /dev/rd0        Enabled: True

Exported NVMe Ports

        None.

shub@ubuserver:~/rapiddisk/src$ sudo ./rapiddisk_debug -N
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

Exported NVMe Ports

        1: Port: 1 - 192.168.20.40 (rdma)
shub@ubuserver:~/rapiddisk/src$ sudo ./rapiddisk_debug -x -b rd0 -P 1 -H test
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

Block device rd0 has been unmapped from NVMe Initiator host test.
nvmet_unexport_volume: Port 1 and / or export does not exist.
shub@ubuserver:~/rapiddisk/src$ sudo ./rapiddisk_debug -n
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

NVMe Target Exports

        1: NQN: nqn.2021-06.org.rapiddisk:ubuserver-rd0         Namespace: 1    Device: /dev/rd0        Enabled: True

Exported NVMe Ports

        None.

shub@ubuserver:~/rapiddisk/src$ sudo ./rapiddisk_debug -N
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

Exported NVMe Ports

        1: Port: 1 - 192.168.20.40 (rdma)
shub@ubuserver:~/rapiddisk/src$ sudo ./rapiddisk_debug -x -b rd0 -P 1 -H test
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

NVMe Initiator Host 'test' does not exist.
shub@ubuserver:~/rapiddisk/src$ sudo ./rapiddisk_debug -x -b rd0 -P 1
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

nvmet_unexport_volume: Port 1 and / or export does not exist.
shub@ubuserver:~/rapiddisk/src$ sudo ./rapiddisk_debug -n
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

NVMe Target Exports

        1: NQN: nqn.2021-06.org.rapiddisk:ubuserver-rd0         Namespace: 1    Device: /dev/rd0        Enabled: True

Exported NVMe Ports

        None.

shub@ubuserver:~/rapiddisk/src$ sudo ./rapiddisk_debug -N
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

Exported NVMe Ports

        1: Port: 1 - 192.168.20.40 (rdma)
shub@ubuserver:~/rapiddisk/src$ sudo ./rapiddisk_debug -X -P 1
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

NVMe Target port 1 has been removed.
shub@ubuserver:~/rapiddisk/src$ sudo ./rapiddisk_debug -n
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

NVMe Target Exports

        1: NQN: nqn.2021-06.org.rapiddisk:ubuserver-rd0         Namespace: 1    Device: /dev/rd0        Enabled: True

Exported NVMe Ports

        None.

shub@ubuserver:~/rapiddisk/src$ sudo ./rapiddisk_debug -N
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

Exported NVMe Ports

        None.

shub@ubuserver:~/rapiddisk/src$

My reply:

As with the second part of your comment:
It looks like the error is a result of the port not being configured for RDMA. dmesg states the following:

[ 3664.472868] nvmet_rdma: binding CM ID to 10.0.0.185:4420 failed (-19)
Which makes sense. In order to use RDMA over standard ethernet, you need to configure soft RoCE. Years ago, I wrote an article for Linux Journal to do exactly that: https://www.linuxjournal.com/content/data-flash-part-ii-using-nvme-drives-and-creating-nvme-over-fabrics-network. Anyway, it seems like the process has changed since and instead of typing:

# rxe_cfg add eth0
You need to type:

# rdma link add rxe_eth0 type rxe netdev eth0
I just tested it and it worked for RDMA without error.

petros@dev-machine:~/rapiddisk/src$ sudo ./rapiddisk -n
rapiddisk 9.0.0
Copyright 2011 - 2023 Petros Koutoupis

NVMe Target Exports

    1: NQN: nqn.2021-06.org.rapiddisk:dev-machine-rd0   Namespace: 1    Device: /dev/rd0    Enabled: True

Exported NVMe Ports

    1: Port: 2 - 10.0.0.185 (tcp)   NQN: nqn.2021-06.org.rapiddisk:dev-machine-rd0
    2: Port: 1 - 10.0.0.185 (rdma)  NQN: nqn.2021-06.org.rapiddisk:dev-machine-rd0
So, to get back to your original point:
This will be better addressed in the NVMe target subsystem rework designated for https://github.com/pkoutoupis/rapiddisk/issues/148. I will add these notes there. Thank you for testing this.