Can't Create more than 1 NFS per satellite

Smithx10 commented 2 years ago

It appears that in the create function here: https://github.com/LINBIT/linstor-gateway/blob/master/pkg/nfs/nfs.go#L60 we may be erroneously checking for duplicates. (depending the intention) I've added comments to the below code block.

    for _, c := range configs {
               // We check if the incoming create matches any of the configs that we already have.  If they match continue.
        if c.ID == rsc.ID() {
            continue
        }
        for _, r := range c.Resources {
            for _, s := range r.Start {
                if agent, ok := s.(*reactor.ResourceAgent); ok {
                                        //  We are definitely a new incoming configuration at this point... and are going to be an NFS server so why
                                        // error?  Do we only want 1 export per machine here?  I've taken out this line and had more than 1 with 
                                        // normal behavior.             
                    if agent.Type == "ocf:heartbeat:nfsserver" {
                        return nil, fmt.Errorf("an NFS config with a different ID already exists: %s", c.ID)
                    }
                }
            }
        }
    }

[root@drbd-lsc-0 ~]# linstor-gateway nfs create foo 10.91.197.28/32 2G --resource-group=nfs_group
Created export 'foo' at 10.91.197.28:/srv/gateway-exports/foo
[root@drbd-lsc-0 ~]# linstor-gateway nfs create bar 10.91.197.28/32 5G --resource-group=nfs_group
Error: failed to create nfs resource: an NFS config with a different ID already exists: nfs-foo
Usage:
  linstor-gateway nfs create NAME SERVICE_IP SIZE [flags]

Examples:
linstor-gateway nfs create example 192.168.211.122/24 2G
linstor-gateway nfs create restricted 10.10.22.44/16 2G --allowed-ips 10.10.0.0/16

Flags:
      --allowed-ips ip-cidr     Set the IP address mask of clients that are allowed access (default 0.0.0.0/0)
  -p, --export-path string      Set the export path, relative to /srv/gateway-exports (default "/")
  -h, --help                    help for create
  -r, --resource-group string   LINSTOR resource group to use (default "DfltRscGrp")

Global Flags:
      --config string     Config file to load (default "/etc/linstor-gateway/linstor-gateway.toml")
  -c, --connect string    LINSTOR Gateway server to connect to (default "http://localhost:8080")
      --loglevel string   Set the log level (as defined by logrus) (default "info")

failed to create nfs resource: an NFS config with a different ID already exists: nfs-foo

[root@drbd-lsc-0 ~]# lin sp l
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node                                                    ┊ Driver   ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ drbd-nfs-1.inst.bruce-dev.us-east-2.bdf-cloud.iqvia.net ┊ DISKLESS ┊          ┊              ┊               ┊ False        ┊ Ok    ┊            ┊
┊ DfltDisklessStorPool ┊ drbd-nfs-2.inst.bruce-dev.us-east-2.bdf-cloud.iqvia.net ┊ DISKLESS ┊          ┊              ┊               ┊ False        ┊ Ok    ┊            ┊
┊ DfltDisklessStorPool ┊ drbd-nfs-3.inst.bruce-dev.us-east-2.bdf-cloud.iqvia.net ┊ DISKLESS ┊          ┊              ┊               ┊ False        ┊ Ok    ┊            ┊
┊ lvmpool              ┊ drbd-nfs-1.inst.bruce-dev.us-east-2.bdf-cloud.iqvia.net ┊ LVM      ┊ lvpool   ┊    47.93 GiB ┊     50.00 GiB ┊ False        ┊ Ok    ┊            ┊
┊ lvmpool              ┊ drbd-nfs-2.inst.bruce-dev.us-east-2.bdf-cloud.iqvia.net ┊ LVM      ┊ lvpool   ┊    47.93 GiB ┊     50.00 GiB ┊ False        ┊ Ok    ┊            ┊
┊ lvmpool              ┊ drbd-nfs-3.inst.bruce-dev.us-east-2.bdf-cloud.iqvia.net ┊ LVM      ┊ lvpool   ┊    47.93 GiB ┊     50.00 GiB ┊ False        ┊ Ok    ┊            ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

[root@drbd-lsc-0 ~]# lin rg l
╭────────────────────────────────────────────────────────────────╮
┊ ResourceGroup ┊ SelectFilter            ┊ VlmNrs ┊ Description ┊
╞════════════════════════════════════════════════════════════════╡
┊ DfltRscGrp    ┊ PlaceCount: 2           ┊        ┊             ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ nfs_group     ┊ PlaceCount: 3           ┊ 0      ┊             ┊
┊               ┊ StoragePool(s): lvmpool ┊        ┊             ┊
╰────────────────────────────────────────────────────────────────╯

chrboe commented 2 years ago

This is actually intentional.

The issue is that an NFS resource in the context of LINSTOR Gateway maps to an ocf:heartbeat:nfsserver resource agent, and that can only be started once per node.

You may now ask yourself why we cannot just implement a "volume add" command, like we do for the other technologies, and have multiple NFS exports that way.

Unfortunately, that is also not easily possible because of the way LINSTOR treats new volumes. You don't want to know the details (trust me :smile:), but essentially it is not trivial to execute an mkfs -- which we need for NFS -- right after a new volume was added to a DRBD resource.

So, the intended way of having multiple NFS exports is to just create subdirectories and mount those. For example:

(on some cluster node)

[root@gateway-30 ~]# linstor-gateway nfs create example 192.168.122.222/24 1G
Created export 'example' at 192.168.122.222:/srv/gateway-exports/example

(on the node where the NFS server was created)

[root@gateway-31 ~]# mkdir /srv/gateway-exports/example/test{1,2}

(on the NFS client)

# mount -t nfs 192.168.122.222:/srv/gateway-exports/example/test1 /mnt/mynfs/

Of course, your bug report is still absolutely valid because we completely failed to document this. I will make sure to highlight this design decision in the documentation.

Smithx10 commented 2 years ago

One of the use cases we had was for multi-tenancy for our Cloud. We'd like to have ACL per export.

I don't think we can use the NFC ACL with subdir right?

We enjoy Linstor-Gateway for making it simple to configure Reactor :)

To confirm it's safe to have multiple exports and this is just a problem with how to provision and configure the NFS exports?

If / Until Linstor is patched in order to execute the mkfs predictably we will have to manually do what the Linstor Gateway is doing?

chrboe commented 1 year ago

I just stumbled upon this again; sorry for the delay here.

Yes, it is safe to have multiple exports in principle. The problem is in automatically creating the file system using LINSTOR. If you were to create the file systems yourself, there should be no issue.

I actually suspect setting ACLs on subdirectories works just fine, but I have not personally tried it.

I will close this issue for now since it is not an "issue" with LINSTOR Gateway per se (though I recognize that the current behavior is confusing). We are working on making this clearer, at least through documentation...

Smithx10 commented 1 year ago

Just a note....

Luckily for us, we are deploying the client protocols over a switch fabric that we manage.

We decided to just create a larger cidr block that is non routable network, and just use the address space as needed.

LINBIT / linstor-gateway

Can't Create more than 1 NFS per satellite #9