search

LINBIT / linstor-gateway

Manages Highly-Available iSCSI targets, NVMe-oF targets, and NFS exports via LINSTOR
GNU General Public License v3.0
28 stars 6 forks source link

Unable to finalize NFS share creation #5

Closed aellert closed 2 years ago

aellert commented 2 years ago

Hi,

I have a test cluster with 3 nodes (+1 dedicated node for the controller):

linstor n l
╭─────────────────────────────────────────────────────╮
┊ Node ┊ NodeType  ┊ Addresses               ┊ State  ┊
╞═════════════════════════════════════════════════════╡
┊ lin1 ┊ SATELLITE ┊ 10.16.0.41:3366 (PLAIN) ┊ Online ┊
┊ lin2 ┊ SATELLITE ┊ 10.16.0.42:3366 (PLAIN) ┊ Online ┊
┊ lin3 ┊ SATELLITE ┊ 10.16.0.43:3366 (PLAIN) ┊ Online ┊
╰─────────────────────────────────────────────────────╯
linstor sp l
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node ┊ Driver   ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName ┊
╞══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ lin1 ┊ DISKLESS ┊          ┊              ┊               ┊ False        ┊ Ok    ┊            ┊
┊ DfltDisklessStorPool ┊ lin2 ┊ DISKLESS ┊          ┊              ┊               ┊ False        ┊ Ok    ┊            ┊
┊ DfltDisklessStorPool ┊ lin3 ┊ DISKLESS ┊          ┊              ┊               ┊ False        ┊ Ok    ┊            ┊
┊ pool_ssd             ┊ lin1 ┊ ZFS      ┊ DataPool ┊     1.32 GiB ┊      2.81 GiB ┊ True         ┊ Ok    ┊            ┊
┊ pool_ssd             ┊ lin2 ┊ ZFS      ┊ DataPool ┊     1.84 GiB ┊      2.81 GiB ┊ True         ┊ Ok    ┊            ┊
┊ pool_ssd             ┊ lin3 ┊ ZFS      ┊ DataPool ┊     1.15 GiB ┊      2.81 GiB ┊ True         ┊ Ok    ┊            ┊
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
linstor rg l
╭─────────────────────────────────────────────────────────────────╮
┊ ResourceGroup ┊ SelectFilter             ┊ VlmNrs ┊ Description ┊
╞═════════════════════════════════════════════════════════════════╡
┊ DfltRscGrp    ┊ PlaceCount: 2            ┊        ┊             ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ my_ssd_group  ┊ PlaceCount: 2            ┊ 0      ┊             ┊
┊               ┊ StoragePool(s): pool_ssd ┊        ┊             ┊
╰─────────────────────────────────────────────────────────────────╯

And when I create an NFS share, it doesn't complete:

linstor-gateway --loglevel debug nfs create --resource=nfstest1 --service-ip=10.16.0.45/16 --allowed-ips=10.0.0.0/8 --resource-group=my_ssd_group --size=150M
DEBU[0000] curl -X 'GET' -H 'Accept: application/json' 'http://localhost:3370/v1/files?content=true&limit=0&offset=0'
DEBU[0000] {"name":"my_ssd_group","select_filter":{}}
DEBU[0000] curl -X 'POST' -d '{"name":"my_ssd_group","select_filter":{}}
' -H 'Accept: application/json' -H 'Content-Type: application/json' 'http://localhost:3370/v1/resource-groups'
DEBU[0000] Status code not within 200 to 400, but 500 (Internal Server Error)
DEBU[0000] {"resource_definition":{"name":"nfstest1","props":{"FileSystem/Type":"ext4"},"resource_group_name":"my_ssd_group"}}
DEBU[0000] curl -X 'POST' -d '{"resource_definition":{"name":"nfstest1","props":{"FileSystem/Type":"ext4"},"resource_group_name":"my_ssd_group"}}
' -H 'Accept: application/json' -H 'Content-Type: application/json' 'http://localhost:3370/v1/resource-definitions'
DEBU[0000] {"volume_definition":{"volume_number":1,"size_kib":153600}}
DEBU[0000] curl -X 'POST' -d '{"volume_definition":{"volume_number":1,"size_kib":153600}}
' -H 'Accept: application/json' -H 'Content-Type: application/json' 'http://localhost:3370/v1/resource-definitions/nfstest1/volume-definitions'
DEBU[0000] {"select_filter":{}}
DEBU[0000] curl -X 'POST' -d '{"select_filter":{}}
' -H 'Accept: application/json' -H 'Content-Type: application/json' 'http://localhost:3370/v1/resource-definitions/nfstest1/autoplace'
DEBU[0003] curl -X 'GET' -H 'Accept: application/json' 'http://localhost:3370/v1/resource-definitions/nfstest1'
DEBU[0003] curl -X 'GET' -H 'Accept: application/json' 'http://localhost:3370/v1/view/resources?limit=0&offset=0&resources=nfstest1'
DEBU[0003] {"path":"/etc/drbd-reactor.d/linstor-gateway-nfs-nfstest1.toml","content":"W1twcm9tb3Rlcl1dCiAgaWQgPSAibmZzLW5mc3Rlc3QxIgogIFtwcm9tb3Rlci5yZXNvdXJjZXNdCiAgICBbcHJvbW90ZXIucmVzb3VyY2VzLm5mc3Rlc3QxXQogICAgICBzdGFydCA9IFsib2NmOmhlYXJ0YmVhdDpGaWxlc3lzdGVtIGZzXzEgZGV2aWNlPS9kZXYvZHJiZC9ieS1yZXMvbmZzdGVzdDEvMSBkaXJlY3Rvcnk9L3Nydi9nYXRld2F5LWV4cG9ydHMvbmZzdGVzdDEgZnN0eXBlPWV4dDQgcnVuX2ZzY2s9bm8iLCAib2NmOmhlYXJ0YmVhdDpleHBvcnRmcyBleHBvcnRfMV8wIGNsaWVudHNwZWM9MTAuMC4wLjAvOCBkaXJlY3Rvcnk9L3Nydi9nYXRld2F5LWV4cG9ydHMvbmZzdGVzdDEgZnNpZD1iZmYzYmQ5MS02Mzc3LTU2YTYtYjcwMi04ODY3MGFjOWYxYmMgb3B0aW9ucz1ydyB3YWl0X2Zvcl9sZWFzZXRpbWVfb25fc3RvcD0xIiwgIm9jZjpoZWFydGJlYXQ6SVBhZGRyMiBzZXJ2aWNlX2lwIGNpZHJfbmV0bWFzaz0xNiBpcD0xMC4xNi4wLjQ1Il0KICAgICAgcnVubmVyID0gInN5c3RlbWQiCiAgICAgIG9uLXN0b3AtZmFpbHVyZSA9ICJlY2hvIGIgPiAvcHJvYy9zeXNycS10cmlnZ2VyIgogICAgICBzdG9wLXNlcnZpY2VzLW9uLWV4aXQgPSB0cnVlCiAgICAgIHRhcmdldC1hcyA9ICJCaW5kc1RvIgo="}
DEBU[0003] curl -X 'PUT' -d '{"path":"/etc/drbd-reactor.d/linstor-gateway-nfs-nfstest1.toml","content":"W1twcm9tb3Rlcl1dCiAgaWQgPSAibmZzLW5mc3Rlc3QxIgogIFtwcm9tb3Rlci5yZXNvdXJjZXNdCiAgICBbcHJvbW90ZXIucmVzb3VyY2VzLm5mc3Rlc3QxXQogICAgICBzdGFydCA9IFsib2NmOmhlYXJ0YmVhdDpGaWxlc3lzdGVtIGZzXzEgZGV2aWNlPS9kZXYvZHJiZC9ieS1yZXMvbmZzdGVzdDEvMSBkaXJlY3Rvcnk9L3Nydi9nYXRld2F5LWV4cG9ydHMvbmZzdGVzdDEgZnN0eXBlPWV4dDQgcnVuX2ZzY2s9bm8iLCAib2NmOmhlYXJ0YmVhdDpleHBvcnRmcyBleHBvcnRfMV8wIGNsaWVudHNwZWM9MTAuMC4wLjAvOCBkaXJlY3Rvcnk9L3Nydi9nYXRld2F5LWV4cG9ydHMvbmZzdGVzdDEgZnNpZD1iZmYzYmQ5MS02Mzc3LTU2YTYtYjcwMi04ODY3MGFjOWYxYmMgb3B0aW9ucz1ydyB3YWl0X2Zvcl9sZWFzZXRpbWVfb25fc3RvcD0xIiwgIm9jZjpoZWFydGJlYXQ6SVBhZGRyMiBzZXJ2aWNlX2lwIGNpZHJfbmV0bWFzaz0xNiBpcD0xMC4xNi4wLjQ1Il0KICAgICAgcnVubmVyID0gInN5c3RlbWQiCiAgICAgIG9uLXN0b3AtZmFpbHVyZSA9ICJlY2hvIGIgPiAvcHJvYy9zeXNycS10cmlnZ2VyIgogICAgICBzdG9wLXNlcnZpY2VzLW9uLWV4aXQgPSB0cnVlCiAgICAgIHRhcmdldC1hcyA9ICJCaW5kc1RvIgo="}
' -H 'Accept: application/json' -H 'Content-Type: application/json' 'http://localhost:3370/v1/files/%2Fetc%2Fdrbd-reactor.d%2Flinstor-gateway-nfs-nfstest1.toml'
DEBU[0004] curl -X 'GET' -H 'Accept: application/json' 'http://localhost:3370/v1/files?content=true&limit=0&offset=0'
DEBU[0004] curl -X 'POST' -H 'Accept: application/json' 'http://localhost:3370/v1/resource-definitions/nfstest1/files/%2Fetc%2Fdrbd-reactor.d%2Flinstor-gateway-nfs-nfstest1.toml'
DEBU[0004] curl -X 'GET' -H 'Accept: application/json' 'http://localhost:3370/v1/view/resources?limit=0&offset=0&resources=nfstest1'
DEBU[0007] curl -X 'GET' -H 'Accept: application/json' 'http://localhost:3370/v1/view/resources?limit=0&offset=0&resources=nfstest1'
DEBU[0010] curl -X 'GET' -H 'Accept: application/json' 'http://localhost:3370/v1/view/resources?limit=0&offset=0&resources=nfstest1'
DEBU[0013] curl -X 'GET' -H 'Accept: application/json' 'http://localhost:3370/v1/view/resources?limit=0&offset=0&resources=nfstest1'
DEBU[0016] curl -X 'GET' -H 'Accept: application/json' 'http://localhost:3370/v1/view/resources?limit=0&offset=0&resources=nfstest1'
DEBU[0019] curl -X 'GET' -H 'Accept: application/json' 'http://localhost:3370/v1/view/resources?limit=0&offset=0&resources=nfstest1'
DEBU[0022] curl -X 'GET' -H 'Accept: application/json' 'http://localhost:3370/v1/view/resources?limit=0&offset=0&resources=nfstest1'
DEBU[0025] curl -X 'GET' -H 'Accept: application/json' 'http://localhost:3370/v1/view/resources?limit=0&offset=0&resources=nfstest1'
DEBU[0028] curl -X 'GET' -H 'Accept: application/json' 'http://localhost:3370/v1/view/resources?limit=0&offset=0&resources=nfstest1'
DEBU[0031] curl -X 'GET' -H 'Accept: application/json' 'http://localhost:3370/v1/view/resources?limit=0&offset=0&resources=nfstest1'
Error: failed to start resources: error waiting for resource to become used: context deadline exceeded
Usage:
  linstor-gateway nfs create [flags]

Examples:
linstor-gateway nfs create --resource=example --service-ip=192.168.211.122/24 --allowed-ips=192.168.0.0/16 --resource-group=ssd_thin_2way --size=2G

Flags:
      --allowed-ips ip-cidr     Set the IP address mask of clients that are allowed access (default ::1/64)
  -p, --export-path string      Set the export path (default "/")
  -h, --help                    help for create
  -r, --resource string         Set the resource name (required)
  -g, --resource-group string   Set the LINSTOR resource group name
      --service-ip ip-cidr      Set the service IP and netmask of the target (required) (default ::1/64)
      --size unit               Set a size (e.g, 1TiB) (default 1GiB)

Global Flags:
      --config string         Config file to load (default "/etc/linstor-gateway/linstor-gateway.toml")
      --controllers strings   List of LINSTOR controllers to try to connect to (default from $LS_CONTROLLERS, or localhost:3370)
      --loglevel string       Set the log level (as defined by logrus) (default "info")

failed to start resources: error waiting for resource to become used: context deadline exceeded

I finally get it working after rebooting my nodes or restart drbd-reactor service.

linstor-gateway nfs list
+----------+---------------+---------------+-------------------------------+---------------+
| Resource |  Service IP   | Service state |          NFS export           | LINSTOR state |
+----------+---------------+---------------+-------------------------------+---------------+
| nfstest1 | 10.16.0.45/16 | Started       | /srv/gateway-exports/nfstest1 | OK            |
| nfstest2 | 10.16.0.46/16 | Started       | /srv/gateway-exports/nfstest2 | OK            |
+----------+---------------+---------------+-------------------------------+---------------+

Can you help me debug ?

Thank you

aellert commented 2 years ago

Reply to myself, after better understanding of drbd-reactor, I simply enable automatic reload. On each of my nodes (Ubuntu 20.04) :

cp /usr/share/doc/drbd-reactor/examples/drbd-reactor-reload.* /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now drbd-reactor-reload.path

And everything if working fine :)

rck commented 2 years ago

yes, I implemented automatic reloading in drbd-reactor specifically for gateway/vSAN, so this might be a documentation issue. I will re-open this till @chrboe is back in action and can act on it.

chrboe commented 2 years ago

Yeah, this is a documentation issue. The README already says:

The daemon should be configured to reload automatically when the configuration changes.

, but I agree that we could very easily document this more clearly.

chrboe commented 2 years ago

Improved in bfe6dd006110658acd54edf524fc39250199a92b. I hope that is clearer now.