search

siderolabs / extensions

Talos Linux System Extensions
Mozilla Public License 2.0
104 stars 105 forks source link

iscsi-tools not binding some mounts into filesystem #38

Closed zebernst closed 2 years ago

zebernst commented 2 years ago

Hello!

My situation is almost identical to #33 - I've got a cluster of raspberry pi 4's and a Synology NAS that I'm trying to use as a storage backend for my PVCs, and I can't use NFS due to the limitations of file locking SQLite databases on network filesystems. I'm using the synology-csi driver to handle creation, deletion, and connection of iSCSI-backed LUNs on my NAS as PersistentVolumes in my cluster.

The initial setup went smoothly, with the driver able to create and delete PersistentVolumes and their backing LUNs with no issue on my NAS. I now suspect that it was accomplishing that via the Synology API and not via iSCSI, as I have not actually been able to mount the volume via iSCSI.

Here's my talosctl version:

❯ talosctl version
Client:
    Tag:         v1.0.0
    SHA:         80167fd2
    Built:       
    Go version:  go1.17.8
    OS/Arch:     darwin/amd64
Server:
    NODE:        192.168.4.9
    Tag:         v1.1.0-alpha.1-65-gafb679586
    SHA:         afb67958
    Built:       
    Go version:  go1.18.2
    OS/Arch:     linux/arm64
    Enabled:     RBAC

Originally, this was the event trace when trying to mount a volume to a pod:

❯ kubectl describe pod pvc-inspector
...
Events:
  Type     Reason                  Age                From                     Message
  ----     ------                  ----               ----                     -------
  Warning  FailedScheduling        86s                default-scheduler        0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
  Warning  FailedScheduling        85s                default-scheduler        0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
  Normal   Scheduled               83s                default-scheduler        Successfully assigned synology-csi/pvc-inspector to pi-sidero
  Normal   SuccessfulAttachVolume  83s                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-1969eb78-dbb4-43e1-9659-6b6bdff9167b"
  Warning  FailedMount             10s (x8 over 76s)  kubelet                  MountVolume.MountDevice failed for volume "pvc-1969eb78-dbb4-43e1-9659-6b6bdff9167b" : rpc error: code = Internal desc = rpc error: code = Internal desc = Failed to login with target iqn [iqn.2000-01.com.synology:<hostname>.pvc-1969eb78-dbb4-43e1-9659-6b6bdff9167b], err: chroot: can't execute '/usr/bin/env': No such file or directory
 (exit status 127)

I dug around and determined it was due to iscsiadm being aliased to this code snippet (see the Dockerfile): https://github.com/SynologyOpenSource/synology-csi/blob/fc3359223fe51a13bcfa5a7cabbf59611bbeb901/chroot/chroot.sh#L4-L9

It looks like synology-csi just mounts the whole host filesystem into /host (source), and uses chroot to emulate calling iscsiadm from the host environment. Since there's no /usr/bin/env in the Talos environment, the command fails. I changed it to directly link to the iscsiadm binary contained in the iscsi-tools extension like so:

exec chroot $DIR "/usr/local/sbin/$BIN" "$@"

That fixed the issue of not actually being able to run iscsiadm, but I'm now running into this error:

❯ kubectl describe pod pvc-inspector
...
Events:
  Type     Reason                  Age              From                     Message
  ----     ------                  ----             ----                     -------
  Warning  FailedScheduling        3m19s            default-scheduler        0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
  Warning  FailedScheduling        3m18s            default-scheduler        0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
  Warning  FailedScheduling        117s             default-scheduler        0/1 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
  Normal   Scheduled               11s              default-scheduler        Successfully assigned synology-csi/pvc-inspector to pi-sidero
  Normal   SuccessfulAttachVolume  11s              attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-4ad7b598-50a7-423e-84c5-d4f0c4edb083"
  Warning  FailedMount             1s (x2 over 2s)  kubelet                  MountVolume.MountDevice failed for volume "pvc-4ad7b598-50a7-423e-84c5-d4f0c4edb083" : rpc error: code = Internal desc = rpc error: code = Internal desc = Failed to login with target iqn [iqn.2000-01.com.synology:<hostname>.pvc-4ad7b598-50a7-423e-84c5-d4f0c4edb083], err: iscsiadm: Could not make /etc/iscsi/ 30
iscsiadm: exiting due to idbm configuration error
 (exit status 6)

From what I understand, the /etc/iscsi folder should be bound to /system/iscsi: https://github.com/siderolabs/extensions/blob/7cf2843cd73aa807d72151ec64bb0dc913492d63/storage/iscsi-tools/iscsid.yaml#L41-L47

The iscsi-tools extension seems to be working for the most part:

However, the /system/iscsi directory is not being bound correctly to /etc/iscsi:

❯ talosctl ls system
NODE          NAME
192.168.4.9   .
192.168.4.9   config
192.168.4.9   etc
192.168.4.9   iscsi
192.168.4.9   libexec
192.168.4.9   overlays
192.168.4.9   run
192.168.4.9   secrets
192.168.4.9   state
192.168.4.9   var

❯ talosctl ls etc
NODE          NAME
192.168.4.9   .
192.168.4.9   ca-certificates
192.168.4.9   cni
192.168.4.9   containerd
192.168.4.9   cri
192.168.4.9   extensions.yaml
192.168.4.9   hosts
192.168.4.9   kubernetes
192.168.4.9   localtime
192.168.4.9   lvm
192.168.4.9   machine-id
192.168.4.9   os-release
192.168.4.9   pki
192.168.4.9   resolv.conf
192.168.4.9   ssl
192.168.4.9   ssl1.1

Is there something I need to be doing in the machineconfig to get this directory bound as expected?

frezbo commented 2 years ago

yes, this is expected of any CSI drivers that does chroot or nsenter into the pid 1 namespace. These CSI drivers assume iscsid is running on the host, whereas when ran as a system extension it runs as a container (also the same reason you don't see /etc/iscsiwhen running talosctl ls /etc , /etc/iscsi is available from /system/extension for the iscsid process). So ideally the csi would have to execute the process in the pid namespace of the iscsi process. Since this is script based I believe we can modify the script to use nsenter instead of chroot and using the mount namespace of the iscsid process, something along the lines of

iscsid_pid=$(pgrep iscsid)

nsenter --mount="/proc/${iscsid_pid}/ns/mnt" --net="/proc/${iscsid_pid}/ns/net" -- /usr/local/sbin/iscsiadm "$@"

this requires the csi pod to have hostPID: true in the pod spec of the pod that runs this script

frezbo commented 2 years ago

Also any particular reason not to use the synology api as opposed to iscsi?

zebernst commented 2 years ago

Thanks for the quick reply @frezbo! That snippet fixed the iscsiadm call, but I'm still unable to mount a disk with the CSI plugin throwing the following error:

2022-05-12T21:05:52Z [INFO] [driver/utils.go:105] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-54b49ea5-1270-446d-88d9-4a511c08c363/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"btrfs"}},"access_mode":{"mode":1}},"volume_context":{"dsm":"<nas ip>","protocol":"iscsi","source":"","storage.kubernetes.io/csiProvisionerIdentity":"1652385377888-8081-csi.san.synology.com"},"volume_id":"415fcc44-b9a4-4b99-afe8-dee8c2971652"}
2022-05-12T21:05:52Z [ERROR] [driver/nodeserver.go:60] Can't find device path [/dev/disk/by-path/ip-<nas ip>:3260-iscsi-iqn.2000-01.com.synology:<hostname>.pvc-e00e9dc0-1a46-4a68-9a50-4ecdc7fbd5a8-lun-1], err: <nil>
2022-05-12T21:05:54Z [INFO] [driver/initiator.go:123] Login target portal [<nas ip>:3260], iqn [iqn.2000-01.com.synology:<hostname>.pvc-54b49ea5-1270-446d-88d9-4a511c08c363].
2022-05-12T21:05:54Z [ERROR] [driver/nodeserver.go:60] Can't find device path [/dev/disk/by-path/ip-<nas ip>:3260-iscsi-iqn.2000-01.com.synology:<hostname>.pvc-e00e9dc0-1a46-4a68-9a50-4ecdc7fbd5a8-lun-1], err: <nil>
.... [repeated about 60 times] ....
2022-05-12T21:06:37Z [ERROR] [driver/nodeserver.go:92] Volume mount path is not exist.
2022-05-12T21:06:37Z [ERROR] [driver/utils.go:108] GRPC error: rpc error: code = Internal desc = Can't get volume mount path

Since I can't see directly what iscsi commands are being called under the hood I'll keep digging and probably raise an issue over at the synology-csi project (unless this looks familiar to you?).

Also any particular reason not to use the synology api as opposed to iscsi?

Is that possible? I'm not dead-set on using iscsi but the synology-csi only offers iscsi or smb support for backing connections, and I was under the impression that storage over a network needed a protocol like iscsi. The creation of something like LUN volume seems like it fits into the purview of an API, but I didn't think that an API would be able to support the reads and writes of a filesystem.

frezbo commented 2 years ago

Is that possible? I'm not dead-set on using iscsi but the synology-csi only offers iscsi or smb support for backing connections, and I was under the impression that storage over a network needed a protocol like iscsi. The creation of something like LUN volume seems like it fits into the purview of an API, but I didn't think that an API would be able to support the reads and writes of a filesystem.

My apologies, I assumed the csi driver had an alternate mode of operation.

regarding the error I think it's due to the sysfs not being writable, once https://github.com/siderolabs/talos/pull/5550 is merged that should be easily done. I tested with a local patch of mine (i had forgot about that).

meanwhile can you upgrade the talos with this installer image: ghcr.io/frezbo/installer:v1.1.0-alpha.1-67-gcc4131c41

once that's done can you have talosctl dmesg --tail --follow on another terminal and provide the output after when the volume mount is happening via the CSI driver

zebernst commented 2 years ago

Using that installer image worked! The PVC mounted correctly and I verified that reading and writing works as expected. Thank you so much for your help @frezbo!

frezbo commented 2 years ago

@zebernst That's so awesome :muscle: Also now we know at least one csi can work with some changes. Would you mind sending over some notes or a PR as an example?

zebernst commented 2 years ago

Happy to - did you mean write up some notes for y'all or open a PR with synology-csi?

frezbo commented 2 years ago

i think just here or the talos docs are fine, since this is mosly talos specific.

zebernst commented 2 years ago

I can open a PR with a guide to setting up Synology CSI in the Talos v1.1 docs if that works? Let me know where you think this best belongs, this is your project 🙂

frezbo commented 2 years ago

https://www.talos.dev/v1.0/kubernetes-guides/configuration/ I believe this works https://github.com/siderolabs/talos/tree/master/website/content/v1.1/kubernetes-guides/configuration

zebernst commented 2 years ago

Forked the original synology-csi repository and added the Talos compatibility in this commit!

I'll write up a guide for setting this all up in the next few days and open a PR into the Talos repository then. Thanks again @frezbo :)

btkostner commented 2 years ago

Sorry to piggyback off this issue (let me know if you want me to open another one).

regarding the error I think it's due to the sysfs not being writable, once https://github.com/siderolabs/talos/pull/5550 is merged that should be easily done. I tested with a local patch of mine (i had forgot about that).My apologies, I assumed the csi driver had an alternate mode of operation.

Looking at GitHub, this issue should be fixed and released in talos 1.1.0 stable. I'm still getting a similar error when trying to use the iscsi extension. I'm using the https://github.com/zebernst/synology-csi-talos fork btw.

Talos version:

Client:
    Tag:         v1.1.0
    SHA:         d55a1871
    Built:       
    Go version:  go1.18.3
    OS/Arch:     linux/amd64
Server:
    NODE:        192.168.1.153
    Tag:         v1.1.0
    SHA:         d55a1871
    Built:       
    Go version:  go1.18.3
    OS/Arch:     linux/amd64
    Enabled:     RBAC

Everything looks correct in the Synology interface. It's all created on demand from Kubernetes. But when trying to mount to a pod, the synology-csi-node pod csi-plugin container shows a huge amount of this:

2022-07-14T18:33:39Z [INFO] [driver/utils.go:105] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/csi.san.synology.com/f526502c071f6ab31a75399a71c1f6a1d225a478b7ac7f5d43f5fdb30e671b4f/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"btrfs"}},"access_mode":{"mode":1}},"volume_context":{"dsm":"192.168.1.21","protocol":"iscsi","source":"","storage.kubernetes.io/csiProvisionerIdentity":"1657671234633-8081-csi.san.synology.com"},"volume_id":"baa7231d-dd83-4c6d-b098-614c34555d83"}
2022-07-14T18:33:39Z [INFO] [driver/initiator.go:109] Session[iqn.2000-01.com.synology:Behemoth.pvc-65396d70-131d-4aa0-9356-866de652b73d] already exists.
2022-07-14T18:33:40Z [ERROR] [driver/nodeserver.go:60] Can't find device path [/dev/disk/by-path/ip-192.168.1.21:3260-iscsi-iqn.2000-01.com.synology:Behemoth.pvc-65396d70-131d-4aa0-9356-866de652b73d-lun-1], err: <nil>
2022-07-14T18:33:41Z [ERROR] [driver/nodeserver.go:60] Can't find device path [/dev/disk/by-path/ip-192.168.1.21:3260-iscsi-iqn.2000-01.com.synology:Behemoth.pvc-65396d70-131d-4aa0-9356-866de652b73d-lun-1], err: <nil>
2022-07-14T18:33:42Z [ERROR] [driver/nodeserver.go:60] Can't find device path [/dev/disk/by-path/ip-192.168.1.21:3260-iscsi-iqn.2000-01.com.synology:Behemoth.pvc-65396d70-131d-4aa0-9356-866de652b73d-lun-1], err: <nil>

iscsiadm shows an active session and an attached scsi devices, but it does not show any disk path.

                ************************
                Attached SCSI devices:
                ************************
                Host Number: 1  State: running
bash-5.1# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0         7:0    0   2.1M  1 loop 
loop1         7:1    0     4K  1 loop 
loop2         7:2    0  43.3M  1 loop 
nvme0n1     259:0    0 931.5G  0 disk 
├─nvme0n1p1 259:1    0   100M  0 part 
├─nvme0n1p2 259:2    0     1M  0 part 
├─nvme0n1p3 259:3    0  1000M  0 part 
├─nvme0n1p4 259:4    0     1M  0 part 
├─nvme0n1p5 259:5    0   100M  0 part /host/system/state
└─nvme0n1p6 259:6    0 930.3G  0 part /var/lib/kubelet
                                      /etc/resolv.conf
                                      /etc/hostname
                                      /dev/termination-log
                                      /etc/hosts
                                      /host/var
                                      /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/e0a1b3ecebd40977ce63b5ce4d8957964f8e7c04d332d041da6a9d336da33603/rootfs/csi
                                      /csi
frezbo commented 2 years ago

are you using v1.0.1 of the iscsi extension?

btkostner commented 2 years ago

I'm using v0.1.1 from here: https://github.com/siderolabs/extensions/pkgs/container/iscsi-tools/28363622?tag=v0.1.1

frezbo commented 2 years ago

yeh, that along with 1.1.x should work:

 talosctl -e 192.168.20.61 -n 192.168.20.61 version
Client:
    Tag:         v1.1.1
    SHA:         40a050c6
    Built:       
    Go version:  go1.18.4
    OS/Arch:     linux/amd64
Server:
    NODE:        192.168.20.61
    Tag:         v1.1.1
    SHA:         40a050c6
    Built:       
    Go version:  go1.18.4
    OS/Arch:     linux/arm64
    Enabled:     RBAC

❯ talosctl -e 192.168.20.61 -n 192.168.20.61 get extensions 
NODE            NAMESPACE   TYPE              ID                                          VERSION   NAME          VERSION
192.168.20.61   runtime     ExtensionStatus   000.ghcr.io-siderolabs-iscsi-tools-v0.1.1   1         iscsi-tools   v0.1.1
frezbo commented 2 years ago

what does talosctl read /usr/local/etc/containers/iscsid.yaml return?

zebernst commented 2 years ago

@btkostner I ran into that issue last week - I'm pretty sure that either deleting and recreating the PVs and PVCs and/or just plain rebooting the nodes fixed the issue for me. Looking at my Git history, there were no config changes needed to get that working again.

btkostner commented 2 years ago

Ah, thanks for the help there. Probably should have tried that before commenting on this issue. I wiped the node (just to make sure everything was good) and it is working now after that. Thank you for that help!

travisghansen commented 2 years ago

Support was added for talos to https://github.com/democratic-csi/democratic-csi with details available here: https://github.com/democratic-csi/democratic-csi/pull/225 All requirements are now upstream and can be enabled via a couple options in the helm chart.

The project is a a sort of meta-driver that works with many different storage systems including synology for those on this thread who may be looking for an alternative to the upstream synology driver.

frezbo commented 2 years ago

Support was added for talos to https://github.com/democratic-csi/democratic-csi

Awesome work :heart: