Closed zebernst closed 2 years ago
yes, this is expected of any CSI drivers that does chroot
or nsenter
into the pid 1 namespace. These CSI drivers assume iscsid is running on the host, whereas when ran as a system extension it runs as a container (also the same reason you don't see /etc/iscsi
when running talosctl ls /etc
, /etc/iscsi
is available from /system/extension
for the iscsid process). So ideally the csi would have to execute the process in the pid namespace of the iscsi process. Since this is script based I believe we can modify the script to use nsenter
instead of chroot
and using the mount namespace of the iscsid process, something along the lines of
iscsid_pid=$(pgrep iscsid)
nsenter --mount="/proc/${iscsid_pid}/ns/mnt" --net="/proc/${iscsid_pid}/ns/net" -- /usr/local/sbin/iscsiadm "$@"
this requires the csi pod to have hostPID: true
in the pod spec of the pod that runs this script
Also any particular reason not to use the synology api as opposed to iscsi?
Thanks for the quick reply @frezbo! That snippet fixed the iscsiadm
call, but I'm still unable to mount a disk with the CSI plugin throwing the following error:
2022-05-12T21:05:52Z [INFO] [driver/utils.go:105] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-54b49ea5-1270-446d-88d9-4a511c08c363/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"btrfs"}},"access_mode":{"mode":1}},"volume_context":{"dsm":"<nas ip>","protocol":"iscsi","source":"","storage.kubernetes.io/csiProvisionerIdentity":"1652385377888-8081-csi.san.synology.com"},"volume_id":"415fcc44-b9a4-4b99-afe8-dee8c2971652"}
2022-05-12T21:05:52Z [ERROR] [driver/nodeserver.go:60] Can't find device path [/dev/disk/by-path/ip-<nas ip>:3260-iscsi-iqn.2000-01.com.synology:<hostname>.pvc-e00e9dc0-1a46-4a68-9a50-4ecdc7fbd5a8-lun-1], err: <nil>
2022-05-12T21:05:54Z [INFO] [driver/initiator.go:123] Login target portal [<nas ip>:3260], iqn [iqn.2000-01.com.synology:<hostname>.pvc-54b49ea5-1270-446d-88d9-4a511c08c363].
2022-05-12T21:05:54Z [ERROR] [driver/nodeserver.go:60] Can't find device path [/dev/disk/by-path/ip-<nas ip>:3260-iscsi-iqn.2000-01.com.synology:<hostname>.pvc-e00e9dc0-1a46-4a68-9a50-4ecdc7fbd5a8-lun-1], err: <nil>
.... [repeated about 60 times] ....
2022-05-12T21:06:37Z [ERROR] [driver/nodeserver.go:92] Volume mount path is not exist.
2022-05-12T21:06:37Z [ERROR] [driver/utils.go:108] GRPC error: rpc error: code = Internal desc = Can't get volume mount path
Since I can't see directly what iscsi
commands are being called under the hood I'll keep digging and probably raise an issue over at the synology-csi
project (unless this looks familiar to you?).
Also any particular reason not to use the synology api as opposed to iscsi?
Is that possible? I'm not dead-set on using iscsi but the synology-csi only offers iscsi or smb support for backing connections, and I was under the impression that storage over a network needed a protocol like iscsi. The creation of something like LUN volume seems like it fits into the purview of an API, but I didn't think that an API would be able to support the reads and writes of a filesystem.
Is that possible? I'm not dead-set on using iscsi but the synology-csi only offers iscsi or smb support for backing connections, and I was under the impression that storage over a network needed a protocol like iscsi. The creation of something like LUN volume seems like it fits into the purview of an API, but I didn't think that an API would be able to support the reads and writes of a filesystem.
My apologies, I assumed the csi driver had an alternate mode of operation.
regarding the error I think it's due to the sysfs
not being writable, once https://github.com/siderolabs/talos/pull/5550 is merged that should be easily done. I tested with a local patch of mine (i had forgot about that).
meanwhile can you upgrade the talos with this installer image: ghcr.io/frezbo/installer:v1.1.0-alpha.1-67-gcc4131c41
once that's done can you have talosctl dmesg --tail --follow
on another terminal and provide the output after when the volume mount is happening via the CSI driver
Using that installer image worked! The PVC mounted correctly and I verified that reading and writing works as expected. Thank you so much for your help @frezbo!
@zebernst That's so awesome :muscle: Also now we know at least one csi can work with some changes. Would you mind sending over some notes or a PR as an example?
Happy to - did you mean write up some notes for y'all or open a PR with synology-csi?
i think just here or the talos docs are fine, since this is mosly talos specific.
I can open a PR with a guide to setting up Synology CSI in the Talos v1.1 docs if that works? Let me know where you think this best belongs, this is your project 🙂
Forked the original synology-csi repository and added the Talos compatibility in this commit!
I'll write up a guide for setting this all up in the next few days and open a PR into the Talos repository then. Thanks again @frezbo :)
Sorry to piggyback off this issue (let me know if you want me to open another one).
regarding the error I think it's due to the sysfs not being writable, once https://github.com/siderolabs/talos/pull/5550 is merged that should be easily done. I tested with a local patch of mine (i had forgot about that).My apologies, I assumed the csi driver had an alternate mode of operation.
Looking at GitHub, this issue should be fixed and released in talos 1.1.0 stable. I'm still getting a similar error when trying to use the iscsi extension. I'm using the https://github.com/zebernst/synology-csi-talos fork btw.
Talos version:
Client:
Tag: v1.1.0
SHA: d55a1871
Built:
Go version: go1.18.3
OS/Arch: linux/amd64
Server:
NODE: 192.168.1.153
Tag: v1.1.0
SHA: d55a1871
Built:
Go version: go1.18.3
OS/Arch: linux/amd64
Enabled: RBAC
Everything looks correct in the Synology interface. It's all created on demand from Kubernetes. But when trying to mount to a pod, the synology-csi-node
pod csi-plugin
container shows a huge amount of this:
2022-07-14T18:33:39Z [INFO] [driver/utils.go:105] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/csi.san.synology.com/f526502c071f6ab31a75399a71c1f6a1d225a478b7ac7f5d43f5fdb30e671b4f/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"btrfs"}},"access_mode":{"mode":1}},"volume_context":{"dsm":"192.168.1.21","protocol":"iscsi","source":"","storage.kubernetes.io/csiProvisionerIdentity":"1657671234633-8081-csi.san.synology.com"},"volume_id":"baa7231d-dd83-4c6d-b098-614c34555d83"}
2022-07-14T18:33:39Z [INFO] [driver/initiator.go:109] Session[iqn.2000-01.com.synology:Behemoth.pvc-65396d70-131d-4aa0-9356-866de652b73d] already exists.
2022-07-14T18:33:40Z [ERROR] [driver/nodeserver.go:60] Can't find device path [/dev/disk/by-path/ip-192.168.1.21:3260-iscsi-iqn.2000-01.com.synology:Behemoth.pvc-65396d70-131d-4aa0-9356-866de652b73d-lun-1], err: <nil>
2022-07-14T18:33:41Z [ERROR] [driver/nodeserver.go:60] Can't find device path [/dev/disk/by-path/ip-192.168.1.21:3260-iscsi-iqn.2000-01.com.synology:Behemoth.pvc-65396d70-131d-4aa0-9356-866de652b73d-lun-1], err: <nil>
2022-07-14T18:33:42Z [ERROR] [driver/nodeserver.go:60] Can't find device path [/dev/disk/by-path/ip-192.168.1.21:3260-iscsi-iqn.2000-01.com.synology:Behemoth.pvc-65396d70-131d-4aa0-9356-866de652b73d-lun-1], err: <nil>
iscsiadm
shows an active session and an attached scsi devices, but it does not show any disk path.
************************
Attached SCSI devices:
************************
Host Number: 1 State: running
bash-5.1# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 2.1M 1 loop
loop1 7:1 0 4K 1 loop
loop2 7:2 0 43.3M 1 loop
nvme0n1 259:0 0 931.5G 0 disk
├─nvme0n1p1 259:1 0 100M 0 part
├─nvme0n1p2 259:2 0 1M 0 part
├─nvme0n1p3 259:3 0 1000M 0 part
├─nvme0n1p4 259:4 0 1M 0 part
├─nvme0n1p5 259:5 0 100M 0 part /host/system/state
└─nvme0n1p6 259:6 0 930.3G 0 part /var/lib/kubelet
/etc/resolv.conf
/etc/hostname
/dev/termination-log
/etc/hosts
/host/var
/host/run/containerd/io.containerd.runtime.v2.task/k8s.io/e0a1b3ecebd40977ce63b5ce4d8957964f8e7c04d332d041da6a9d336da33603/rootfs/csi
/csi
are you using v1.0.1
of the iscsi extension?
I'm using v0.1.1
from here: https://github.com/siderolabs/extensions/pkgs/container/iscsi-tools/28363622?tag=v0.1.1
yeh, that along with 1.1.x should work:
talosctl -e 192.168.20.61 -n 192.168.20.61 version
Client:
Tag: v1.1.1
SHA: 40a050c6
Built:
Go version: go1.18.4
OS/Arch: linux/amd64
Server:
NODE: 192.168.20.61
Tag: v1.1.1
SHA: 40a050c6
Built:
Go version: go1.18.4
OS/Arch: linux/arm64
Enabled: RBAC
❯ talosctl -e 192.168.20.61 -n 192.168.20.61 get extensions
NODE NAMESPACE TYPE ID VERSION NAME VERSION
192.168.20.61 runtime ExtensionStatus 000.ghcr.io-siderolabs-iscsi-tools-v0.1.1 1 iscsi-tools v0.1.1
what does talosctl read /usr/local/etc/containers/iscsid.yaml
return?
@btkostner I ran into that issue last week - I'm pretty sure that either deleting and recreating the PVs and PVCs and/or just plain rebooting the nodes fixed the issue for me. Looking at my Git history, there were no config changes needed to get that working again.
Ah, thanks for the help there. Probably should have tried that before commenting on this issue. I wiped the node (just to make sure everything was good) and it is working now after that. Thank you for that help!
Support was added for talos
to https://github.com/democratic-csi/democratic-csi with details available here: https://github.com/democratic-csi/democratic-csi/pull/225 All requirements are now upstream and can be enabled via a couple options in the helm chart.
The project is a a sort of meta-driver that works with many different storage systems including synology for those on this thread who may be looking for an alternative to the upstream synology driver.
Support was added for talos to https://github.com/democratic-csi/democratic-csi
Awesome work :heart:
Hello!
My situation is almost identical to #33 - I've got a cluster of raspberry pi 4's and a Synology NAS that I'm trying to use as a storage backend for my PVCs, and I can't use NFS due to the limitations of file locking SQLite databases on network filesystems. I'm using the synology-csi driver to handle creation, deletion, and connection of iSCSI-backed LUNs on my NAS as PersistentVolumes in my cluster.
The initial setup went smoothly, with the driver able to create and delete PersistentVolumes and their backing LUNs with no issue on my NAS. I now suspect that it was accomplishing that via the Synology API and not via iSCSI, as I have not actually been able to mount the volume via iSCSI.
Here's my
talosctl version
:Originally, this was the event trace when trying to mount a volume to a pod:
I dug around and determined it was due to
iscsiadm
being aliased to this code snippet (see the Dockerfile): https://github.com/SynologyOpenSource/synology-csi/blob/fc3359223fe51a13bcfa5a7cabbf59611bbeb901/chroot/chroot.sh#L4-L9It looks like
synology-csi
just mounts the whole host filesystem into/host
(source), and useschroot
to emulate callingiscsiadm
from the host environment. Since there's no/usr/bin/env
in the Talos environment, the command fails. I changed it to directly link to theiscsiadm
binary contained in theiscsi-tools
extension like so:That fixed the issue of not actually being able to run
iscsiadm
, but I'm now running into this error:From what I understand, the
/etc/iscsi
folder should be bound to/system/iscsi
: https://github.com/siderolabs/extensions/blob/7cf2843cd73aa807d72151ec64bb0dc913492d63/storage/iscsi-tools/iscsid.yaml#L41-L47The
iscsi-tools
extension seems to be working for the most part:talosctl ls /usr/local/sbin
returns a list ofiscsi
-related binariestalosctl get extensions
lists the correct information foriscsi-tools
iscsiadm --help
in the synology-csi container that mounts the host fsHowever, the
/system/iscsi
directory is not being bound correctly to/etc/iscsi
:Is there something I need to be doing in the machineconfig to get this directory bound as expected?