iscsiadm tries to write to /etc/iscsi but that folder is readonly, how is iscsiadm ever meant to function, or is it customiseable?

[MysticalMount]

Apologies if this isnt a bug, and is just me missing something somewhere, here goes...

My evaluation of Talos (a personal evaluation for my own Homelab usage!) has gone pretty well, k3os has reached EOL, and Talos just does the node management better overall.However Ive spent all weekend trying to get iScsi working, to start with from a Synology NAS but potentially other types of iScsi capable CSI devices in the future e.g. TrueNAS/CephFS

Managed to get to the point of adding the iscsi-tools extension - so now iscsiadm is installed on the node. I am using Talos v1.3.5 and Kubernetes 1.26.1 (whatever is default or Talos v1.3.5) However, iscsiadm seems to want to write to /etc/iscsi - I checked through a container mounted to the host root at /host, and was able to touch a hello file in /var, but in etc its readonly. I understand this is probably for good reason, what I cant understand, is how everyone has got iScsi to work with Talos, if iscsiadm cant be customised and always tries this write during iScsi discovery. The current logs:

2023-03-12T12:47:49Z [INFO] [driver/utils.go:104] GRPC call: /csi.v1.Node/NodeGetCapabilities
2023-03-12T12:47:49Z [INFO] [driver/utils.go:105] GRPC request: {}
2023-03-12T12:47:49Z [INFO] [driver/utils.go:110] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}}]}
2023-03-12T12:47:49Z [INFO] [driver/utils.go:104] GRPC call: /csi.v1.Node/NodeGetCapabilities
2023-03-12T12:47:49Z [INFO] [driver/utils.go:105] GRPC request: {}
2023-03-12T12:47:49Z [INFO] [driver/utils.go:110] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}}]}
2023-03-12T12:47:49Z [INFO] [driver/utils.go:104] GRPC call: /csi.v1.Node/NodeGetCapabilities
2023-03-12T12:47:49Z [INFO] [driver/utils.go:105] GRPC request: {}
2023-03-12T12:47:49Z [INFO] [driver/utils.go:110] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}}]}
2023-03-12T12:47:49Z [INFO] [driver/utils.go:104] GRPC call: /csi.v1.Node/NodeStageVolume
2023-03-12T12:47:49Z [INFO] [driver/utils.go:105] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/csi.san.synology.com/a3df53ebfeca2c31ff4240cc280acc6c584482c1bd3a040de6f31951e70778d5/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":5}},"volume_context":{"dsm":"10.x.x.x","protocol":"iscsi","source":"","storage.kubernetes.io/csiProvisionerIdentity":"1678541037198-8081-csi.san.synology.com"},"volume_id":"0d06308d-7215-4ed8-b2ae-ca9299580437"}
2023-03-12T12:47:50Z [ERROR] [driver/initiator.go:92] Failed to run iscsiadm session: exit status 6
2023-03-12T12:47:50Z [ERROR] [driver/initiator.go:179] Failed in discovery of the target: + DIR=/host
++ basename /csibin/iscsiadm
+ BIN=iscsiadm
+ '[' -d /host ']'
+ exec chroot /host iscsiadm -m discoverydb --type sendtargets --portal 10.x.x.x:3260 --discover
iscsiadm: Could not make /etc/iscsi 30
iscsiadm: exiting due to idbm configuration error
 (exit status 6)
2023-03-12T12:47:50Z [ERROR] [driver/utils.go:108] GRPC error: rpc error: code = Internal desc = rpc error: code = Internal desc = Failed to login with target iqn [iqn.2000-01.com.synology:ayanas.pvc-32f7e29f-b626-4088-b157-e00a044c6009], err: + DIR=/host
++ basename /csibin/iscsiadm
+ BIN=iscsiadm
+ '[' -d /host ']'
+ exec chroot /host iscsiadm -m discoverydb --type sendtargets --portal 10.x.x.x:3260 --discover
iscsiadm: Could not make /etc/iscsi 30
iscsiadm: exiting due to idbm configuration error
 (exit status 6)       

Im probably missing something obvious, to get here there were a few things I had to do which Im going to revisit - as a new user to Talos, and my use case this might be worth mentioning:

• I spent 4 hours trying to get a trusted root CA installed, it was installed, that was fairly easy, but no matter what I did, even cat'ed the ca-certificate file in /etc/ssl/certs/ca-certificates it would not trust my root CA (added both root and int just incase). Eventually gave up and used insecure mode on my custom registries. Where it could not talk to the custom registries, talosctl api gets disabled upon configuring the node, as in the API is connection refused on port 50000 - I must have resetted the configuration using the boot menu option, about 30 times trying to get it to work before reverting to insecure.
• Installed the Talos iScsi extension - via a patch
• Enabled the namespace containing the resources to use Pod Security Admission mode privileged as it does require these permissions (New to Kube 1.25+ and was new to me too!)

Any help appreciated, been delving into some of the code but it seems no matter what I do Im hard blocked if iscsiadm always tries to write to /etc instead of some other writeable loction.

The provisioner is working as I can see that in the Synology admin interface, but mounting the drive from the node grrrrr having fun with this lolI staretd to look at democratic-csi which also supports Synology iScsi (experimentally) as it would be easier to adjust the deployment to work with TrueNAS/CephFS in the future - but then realised it may have the same problem with iscsiadm - hence this message

[smira]

@frezbo could you please take a look, as you know this better

[frezbo]

I think I replied to a similar issue on slack, but the csi has to be updated to execute commands in the iscsiadm pid namespace as mentioned here https://github.com/siderolabs/extensions/issues/38#issuecomment-1125403043

[frezbo]

https://www.talos.dev/v1.3/kubernetes-guides/configuration/synology-csi/

[frezbo]

https://github.com/democratic-csi/democratic-csi#talos more detailed instructions are here

[MysticalMount]

Thanks @frezbo - will try democratic-csi now

[MysticalMount]

I can confirm I now have successfully connected using Talos and democratic-csi to my Synology NAS, could be just me but feel like democratic-csi is the way to go for this, but then thats the only method I managed to get working.

The key bit was as you highlighted in the Talos section above, to allow the alternate method of accessing iscsiadm in the node settings.

Much thanks, definitely improved my skills to a more advanced level in areas I thought I already knew!