zulhfreelancer
commented
1 year ago @travisghansen - any chance you could take a look at my question above? Your help would be greatly appreciated! 🙂
Closed zulhfreelancer closed 1 year ago
zulhfreelancer
commented
1 year ago @travisghansen - any chance you could take a look at my question above? Your help would be greatly appreciated! 🙂
travisghansen
commented
1 year ago Welcome! Been super busy and on my phone atm but I’ll do my best right now.
dataset user/group does a chmod on the fs server-side. If you are using proper alignment of configs etc kubelet can set that for you using pod security context. The share map values are standard nfs things so I’m sure others have better explanations on those than I do. I would like to see the error logs for the failures when they were set.
with nfs the usage is almost certainly unnecessary. At the end of the day if you use the security context as mentioned above kubelet will set the right permissions for you. Setting the user/group would generally be counter-productive as the user/group of the pod will almost certainly not match the values you set in the config. In short I would leave them empty unless you otherwise have a need.
this is a good question and easy to answer…all zfs operations are done over ssh (and have shared code paths with all the other zfs/ssh drivers). All share buildup/teardown are done using the api. It’s a very clear line of separation in the code as so much of the logic is shared across drivers.
As an aside, I always recommend nfsv3 unless you have a very specific need to use v4. nfs v4 is a case of newer doesn’t mean better and frankly the things trying to be addressed by v4 are generally a waste in a k8s world.
https://github.com/democratic-csi/democratic-csi/issues/274#issuecomment-1426922600
zulhfreelancer
commented
1 year ago Thank you for the answers, @travisghansen. Much appreciated.
Here is how everything looks when I set 4 fields in question above.
All pods including test pod
All democratic-csi pods are happy
Containers in controller pod
All containers in this democratic-csi controller pod are happy
Test PVC
It's not happy. Thus, my test pod (orang color above) also not happy. See
kubectl describeoutput below.
Test Pod & PVC manifests
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: test-claim-nfs-pvc
namespace: democratic-csi
annotations:
volume.beta.kubernetes.io/storage-class: "freenas-nfs-csi"
spec:
storageClassName: freenas-nfs-csi
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
name: test-claim-nfs-pod
namespace: democratic-csi
spec:
volumes:
- name: task-pv-storage
persistentVolumeClaim:
claimName: test-claim-nfs-pvc
containers:
- name: task-pv-container
image: nginx
ports:
- containerPort: 80
name: "http-server"
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: task-pv-storageOutput of kubectl describe pvc test-claim-nfs-pvc
Name: test-claim-nfs-pvc
Namespace: democratic-csi
StorageClass: freenas-nfs-csi
Status: Pending
Volume:
Labels: <none>
Annotations: volume.beta.kubernetes.io/storage-class: freenas-nfs-csi
volume.beta.kubernetes.io/storage-provisioner: org.democratic-csi.nfs
volume.kubernetes.io/storage-provisioner: org.democratic-csi.nfs
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: test-claim-nfs-pod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 21s (x5 over 60s) org.democratic-csi.nfs_zfs-nfs-democratic-csi-controller-8657f8468f-nfr6d_7cfa307e-ea4e-49bd-b781-77e22c87dcf1 failed to provision volume with StorageClass "freenas-nfs-csi": rpc error: code = Unknown desc = received error creating nfs share - code: 422 body: {"sharingnfs_create.maproot_user":[{"message":"User not found","errno":22}],"sharingnfs_create.maproot_group":[{"message":"Group not found","errno":22}]}
Normal ExternalProvisioning 9s (x6 over 67s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "org.democratic-csi.nfs" or manually created by system administrator
Normal Provisioning 5s (x6 over 67s) org.democratic-csi.nfs_zfs-nfs-democratic-csi-controller-8657f8468f-nfr6d_7cfa307e-ea4e-49bd-b781-77e22c87dcf1 External provisioner is provisioning volume for claim "democratic-csi/test-claim-nfs-pvc"Any idea how to deal with this? I can confirm the group and user were created in my TrueNAS server. The GID and UID are also matched.
travisghansen
commented
1 year ago For the nfs/share values use the name (of user and group) instead of id.
zulhfreelancer
commented
1 year ago When I set the user and group to strings instead of IDs, I got the following. And the test PVC and pod remain unhappy like in my previous comment.
Name: test-claim-nfs-pvc
Namespace: democratic-csi
StorageClass: freenas-nfs-csi
Status: Pending
Volume:
Labels: <none>
Annotations: volume.beta.kubernetes.io/storage-class: freenas-nfs-csi
volume.beta.kubernetes.io/storage-provisioner: org.democratic-csi.nfs
volume.kubernetes.io/storage-provisioner: org.democratic-csi.nfs
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: test-claim-nfs-pod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 9s (x3 over 12s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "org.democratic-csi.nfs" or manually created by system administrator
Warning ProvisioningFailed 3s (x2 over 8s) org.democratic-csi.nfs_zfs-nfs-democratic-csi-controller-578fc6f4bd-bg9cv_48ee7055-5864-4a38-bf69-c5a768a64f56 failed to provision volume with StorageClass "freenas-nfs-csi": rpc error: code = FailedPrecondition desc = BREAKING CHANGE since v1.5.3! datasetPermissionsUser must be numeric: k8s-nfs is invalid
Normal Provisioning 1s (x3 over 12s) org.democratic-csi.nfs_zfs-nfs-democratic-csi-controller-578fc6f4bd-bg9cv_48ee7055-5864-4a38-bf69-c5a768a64f56 External provisioner is provisioning volume for claim "democratic-csi/test-claim-nfs-pvc"It's a bit confusing, the dataset/fs permissions need to be id, the share need to be name: https://github.com/democratic-csi/democratic-csi/blob/master/src/driver/freenas/ssh.js#L1908-L1927
It's just driven by the TrueNAS API requirements currently. I probably could lookup user based on ID or whatever but currently I just pass the values through to the API.
zulhfreelancer
commented
1 year ago That's fixed it, thank you. For reference, this is the working Helm chart values:
csiDriver:
name: "org.democratic-csi.nfs"
storageClasses:
- name: freenas-nfs-csi
defaultClass: false
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
parameters:
fsType: nfs
mountOptions:
- noatime
- nfsvers=4
secrets:
provisioner-secret:
controller-publish-secret:
node-stage-secret:
node-publish-secret:
controller-expand-secret:
driver:
config:
driver: freenas-nfs
instance_id:
httpConnection:
protocol: http
host: <my-truenas-core-ip>
port: 80
apiVersion: 2
apiKey: redacted
allowInsecure: true
sshConnection:
host: <my-truenas-core-ip>
port: 22
username: k8s-nfs
privateKey: |
-----BEGIN OPENSSH PRIVATE KEY-----
redacted
-----END OPENSSH PRIVATE KEY-----
zfs:
cli:
sudoEnabled: true
datasetParentName: foo/k8s/nfs/vols
detachedSnapshotsDatasetParentName: foo/k8s/nfs/snaps
datasetEnableQuotas: true
datasetEnableReservation: false
datasetPermissionsMode: "0777"
datasetPermissionsUser: 1007
datasetPermissionsGroup: 1001
nfs:
shareHost: <my-truenas-core-ip>
shareAlldirs: false
shareAllowedHosts: []
shareAllowedNetworks: []
shareMaprootUser: k8s-nfs
shareMaprootGroup: k8s-nfs
shareMapallUser: ""
shareMapallGroup: ""Going to close this issue and re-open if needed
I'm running TrueNAS Core 13 and democratic-csi Helm chart v0.13.7 (scroll down to see my Helm chart's values file).
In my TrueNAS, I've a created
k8s-nfsgroup (GID:1001) withk8s-nfsuser (UID:1007) as group member.I also have followed the Server Prep section based on the democratic-csi's README, specifically this section:
FYI, I've replaced the sample user (
csi) above with my user (k8s-nfs) when executing the commands.I noticed that my PVCs failed to provision when I set these fields:
driver.config.zfs.datasetPermissionsUserto1007driver.config.zfs.datasetPermissionsGroupto1001driver.config.nfs.shareMaprootUserto1007driver.config.nfs.shareMaprootGroupto1001But when I did not set them like below, my PVCs are happy.
Questions
driver.config.zfs.datasetPermissionsUser,driver.config.zfs.datasetPermissionsGroup,driver.config.nfs.shareMaprootUseranddriver.config.nfs.shareMaprootGroupfields?driver.config.httpConnectionanddriver.config.sshConnectionblocks need to co-exist. My PVCs did not get created properly if I didn't specify both blocks.