[nfs-provisioner] xfs_quota permissions

[timstoop]

Hi,

The docs state:

enable-xfs-quota - If the provisioner will set xfs quotas for each volume it provisions. Requires that the directory it creates volumes in ('/export') is xfs mounted with option prjquota/pquota, **and that it has the privilege to run xfs_quota**. Default false.

What would be those permissions? I'm trying to run this on Kubernetes on GCE, using an XFS disk with the prjquota mount option, but no quota seems to be enforced when I create a PVC of a specific size. When I shell into the nfs-server-provisioner, I notice the /export/projects file which seem to have the correct information, but it's not being enforced for some reason.

When I manually try to run the command from the shell, xfs_quote gives me some weird errors:

bash-5.0# xfs_quota /export/
xfs_quota: cannot setup path for mount /export/: No such device or address
xfs_quota> print
xfs_quota>

and

bash-5.0# xfs_quota -P /export/projects 
xfs_quota> print
print: foreign filesystem. Invoke xfs_quota with -f to enable.

The contents of /export/projects:

bash-5.0# cat /export/projects 

1:/export/pvc-dd236182-9a04-4c5d-bc59-ffda40094afb:262144000
bash-5.0#

I tried with CAP_SYS_ADMIN on the container as well, but this seemed to have no effect.

Has anyone gotten this to work? Anything I should check or be aware of? Are there any docs available about this?

[wongma7]

That "foreign filesystem" error is suspect. How is /export mounted into the container, like did you use a GCE PD PersistentVolume and give it mountOptions: prjquota, fsType: xfs? If you run mount in the container, what do you see, /export mounted as type xfs with prjquota?

[timstoop]

Hi Matthew,

Thanks for responding! Yes, I created a dedicated StorageClass with exactly those features:

$ kubectl get sc standard-xfs-quota -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"standard-xfs-quota"},"mountOptions":["pquota","prjquota"],"parameters":{"fsType":"xfs","type":"pd-standard"},"provisioner":"kubernetes.io/gce-pd","reclaimPolicy":"Delete","volumeBindingMode":"WaitForFirstConsumer"}
  creationTimestamp: "2020-06-26T11:56:29Z"
  name: standard-xfs-quota
  resourceVersion: "194023116"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/standard-xfs-quota
  uid: 2e50dc3a-075d-4689-8b00-6b1d632d3161
mountOptions:
- pquota
- prjquota
parameters:
  fsType: xfs
  type: pd-standard
provisioner: kubernetes.io/gce-pd
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

I noticed afterwards that the pquota option is not actually necessary, as it's not seen in the mount itself:

bash-5.0# mount | grep sdg
/dev/sdg on /export type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
bash-5.0# 

This is running on Kubernetes 1.16.10 using Debian 10 (Buster) node images that use the 5.4 kernel from backports. Not sure if that might matter.

[wongma7]

Yes, prjquota and pjquota are equivalent.

The last thing I would try is to run the provisioner as privileged, CAP_SYS_ADMIN may not be sufficient. Don't think xfs_quota was ever intended to run in a container.

This is a feature that is not regularly tested (personally I haven't used it in a year) and it is fragile as IIRC it can even depend on the container runtime + kernel. Your kernel is definitely new enough and mount succeeded so of course the xfs kernel module is there.

[timstoop]

Running as privileged seems to solve the issue with xfs_quota!

However, I now get an error when trying to read the information:

bash-5.0# xfs_quota -p /export/
/export/: cannot find mount point for path `/export/pvc-972f5790-1282-436f-a315-fc2f76503b20:262144000': No such file or directory
/export/>

And this makes sense, as it looks like there's an incorrect projects file created. It has the quota added as :262144000 in there, but that does not seem to be the way it's supposed to work: https://linux.die.net/man/5/projects

Removing it seems to work:

bash-5.0# cat projects 

1:/export/pvc-972f5790-1282-436f-a315-fc2f76503b20:262144000
bash-5.0# echo "1:/export/pvc-972f5790-1282-436f-a315-fc2f76503b20" > /export/projects 
bash-5.0# cat /export/projects 
1:/export/pvc-972f5790-1282-436f-a315-fc2f76503b20
bash-5.0# xfs_quota -p /export/
/export/> print
Filesystem          Pathname
/export             /dev/sdg (pquota)
/export/pvc-972f5790-1282-436f-a315-fc2f76503b20 /dev/sdg (project 1)
/export/>

Quota is still set, but for some reason (probably because I started without the privileged set to true), it's not being tracked:

/export/> report
Project quota on /export (/dev/sdg)
                               Blocks                     
Project ID       Used       Soft       Hard    Warn/Grace     
---------- -------------------------------------------------- 
#0             203100          0          0     00 [--------]
#1                  0          0     256000     00 [--------]

If I run project 1, I get a lot of messages like these:

[...]
/export/pvc-972f5790-1282-436f-a315-fc2f76503b20/wp-signup.php - project identifier is not set (inode=0, tree=1)
/export/pvc-972f5790-1282-436f-a315-fc2f76503b20/wp-signup.php - project inheritance flag is not set
/export/pvc-972f5790-1282-436f-a315-fc2f76503b20/wp-trackback.php - project identifier is not set (inode=0, tree=1)
/export/pvc-972f5790-1282-436f-a315-fc2f76503b20/wp-trackback.php - project inheritance flag is not set
/export/pvc-972f5790-1282-436f-a315-fc2f76503b20/xmlrpc.php - project identifier is not set (inode=0, tree=1)
/export/pvc-972f5790-1282-436f-a315-fc2f76503b20/xmlrpc.php - project inheritance flag is not set
/export/pvc-972f5790-1282-436f-a315-fc2f76503b20/.htaccess - project identifier is not set (inode=0, tree=1)
/export/pvc-972f5790-1282-436f-a315-fc2f76503b20/.htaccess - project inheritance flag is not set
/export/pvc-972f5790-1282-436f-a315-fc2f76503b20/wp-config.php - project identifier is not set (inode=0, tree=1)
/export/pvc-972f5790-1282-436f-a315-fc2f76503b20/wp-config.php - project inheritance flag is not set
/export/pvc-972f5790-1282-436f-a315-fc2f76503b20/status.php - project identifier is not set (inode=0, tree=1)
/export/pvc-972f5790-1282-436f-a315-fc2f76503b20/status.php - project inheritance flag is not set
Processed 1 (/etc/projects and cmdline) paths for project 1 with recursion depth infinite (-1).

However, if I create a new mount, which edits the /export/projects file again incorrectly, it seems to work:

bash-5.0# cat /export/projects 
1:/export/pvc-972f5790-1282-436f-a315-fc2f76503b20

2:/export/pvc-34725a29-7ce7-4233-bbfc-826aa44d67d1:157286400
bash-5.0# xfs_quota -x -p /export/
/export/: cannot find mount point for path `/export/pvc-34725a29-7ce7-4233-bbfc-826aa44d67d1:157286400': No such file or directory
/export/>
bash-5.0# echo "1:/export/pvc-972f5790-1282-436f-a315-fc2f76503b20" > /export/projects 
bash-5.0# echo "2:/export/pvc-34725a29-7ce7-4233-bbfc-826aa44d67d1" >> /export/projects
bash-5.0# xfs_quota -x -p /export/
/export/> report
Project quota on /export (/dev/sdg)
                               Blocks                     
Project ID       Used       Soft       Hard    Warn/Grace     
---------- -------------------------------------------------- 
#0             203100          0          0     00 [--------]
#1                  0          0     256000     00 [--------]
#2              68392          0     153600     00 [--------]

And quota seems to work as well:

root@shellpod:/srv# rsync -a www/. test/.
rsync: close failed on "/srv/test/tempbigfile.5dNy2K": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(853) [receiver=3.1.3]
root@shellpod:/srv#

and

/export/> report
Project quota on /export (/dev/sdg)
                               Blocks
Project ID       Used       Soft       Hard    Warn/Grace
---------- --------------------------------------------------
#0             203100          0          0     00 [--------]
#1                  0          0     256000     00 [--------]
#2             153328          0     153600     00 [--------]

To summarize, to make it work, I have to:

• Run the container with securityContext.privileged: true,
• Edit the /export/projects file to sed -i 's/(:[^:]+)$// (remove the last colon and everything after it from each line)

If I was a Go programmer, I'd provide you with something more useful, but hopefully this allows someone to fix the problem easily as well? Thanks in advance!

[wongma7]

Edit the /export/projects file to sed -i 's/(:[^:]+)$// (remove the last colon and everything after it from each line)

It's quite confusing, but that projects file /export/projects is not the same /etc/projects file in https://linux.die.net/man/5/projects. It's a file created and used by the provisioner to store state and repopulate the actual xfsquota /etc/projects file in case the provisioner gets restarted, so editing it shouldn't have any effect. (Probably it should be named projects.backup or something, but it's too late to change.)

So I am not sure if removing :262144000 was the fix or if running as privileged alone did it. If the removal was the fix then yes it will be reasonably easy to patch the provisioner to stop storing it in /export/projects and instead store it in a separate file.

BTW, this project was recently migrated to https://github.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner. I will try my best to continue monitoring this repo (have not done the best job of keeping up) but that repo will be the future.

[timstoop]

It's quite confusing, but that projects file /export/projects is not the same /etc/projects file

You're right, that's indeed quite confusing :) In that case, it seems that running the Pod as privileged is indeed the solution at least. I did not check if the quota worked after setting it to privileged, as the xfs_quota tool was giving an error still (but that's probably due to my inexperience with the tool overall). The /etc/projects file, which is created now and wasn't created before is indeed correct.

If the project is switching to a new repo, I'll use that one! Maybe add a message about that in the top of the README here? Thanks for you help in any case, my problem is solved!