Dir quota issue, non-snap build

s3rj1k commented 5 years ago

Distribution: Ubuntu
Distribution version: 19.10

lxc info:

config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
addresses: []
architectures:
- x86_64
- i686
certificate: |
-----BEGIN CERTIFICATE-----
MIIB6TCCAW+gAwIBAgIRAMipi1laDv98az4lOTcezGcwCgYIKoZIzj0EAwMwMTEc
MBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzERMA8GA1UEAwwIcm9vdEB2bTEw
HhcNMTkxMDIyMTA0MTUxWhcNMjkxMDE5MTA0MTUxWjAxMRwwGgYDVQQKExNsaW51
eGNvbnRhaW5lcnMub3JnMREwDwYDVQQDDAhyb290QHZtMTB2MBAGByqGSM49AgEG
BSuBBAAiA2IABKsQRg/BlGLXNA5Il9Nz3mj0aG7ZeYxXNuiytzSMXyvB273HAba5
kx15faQCdX+FfoW4yzngahGDW2vU+T7DMqZvbF7o0OT+7OB4SeNW1LqEDooMMXkv
hq3NofHvJ5Zv4aNLMEkwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUF
BwMBMAwGA1UdEwEB/wQCMAAwFAYDVR0RBA0wC4IDdm0xhwRZuERuMAoGCCqGSM49
BAMDA2gAMGUCMBysBOJy0cjMSCq4k1Obh19bscduDa60Uz1NI4uVxXKED7SHVcP2
AefxN5wYpwzqdQIxAMiZBEyirelLYXgPkmaCs20q+v/ypZaFf06Oq9gEh+AT9hEW
STeU0KuRAYERDIY2Ig==
-----END CERTIFICATE-----
certificate_fingerprint: ed7df8975917d48705b272c51a1e04a611ce3e01d79a5ce28204ee5dc88c7ae2
driver: lxc
driver_version: 3.2.1
kernel: Linux
kernel_architecture: x86_64
kernel_features:
netnsid_getifaddrs: "true"
seccomp_listener: "true"
shiftfs: "true"
uevent_injection: "true"
unpriv_fscaps: "true"
kernel_version: 5.3.0-19-generic
lxc_features:
mount_injection_file: "true"
network_gateway_device_route: "true"
network_ipvlan: "true"
network_l2proxy: "true"
network_phys_macvlan_mtu: "true"
seccomp_notify: "true"
project: default
server: lxd
server_clustered: false
server_name: vm1
server_pid: 8694
server_version: "3.18"
storage: dir
storage_version: "1"

mount | grep prjquota:

/dev/md1 on /var/lib/lxd/storage-pools/default type ext4 (rw,relatime,prjquota)

tune2fs -l /dev/md1 | grep features:

Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize quota metadata_csum project

CT Storage config:

devices:
root:
path: /
pool: default
size: 8GB
type: disk

repquota -avugPs:

*** Report for project quotas on device /dev/md1
Block grace time: 7days; Inode grace time: 7days
                    Space limits                File limits
Project         used    soft    hard  grace    used  soft  hard  grace
----------------------------------------------------------------------
#0        --      8K      0K      0K              2     0     0       
#10001    --   1722M      0K   7630M          35869     0     0

lxc exec test -- sh -c "df -h":

Filesystem                                                 Size  Used Avail Use% Mounted on
/var/lib/lxd/storage-pools/default/containers/test/rootfs  7.5G  1.7G  5.8G  23% /
none                                                       492K  4.0K  488K   1% /dev
udev                                                       967M     0  967M   0% /dev/tty
tmpfs                                                      100K     0  100K   0% /dev/lxd
tmpfs                                                      100K     0  100K   0% /dev/.lxd-mounts
tmpfs                                                      994M     0  994M   0% /dev/shm
tmpfs                                                      994M  148K  994M   1% /run
tmpfs                                                      5.0M     0  5.0M   0% /run/lock
tmpfs                                                      994M     0  994M   0% /sys/fs/cgroup

Project quota on Dir storage backend backed by ext4 filesystem on soft-raid (mdadm) device works incorectly when this configuration is used in non-snap build of LXD.

To reproduce:

setup VM with above configuration, creating softraid and enabling prjquota on ext4 mount
compile LXC and LXD from sources, install, start, preseed default config
create CT with any OS inside
configure quota inside CT (8GB)
run command to create 1GB file, this command is succesfull:
```
# lxc exec test -- sh -c "fallocate -l 1G big"
```

run command to create 2GB file, this command fails:

# lxc exec test -- sh -c "fallocate -l 2G big"
fallocate: fallocate failed: File too large

Same issue can be reproduced in Ubuntu 18.04.3 with kernel 5.0.x in similar configuration

Any ideas how to fix this?

stgraber commented 5 years ago

I'll need to play with it a bit, but this isn't going to be a LXD bug, repquota shows we set the quota properly and the kernel shows it applied through df, so any issue with accounting/enforcement after that would be a kernel bug.

s3rj1k commented 5 years ago

@stgraber LXD snap version works with this disk layout and configs.

stgraber commented 5 years ago

And a container without a quota doesn't hit that error?

stgraber commented 5 years ago

File too large sounds like you may be hitting a prlimit/ulimit rather than it being a quota issue, which would explain why you would see something different between snap and manual build as one is started through systemd and our wrapper while the other isn't, possibly leading to different limits being applied.

s3rj1k commented 5 years ago

Inside CT:

root@test:~# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7730
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

I am actually using systemd to unit to start LXD

[Unit]
Description=LXD - main daemon
After=network-online.target openvswitch-switch.service lxcfs.service lxd.socket
Requires=network-online.target lxcfs.service lxd.socket

[Service]
EnvironmentFile=-/etc/environment
# ExecStartPre=/usr/lib/x86_64-linux-gnu/lxc/lxc-apparmor-load
ExecStart=/usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log
ExecStartPost=/usr/bin/lxd waitready --timeout=600
KillMode=process
TimeoutStartSec=600s
TimeoutStopSec=30s
Restart=on-failure
LimitNOFILE=infinity
LimitNPROC=infinity
LimitFSIZE=infinity
TasksMax=infinity

[Install]
Also=lxd-containers.service lxd.socket

added LimitFSIZE=infinity, still seeing error

s3rj1k commented 5 years ago

Actually yes, without quota still have this error, WTF?

setting

LimitAS=infinity
LimitCORE=infinity
LimitCPU=infinity
LimitDATA=infinity
LimitFSIZE=infinity
LimitLOCKS=infinity
LimitMEMLOCK=infinity
LimitMSGQUEUE=infinity
LimitNICE=infinity
LimitNOFILE=infinity
LimitNPROC=infinity
LimitRSS=infinity
LimitRTPRIO=infinity
LimitRTTIME=infinity
LimitSIGPENDING=infinity
LimitSTACK=infinity
TasksMax=infinity

does not help

disabled apparmor, still seeing error

s3rj1k commented 5 years ago

@stgraber this is shiftfs related. removing /lib/modules/5.3.0-19-generic/kernel/fs/shiftfs.ko fixed above error with large file.

Now the questing is how do I disable shiftfs property, blacklisting module did not help last time I tried.

Also there are no shiftfs settings in LXD itself.

s3rj1k commented 5 years ago

@stgraber I can confirm that quota works as expected without shiftfs.

Why on earth this is enabled by default on all kernels and LXD and in snap is disabled?

stgraber commented 5 years ago

LXD uses shiftfs whenever available, though due to kernel issues that are still being worked on, we have a patch in the snap that adds a knob to control it, disabling it by default and eventually enabling it if we're running on a known good kernel.

So sounds like this issue has nothing to do with project quotas but instead with handling of a particular filesystem operation. We'll need @brauner to look into that one and see if it's something we've fixed already in shiftfs or that needs extra fixing.

stgraber commented 5 years ago

Closing as there isn't any action for us to pursue in LXD, but will put it on @brauner's todo to sort this out at the shiftfs level.

brauner commented 5 years ago

diff --git a/fs/shiftfs.c b/fs/shiftfs.c
index 55bb32b611f2..49b7777dde22 100644
--- a/fs/shiftfs.c
+++ b/fs/shiftfs.c
@@ -2045,6 +2045,7 @@ static int shiftfs_fill_super(struct super_block *sb, void *raw_data,
                err = -EINVAL;
                goto out_put_path;
        }
+       sb->s_maxbytes = MAX_LFS_FILESIZE;

        inode = new_inode(sb);
        if (!inode) {

This should fix it.

stgraber commented 5 years ago

@brauner thanks, that was quick :) Can you get a test kernel for @s3rj1k to validate that his stuff works fine? And do the usual sending of the fix through the usual Ubuntu channels :)

One of these days we'll actually have shiftfs behave the way we want it!

s3rj1k commented 5 years ago

@stgraber @brauner Thanks )

s3rj1k commented 5 years ago

I am willing to test this out as soon as I get test kernel, also hoping that the fix will be on both LTS and non LTS Ubuntu distributions.

brauner commented 5 years ago

I think having Kinderkrankheiten like this is pretty normal for something like shiftfs. I'm actually more and more happy since it also allowed us to find bugs in other filesystems. I'm pretty sure that our overlayfs changes should be upstreamed but there's only so hours in one day. :)

brauner commented 5 years ago

@s3rj1k building a kernel atm.

brauner commented 5 years ago

With my fix:

Script started on 2019-10-22 20:37:32+0000
root@b3:~# fallocate --length 5g aaaa
root@b3:~# du -sh ./aaaa 
5.0G    ./aaaa
root@b3:~# rm aaaa 
root@b3:~# exit
Script done on 2019-10-22 20:37:50+0000

Here is a kernel for you to test: https://drive.google.com/open?id=15pLG3FAE52h6njRCfzgkZnG3tLen4ct0

brauner commented 5 years ago

@s3rj1k ^^

s3rj1k commented 5 years ago

@brauner kernel fixes File too large error but breaks quota enforcement 2019-10-23_00-02-55

as seen in screen I can create file larger then available quota.

Same kernel but without shiftfs quota works fine.

brauner commented 5 years ago

@stgraber, any idea about the quota stuff for the underlay?

brauner commented 5 years ago

I can reproduce this on non-shiftfs as well though.

s3rj1k commented 5 years ago

@brauner did you do tune2fs -O project,quota -Q prjquota /dev/sda1 on fs before mounting it with prjquota ?

quota for ext4 works only when enabled by tune2fs and mounted with proper mount option

brauner commented 5 years ago

No, I don't think so. @stgraber knows the quota stuff better so I'll let him comment first.

brauner commented 5 years ago

Might be as simple as:

commit 69a28dcd8c22b3afb71ba867837f508e14424910 (HEAD -> shiftfs_fallocate, origin/shiftfs_fallocate)
Author: Christian Brauner <christian.brauner@ubuntu.com>
Date:   Wed Oct 23 00:16:17 2019 +0200

    shiftfs: drop CAP_SYS_RESOURCE to avoid overriding disk quotas

    Signed-off-by: Christian Brauner <christian.brauner@ubuntu.com>

diff --git a/fs/shiftfs.c b/fs/shiftfs.c
index 49b7777dde22..81a73b87c395 100644
--- a/fs/shiftfs.c
+++ b/fs/shiftfs.c
@@ -2046,6 +2046,8 @@ static int shiftfs_fill_super(struct super_block *sb, void *raw_data,
                goto out_put_path;
        }
        sb->s_maxbytes = MAX_LFS_FILESIZE;
+       /* Don't override disk quota limits or use reserved space. */
+       cap_lower(sbinfo->creator_cred->cap_effective, CAP_SYS_RESOURCE);

        inode = new_inode(sb);
        if (!inode) {

s3rj1k commented 5 years ago

@brauner I can check later today (in morning by CEST) in my setup if you prepare updated kernel.

brauner commented 5 years ago

@s3rj1k sure, will link to a new kernel here.

stgraber commented 5 years ago

@brauner yeah, enabling project quotas needs some effort, the tooling for it is rather awful :) LXD dir containers on a filesystem that's had the tune2fs property set and is mounted with the prjquota option will normally do the right thing then.

brauner commented 5 years ago

@brauner yeah, enabling project quotas needs some effort, the tooling for it is rather awful :) LXD dir containers on a filesystem that's had the tune2fs property set and is mounted with the prjquota option will normally do the right thing then.

My Ubuntu vm doesn't support it and I got pretty mad when I realized that adding prjquota as a mount option in /etc/fstab forced me to reboot into rescue mode and remount my rootfs rw, so I could edit my fstab...

brauner commented 5 years ago

@s3rj1k https://drive.google.com/open?id=1hZFPw7cqCQL1ptCHQINgb2NBi4A5rTp2

s3rj1k commented 5 years ago

@brauner Confirming that latest kernel works correctly with quotas and shiftfs. Yay))

When this fix is expected to be publicly available? (To double test this on public kernel)

brauner commented 5 years ago

@brauner Confirming that latest kernel works correctly with quotas and shiftfs. Yay))

Excellent.

When this fix is expected to be publicly available? (To double test this on public kernel)

In a couple of weeks. I'll give you a link to the launchpad bug here.

brauner commented 5 years ago

@s3rj1k here are the launchpad bugs to track:

Once a kernel will be proposed it'll be mentioned in these bugs. You can subscribe to them to get notified when that happens.

s3rj1k commented 5 years ago

@brauner Thanks, hoping to see this fix soon )

s3rj1k commented 5 years ago

@stgraber Small not related question. How can I programmatically match container with project quota ID, to get usage statistics?

stgraber commented 5 years ago

The id is 10000 + id of container as you'd find them in lxd sql global "SELECT id, name FROM instances;"

Note that the used space is also reported through lxc info NAME.

s3rj1k commented 5 years ago

@stgraber Thanks, lxc info NAME sadly does not report correct usage. Sometimes Disk usage: paragraph disappears from lxc info NAME output. Other times it takes a lot of time to sync actual disk usage with what reports lxc info NAME.

s3rj1k commented 5 years ago

@stgraber Should I do separate Issue for lxc info NAME bug?

stgraber commented 5 years ago

Yeah, that'd be good to track down, I would have expected data to be returned just fine for dir backend.

canonical / lxd

Dir quota issue, non-snap build #6333