"unable to create quota group: File exists" when creating new container

[broizter]

Required information

• Distribution: Arch Linux
• Distribution version: All packages updated
• The output of "lxc info" or if that fails:

  config:
  core.metrics_address: 127.0.0.1:9100
  api_extensions:
  - storage_zfs_remove_snapshots
  - container_host_shutdown_timeout
  - container_stop_priority
  - container_syscall_filtering
  - auth_pki
  - container_last_used_at
  - etag
  - patch
  - usb_devices
  - https_allowed_credentials
  - image_compression_algorithm
  - directory_manipulation
  - container_cpu_time
  - storage_zfs_use_refquota
  - storage_lvm_mount_options
  - network
  - profile_usedby
  - container_push
  - container_exec_recording
  - certificate_update
  - container_exec_signal_handling
  - gpu_devices
  - container_image_properties
  - migration_progress
  - id_map
  - network_firewall_filtering
  - network_routes
  - storage
  - file_delete
  - file_append
  - network_dhcp_expiry
  - storage_lvm_vg_rename
  - storage_lvm_thinpool_rename
  - network_vlan
  - image_create_aliases
  - container_stateless_copy
  - container_only_migration
  - storage_zfs_clone_copy
  - unix_device_rename
  - storage_lvm_use_thinpool
  - storage_rsync_bwlimit
  - network_vxlan_interface
  - storage_btrfs_mount_options
  - entity_description
  - image_force_refresh
  - storage_lvm_lv_resizing
  - id_map_base
  - file_symlinks
  - container_push_target
  - network_vlan_physical
  - storage_images_delete
  - container_edit_metadata
  - container_snapshot_stateful_migration
  - storage_driver_ceph
  - storage_ceph_user_name
  - resource_limits
  - storage_volatile_initial_source
  - storage_ceph_force_osd_reuse
  - storage_block_filesystem_btrfs
  - resources
  - kernel_limits
  - storage_api_volume_rename
  - macaroon_authentication
  - network_sriov
  - console
  - restrict_devlxd
  - migration_pre_copy
  - infiniband
  - maas_network
  - devlxd_events
  - proxy
  - network_dhcp_gateway
  - file_get_symlink
  - network_leases
  - unix_device_hotplug
  - storage_api_local_volume_handling
  - operation_description
  - clustering
  - event_lifecycle
  - storage_api_remote_volume_handling
  - nvidia_runtime
  - container_mount_propagation
  - container_backup
  - devlxd_images
  - container_local_cross_pool_handling
  - proxy_unix
  - proxy_udp
  - clustering_join
  - proxy_tcp_udp_multi_port_handling
  - network_state
  - proxy_unix_dac_properties
  - container_protection_delete
  - unix_priv_drop
  - pprof_http
  - proxy_haproxy_protocol
  - network_hwaddr
  - proxy_nat
  - network_nat_order
  - container_full
  - candid_authentication
  - backup_compression
  - candid_config
  - nvidia_runtime_config
  - storage_api_volume_snapshots
  - storage_unmapped
  - projects
  - candid_config_key
  - network_vxlan_ttl
  - container_incremental_copy
  - usb_optional_vendorid
  - snapshot_scheduling
  - snapshot_schedule_aliases
  - container_copy_project
  - clustering_server_address
  - clustering_image_replication
  - container_protection_shift
  - snapshot_expiry
  - container_backup_override_pool
  - snapshot_expiry_creation
  - network_leases_location
  - resources_cpu_socket
  - resources_gpu
  - resources_numa
  - kernel_features
  - id_map_current
  - event_location
  - storage_api_remote_volume_snapshots
  - network_nat_address
  - container_nic_routes
  - rbac
  - cluster_internal_copy
  - seccomp_notify
  - lxc_features
  - container_nic_ipvlan
  - network_vlan_sriov
  - storage_cephfs
  - container_nic_ipfilter
  - resources_v2
  - container_exec_user_group_cwd
  - container_syscall_intercept
  - container_disk_shift
  - storage_shifted
  - resources_infiniband
  - daemon_storage
  - instances
  - image_types
  - resources_disk_sata
  - clustering_roles
  - images_expiry
  - resources_network_firmware
  - backup_compression_algorithm
  - ceph_data_pool_name
  - container_syscall_intercept_mount
  - compression_squashfs
  - container_raw_mount
  - container_nic_routed
  - container_syscall_intercept_mount_fuse
  - container_disk_ceph
  - virtual-machines
  - image_profiles
  - clustering_architecture
  - resources_disk_id
  - storage_lvm_stripes
  - vm_boot_priority
  - unix_hotplug_devices
  - api_filtering
  - instance_nic_network
  - clustering_sizing
  - firewall_driver
  - projects_limits
  - container_syscall_intercept_hugetlbfs
  - limits_hugepages
  - container_nic_routed_gateway
  - projects_restrictions
  - custom_volume_snapshot_expiry
  - volume_snapshot_scheduling
  - trust_ca_certificates
  - snapshot_disk_usage
  - clustering_edit_roles
  - container_nic_routed_host_address
  - container_nic_ipvlan_gateway
  - resources_usb_pci
  - resources_cpu_threads_numa
  - resources_cpu_core_die
  - api_os
  - container_nic_routed_host_table
  - container_nic_ipvlan_host_table
  - container_nic_ipvlan_mode
  - resources_system
  - images_push_relay
  - network_dns_search
  - container_nic_routed_limits
  - instance_nic_bridged_vlan
  - network_state_bond_bridge
  - usedby_consistency
  - custom_block_volumes
  - clustering_failure_domains
  - resources_gpu_mdev
  - console_vga_type
  - projects_limits_disk
  - network_type_macvlan
  - network_type_sriov
  - container_syscall_intercept_bpf_devices
  - network_type_ovn
  - projects_networks
  - projects_networks_restricted_uplinks
  - custom_volume_backup
  - backup_override_name
  - storage_rsync_compression
  - network_type_physical
  - network_ovn_external_subnets
  - network_ovn_nat
  - network_ovn_external_routes_remove
  - tpm_device_type
  - storage_zfs_clone_copy_rebase
  - gpu_mdev
  - resources_pci_iommu
  - resources_network_usb
  - resources_disk_address
  - network_physical_ovn_ingress_mode
  - network_ovn_dhcp
  - network_physical_routes_anycast
  - projects_limits_instances
  - network_state_vlan
  - instance_nic_bridged_port_isolation
  - instance_bulk_state_change
  - network_gvrp
  - instance_pool_move
  - gpu_sriov
  - pci_device_type
  - storage_volume_state
  - network_acl
  - migration_stateful
  - disk_state_quota
  - storage_ceph_features
  - projects_compression
  - projects_images_remote_cache_expiry
  - certificate_project
  - network_ovn_acl
  - projects_images_auto_update
  - projects_restricted_cluster_target
  - images_default_architecture
  - network_ovn_acl_defaults
  - gpu_mig
  - project_usage
  - network_bridge_acl
  - warnings
  - projects_restricted_backups_and_snapshots
  - clustering_join_token
  - clustering_description
  - server_trusted_proxy
  - clustering_update_cert
  - storage_api_project
  - server_instance_driver_operational
  - server_supported_storage_drivers
  - event_lifecycle_requestor_address
  - resources_gpu_usb
  - clustering_evacuation
  - network_ovn_nat_address
  - network_bgp
  - network_forward
  - custom_volume_refresh
  - network_counters_errors_dropped
  - metrics
  - image_source_project
  - clustering_config
  - network_peer
  - linux_sysctl
  - network_dns
  - ovn_nic_acceleration
  - certificate_self_renewal
  - instance_project_move
  - storage_volume_project_move
  - cloud_init
  - network_dns_nat
  - database_leader
  - instance_all_projects
  - clustering_groups
  - ceph_rbd_du
  - instance_get_full
  - qemu_metrics
  - gpu_mig_uuid
  - event_project
  - clustering_evacuation_live
  - instance_allow_inconsistent_copy
  - network_state_ovn
  - storage_volume_api_filtering
  - image_restrictions
  - storage_zfs_export
  - network_dns_records
  - storage_zfs_reserve_space
  - network_acl_log
  - storage_zfs_blocksize
  - metrics_cpu_seconds
  - instance_snapshot_never
  - certificate_token
  - instance_nic_routed_neighbor_probe
  - event_hub
  - agent_nic_config
  - projects_restricted_intercept
  - metrics_authentication
  - images_target_project
  - cluster_migration_inconsistent_copy
  - cluster_ovn_chassis
  - container_syscall_intercept_sched_setscheduler
  - storage_lvm_thinpool_metadata_size
  - storage_volume_state_total
  - instance_file_head
  - instances_nic_host_name
  - image_copy_profile
  - container_syscall_intercept_sysinfo
  - clustering_evacuation_mode
  - resources_pci_vpd
  - qemu_raw_conf
  - storage_cephfs_fscache
  - network_load_balancer
  - vsock_api
  - instance_ready_state
  - network_bgp_holdtime
  - storage_volumes_all_projects
  - metrics_memory_oom_total
  - storage_buckets
  - storage_buckets_create_credentials
  - metrics_cpu_effective_total
  - projects_networks_restricted_access
  - storage_buckets_local
  - loki
  - acme
  - internal_metrics
  - cluster_join_token_expiry
  - remote_token_expiry
  - init_preseed
  - storage_volumes_created_at
  - cpu_hotplug
  api_status: stable
  api_version: "1.0"
  auth: trusted
  public: false
  auth_methods:
  - tls
  environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
  -----BEGIN CERTIFICATE-----
  MIIB/jCCAYSgAwIBAgIRAKcBHnNHxB2s28URcfNvFhAwCgYIKoZIzj0EAwMwMjEc
  MBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzESMBAGA1UEAwwJcm9vdEB0YW5r
  MB4XDTIxMTAwMjE1MDgyOVoXDTMxMDkzMDE1MDgyOVowMjEcMBoGA1UEChMTbGlu
  dXhjb250YWluZXJzLm9yZzESMBAGA1UEAwwJcm9vdEB0YW5rMHYwEAYHKoZIzj0C
  AQYFK4EEACIDYgAE4si3X2g4NqXNd5WZqhgecBTCNMAVzOT+kQhQAIgXYJEJ2yWY
  NmgSCPHnzNTIIN244dJwo7DeF4saV+ZHmOVVW77xOe0GdqWmRiXmhDOIlDmM5wsB
  D4sEfoeq6n36AZvPo14wXDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYB
  BQUHAwEwDAYDVR0TAQH/BAIwADAnBgNVHREEIDAeggR0YW5rhwR/AAABhxAAAAAA
  AAAAAAAAAAAAAAABMAoGCCqGSM49BAMDA2gAMGUCMQDSnU+nmVYO7yPEs+DGZlxq
  wpcgEtK4BbB8R5UsuUSqK04d0jUjhdfVag2pl6MrEP4CME7c5clPSDij9BYHq+xS
  Kq1pO+BFrgAUz/4qj9HNXf1FzkjCrpobubS9hR99VaJPDw==
  -----END CERTIFICATE-----
  certificate_fingerprint: f44fd4d93be86e4dc6383030684f180ac1dca0f00540fc36e7285b5602313a19
  driver: lxc
  driver_version: 5.0.1
  firewall: nftables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
  idmapped_mounts: "true"
  netnsid_getifaddrs: "true"
  seccomp_listener: "true"
  seccomp_listener_continue: "true"
  shiftfs: "false"
  uevent_injection: "true"
  unpriv_fscaps: "true"
  kernel_version: 6.0.11-arch1-1
  lxc_features:
  cgroup2: "true"
  core_scheduling: "true"
  devpts_fd: "true"
  idmapped_mounts_v2: "true"
  mount_injection_file: "true"
  network_gateway_device_route: "true"
  network_ipvlan: "true"
  network_l2proxy: "true"
  network_phys_macvlan_mtu: "true"
  network_veth_router: "true"
  pidfd: "true"
  seccomp_allow_deny_syntax: "true"
  seccomp_notify: "true"
  seccomp_proxy_send_notify_fd: "true"
  os_name: Arch Linux
  os_version: ""
  project: default
  server: lxd
  server_clustered: false
  server_event_mode: full-mesh
  server_name: tank
  server_pid: 1971
  server_version: "5.8"
  storage: btrfs
  storage_version: 6.0.2
  storage_supported_drivers:
  - name: dir
  version: "1"
  remote: false
  - name: btrfs
  version: 6.0.2
  remote: false

Issue description

Trying to create new containers fails with this message Error: Failed instance creation: Failed creating instance from image: Failed to run: btrfs qgroup create 0/104960 /var/lib/lxd/storage-pools/default/images/f161c60ebcfe5806986bcfef748df7cf23bf7eb39eb5b7c130d0e5aa5371522f: exit status 1 (ERROR: unable to create quota group: File exists). My guess is that it started after upgrading btrfs-progs to 6.0.1. Upgrading to the latest version (6.0.2) did not fix the issue.

Steps to reproduce

1. Use BTRFS as the storage backend
2. Create a container

Information to attach

• lxc monitor output https://pastebin.com/g6PxBYcb

  Let me know if there's any other information you want me to attach. Same issue as this post I guess although I don't think downgrading btrfs-progs is a good fix. https://discuss.linuxcontainers.org/t/lxd-btrfs-archlinux-failed-instance-creation/15633

[casperx]

Sadly, disabling quota only fixes the issue if you didn't set the disk size for the container. If you set size of the disk device (for example, lxc launch -d root,size=512MiB images:debian/bookworm where root is disk device), LXD is going to enable btrfs quota on that storage pool and the issue occurs again.

[tomponline]

What were the reproducer steps in lxd to get to that state? That is what I'm after.

[broizter]

I managed to reproduce the issue on a fresh Arch install.

1. Use BTRFS as the filesystem on root
2. lxd init with all defaults (make sure btrfs is used as the storage backend)
3. Edit the "default" profile and add "size: 5GB" to the "root" device (I guess this activates qgroups)
4. Launch a container and you will get the error

As soon as you revert step 3 and also run btrfs quota disable /var/lib/lxd/storage-pools/default things will go back to working as normal again.

[casperx]

just install lxd on latest Arch system with all default lxd init except using btrfs as a storage for default storage pool and lxc launch -d root,size=512MiB images:debian/bookworm

[broizter]

just install lxd on latest Arch system with all default lxd init except using btrfs as a storage for default pool and lxc launch -d root,size=512MiB images:debian/bookworm

btrfs is the default for me during lxd init but I'll update my instructions just in case.

[casperx]

look like btrfs-progs after version 6.0, btrfs qgroup show added path column to be display by default, thus increasing fields from 4 to 5. because https://github.com/lxc/lxd/blob/master/lxd/storage/drivers/driver_btrfs_utils.go#L256 skip line that has more or less than 4 field, it won't find any Qgroupid.

[tomponline]

Ah good spot. Let me try and reproduce.

[tomponline]

Thanks I've reproduced this on Ubuntu Lunar Lobster now:

lxc storage create btrfs btrfs
Storage pool btrfs created

lxc launch images:alpine/3.17 c1 -s btrfs
Creating c1
Starting c1                                   

lxc config device set c1 root size=1GiB
Error: Failed to update device "root": Failed to run: btrfs qgroup create 0/257 /var/lib/lxd/storage-pools/btrfs/containers/c1: exit status 1 (ERROR: unable to create quota group: File exists)