TommyLike commented 2 years ago

Required information

Distribution: CentOS Linux 7 (Core)
Distribution version: CentOS Linux 7 (Core)

The output of "lxc info" or if that fails: Client version: 4.24 Server version: 4.24

config:
core.https_address: 10.0.0.124:8443
core.trust_password: true
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
addresses:
- 10.0.0.124:8443
architectures:
- x86_64
- i686
certificate: 
driver: lxc
driver_version: 4.0.12
firewall: xtables
kernel: Linux
kernel_architecture: x86_64
kernel_features:
idmapped_mounts: "false"
netnsid_getifaddrs: "false"
seccomp_listener: "false"
seccomp_listener_continue: "false"
shiftfs: "false"
uevent_injection: "false"
unpriv_fscaps: "true"
kernel_version: 3.10.0-1160.25.1.el7.x86_64
lxc_features:
cgroup2: "true"
core_scheduling: "true"
devpts_fd: "true"
idmapped_mounts_v2: "true"
mount_injection_file: "true"
network_gateway_device_route: "true"
network_ipvlan: "true"
network_l2proxy: "true"
network_phys_macvlan_mtu: "true"
network_veth_router: "true"
pidfd: "true"
seccomp_allow_deny_syntax: "true"
seccomp_notify: "true"
seccomp_proxy_send_notify_fd: "true"
os_name: CentOS Linux
os_version: "7"
project: default
server: lxd
server_clustered: false
server_event_mode: full-mesh
server_name: osinfra-test-cluster-005-6099x
server_pid: 1020718
server_version: "4.24"
storage: btrfs
storage_version: 5.4.1
storage_supported_drivers:
- name: ceph
version: 15.2.14
remote: true
- name: btrfs
version: 5.4.1
remote: false
- name: cephfs
version: 15.2.14
remote: true
- name: dir
version: "1"
remote: false
- name: lvm
version: 2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.37.1
remote: false

Kernel version: Linux osinfra-test-cluster-005-6099x 3.10.0-1160.25.1.el7.x86_64 #1 SMP Wed Apr 28 21:49:45 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
LXC version:
LXD version:
Storage backend in use: btrfs

Issue description

When creating lxc containers concurrently, there would be a pool performance on assiging the IPv4 address of each containers let's take it to 20, there would be about 40-50 seconds delay after IPv6 address are fully assigned while IPv4 are emtpy.

+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
|                NAME                 |  STATE  | IPV4 |                     IPV6                      |   TYPE    | SNAPSHOTS |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| res44a3e1043840260d3a536dd88deee68a | STOPPED |      |                                               | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| res354ee1745cea9544612b9fa47fc24112 | STOPPED |      |                                               | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| resb2461b8e0881b7d733f960006bbcfa72 | STOPPED |      |                                               | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| tommylike1                          | RUNNING |      | fd42:2c15:68be:30b4:216:3eff:feca:ec94 (eth0) | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| tommylike2                          | RUNNING |      | fd42:2c15:68be:30b4:216:3eff:fed2:2769 (eth0) | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| tommylike3                          | RUNNING |      | fd42:2c15:68be:30b4:216:3eff:fed8:dd9f (eth0) | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| tommylike4                          | RUNNING |      | fd42:2c15:68be:30b4:216:3eff:fe6f:dfd (eth0)  | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| tommylike5                          | RUNNING |      | fd42:2c15:68be:30b4:216:3eff:fe55:7d1 (eth0)  | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| tommylike6                          | RUNNING |      | fd42:2c15:68be:30b4:216:3eff:fed4:14a2 (eth0) | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| tommylike7                          | RUNNING |      | fd42:2c15:68be:30b4:216:3eff:fe25:ba24 (eth0) | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| tommylike8                          | RUNNING |      | fd42:2c15:68be:30b4:216:3eff:fe79:b31b (eth0) | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| tommylike9                          | RUNNING |      | fd42:2c15:68be:30b4:216:3eff:fe21:11b6 (eth0) | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| tommylike10                         | RUNNING |      | fd42:2c15:68be:30b4:216:3eff:fe72:c023 (eth0) | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| tommylike11                         | RUNNING |      | fd42:2c15:68be:30b4:216:3eff:fe37:59e6 (eth0) | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| tommylike12                         | RUNNING |      | fd42:2c15:68be:30b4:216:3eff:fe18:222c (eth0) | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| tommylike13                         | RUNNING |      | fd42:2c15:68be:30b4:216:3eff:fe91:e8d5 (eth0) | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| tommylike14                         | RUNNING |      | fd42:2c15:68be:30b4:216:3eff:fe94:53c4 (eth0) | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| tommylike15                         | RUNNING |      | fd42:2c15:68be:30b4:216:3eff:fe8b:5aa2 (eth0) | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| tommylike16                         | RUNNING |      | fd42:2c15:68be:30b4:216:3eff:fe4a:94a9 (eth0) | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| tommylike17                         | RUNNING |      | fd42:2c15:68be:30b4:216:3eff:fe77:50ad (eth0) | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| tommylike18                         | RUNNING |      | fd42:2c15:68be:30b4:216:3eff:fe12:317b (eth0) | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| tommylike19                         | RUNNING |      | fd42:2c15:68be:30b4:216:3eff:fe0e:efb8 (eth0) | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+
| tommylike20                         | RUNNING |      | fd42:2c15:68be:30b4:216:3eff:fe48:798b (eth0) | CONTAINER | 0         |
+-------------------------------------+---------+------+-----------------------------------------------+-----------+-----------+

Steps to reproduce

Step one: launch 20 ubuntu containers

Information to attach

[ ] Any relevant kernel output (dmesg)
[ ] Container log (lxc info NAME --show-log)
[ ] Container configuration (lxc config show NAME --expanded)
[ ] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
[ ] Output of the client with --debug
[ ] Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)

stgraber commented 2 years ago

That's unlikely to be a LXD issue.

IPv4 only gets assigned once the container has started up enough to run systemd-networkd and have it run the DHCP client. This always takes a little while and likely quite a bit longer on a busy system such like the one you're describing.

IPv6 is instead handled directly by the kernel so doesn't need anything from the container itself and will appear almost immediately.

TommyLike commented 2 years ago

That's unlikely to be a LXD issue.

IPv4 only gets assigned once the container has started up enough to run systemd-networkd and have it run the DHCP client. This always takes a little while and likely quite a bit longer on a busy system such like the one you're describing.

IPv6 is instead handled directly by the kernel so doesn't need anything from the container itself and will appear almost immediately.

Apart from speeding up the container start up process, is there any other way to alleviate this? I mean it's much faster when fewer instances are launching. @stgraber

TommyLike commented 2 years ago

console log for my instance:

systemd v243-54.oe1 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy)
Detected virtualization lxc.
Detected architecture x86-64.
Failed to create symlink /sys/fs/cgroup/cpu: File exists
Failed to create symlink /sys/fs/cgroup/net_cls: File exists

Welcome to openEuler 20.03 (LTS-SP3)!

Initializing machine ID from random generator.
Couldn't move remaining userspace processes, ignoring: Invalid argument
/usr/lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly.
dev-.lxd\x2dmounts.mount: unit configures an IP firewall, but the local system does not support BPF/cgroup firewalling.
(This warning is only shown for the first unit using IP firewalling.)
[  OK  ] Created slice system-getty.slice.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[UNSUPP] Starting of Arbitrary Executable Fi…tem Automount Point not supported.
[  OK  ] Reached target Local Encrypted Volumes.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Reached target Slices.
[  OK  ] Reached target Swap.
[  OK  ] Listening on Process Core Dump Socket.
[  OK  ] Listening on initctl Compatibility Named Pipe.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
[  OK  ] Listening on udev Control Socket.
[  OK  ] Listening on udev Kernel Socket.
         Mounting Temporary Directory (/tmp)...
         Starting Journal Service...
         Starting Remount Root and Kernel File Systems...
         Starting Apply Kernel Variables...
         Starting udev Coldplug all Devices...
[  OK  ] Started Journal Service.
[  OK  ] Mounted Temporary Directory (/tmp).
[  OK  ] Started Remount Root and Kernel File Systems.
[  OK  ] Started Apply Kernel Variables.
[  OK  ] Started udev Coldplug all Devices.
         Starting Flush Journal to Persistent Storage...
         Starting Create System Users...
[  OK  ] Started Flush Journal to Persistent Storage.
[  OK  ] Started Create System Users.
         Starting Create Static Device Nodes in /dev...
[  OK  ] Started Create Static Device Nodes in /dev.
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
         Starting Restore /run/initramfs on shutdown...
         Starting Rebuild Dynamic Linker Cache...
         Starting Create Volatile Files and Directories...
         Starting udev Kernel Device Manager...
[  OK  ] Started udev Kernel Device Manager.
[  OK  ] Started Restore /run/initramfs on shutdown.
[  OK  ] Started Rebuild Dynamic Linker Cache.
[  OK  ] Started Create Volatile Files and Directories.
         Starting Security Auditing Service...
         Starting Rebuild Journal Catalog...
[FAILED] Failed to start Security Auditing Service.
See 'systemctl status auditd.service' for details.
[  OK  ] Started Rebuild Journal Catalog.
         Starting Update is Completed...
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Update is Completed.
[  OK  ] Stopped Security Auditing Service.
         Starting Security Auditing Service...
[FAILED] Failed to start Security Auditing Service.
See 'systemctl status auditd.service' for details.
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Stopped Security Auditing Service.
         Starting Security Auditing Service...
[FAILED] Failed to start Security Auditing Service.
See 'systemctl status auditd.service' for details.
[  OK  ] Reached target System Initialization.
[  OK  ] Started dnf makecache --timer.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target Timers.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Basic System.
[  OK  ] Started D-Bus System Message Bus.
         Starting Update RTC With System Clock...
         Starting LSB: Bring up/down networking...
         Starting Login Service...
[  OK  ] Started Update RTC With System Clock.
[  OK  ] Stopped Security Auditing Service.
         Starting Security Auditing Service...
[  OK  ] Started Login Service.
         Starting Hostname Service...
[FAILED] Failed to start Security Auditing Service.
See 'systemctl status auditd.service' for details.
[  OK  ] Started LSB: Bring up/down networking.
[  OK  ] Started Hostname Service.
[  OK  ] Reached target Network.
[  OK  ] Reached target Network is Online.
         Starting System Logging Service...
         Starting Permit User Sessions...
[  OK  ] Started Permit User Sessions.
[  OK  ] Stopped Security Auditing Service.
         Starting Security Auditing Service...
[  OK  ] Started Console Getty.
[  OK  ] Reached target Login Prompts.
[FAILED] Failed to start Security Auditing Service.
See 'systemctl status auditd.service' for details.
[  OK  ] Started System Logging Service.
[  OK  ] Started Command Scheduler.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Stopped Security Auditing Service.
         Starting Security Auditing Service...
[  OK  ] Started Update UTMP about System Runlevel Changes.
[FAILED] Failed to start Security Auditing Service.
See 'systemctl status auditd.service' for details.

stgraber commented 2 years ago

You can so some tweaks to your image to have less things start on boot. If using Ubuntu, you may want to try images:ubuntu/20.04 instead of ubuntu:20.04 as the images: (community image) is lighter and doesn't run cloud-init on first boot which may save you some time.

But overall, I'd expect it to be I/O and CPU contention slowing things down on startup which isn't something that we can really do anything about on the LXD side of things.

canonical / lxd

Pool performance on Assigin IPV4 address when concurrently launching containers #10142

Required information

Issue description

Steps to reproduce

Information to attach