stgraber
commented
1 year ago Sorry, can't easily load all your logs here, can you check your lxd.log to see if you're getting errors related to the memory cgroup?
Closed exsver closed 1 year ago
stgraber
commented
1 year ago Sorry, can't easily load all your logs here, can you check your lxd.log to see if you're getting errors related to the memory cgroup?
exsver
commented
1 year ago Sorry, can't easily load all your logs here, can you check your
lxd.logto see if you're getting errors related to the memory cgroup?
no errors by keyword "cgroup" in /var/snap/lxd/common/lxd/logs/lxd.log
part of container's log:
... lxc ct01 20221002063501.582 TRACE commands - ../src/src/lxc/commands.c:lxc_cmd_accept:2102 - Accepted new client as fd 13 on command server fd 9 lxc ct01 20221002063501.582 TRACE commands - ../src/src/lxc/commands.c:lxc_cmd_handler:2065 - Processed command get_state; cleaning up client fd 13 lxc ct01 20221002063501.582 TRACE mainloop - ../src/src/lxc/mainloop.c:disarm_handler:65 - Disarmed 13 for "lxc_cmd_handler" handler lxc ct01 20221002063501.582 TRACE commands - ../src/src/lxc/commands.c:lxc_cmd_cleanup_handler:2005 - Closing client fd 13 for "lxc_cmd_cleanup_handler" lxc ct01 20221002063501.582 TRACE mainloop - ../src/src/lxc/mainloop.c:delete_handler:79 - Deleted 13 for "lxc_cmd_handler" handler lxc 20221002063501.584 DEBUG commands - ../src/src/lxc/commands.c:lxc_cmd_rsp_recv_fds:148 - Command "get_state" received response lxc 20221002063501.584 TRACE commands - ../src/src/lxc/commands.c:lxc_cmd:521 - Opened new command socket connection fd 23 for command "get_state" lxc 20221002063501.584 DEBUG commands - ../src/src/lxc/commands.c:lxc_cmd_get_state:1063 - Container "ct01" is in "RUNNING" state lxc ct01 20221002063501.585 TRACE commands - ../src/src/lxc/commands.c:lxc_cmd_accept:2102 - Accepted new client as fd 13 on command server fd 9 lxc ct01 20221002063501.585 TRACE commands - ../src/src/lxc/commands.c:lxc_cmd_handler:2065 - Processed command get_state; cleaning up client fd 13 lxc ct01 20221002063501.585 TRACE mainloop - ../src/src/lxc/mainloop.c:disarm_handler:65 - Disarmed 13 for "lxc_cmd_handler" handler lxc ct01 20221002063501.585 TRACE commands - ../src/src/lxc/commands.c:lxc_cmd_cleanup_handler:2005 - Closing client fd 13 for "lxc_cmd_cleanup_handler" lxc ct01 20221002063501.585 TRACE mainloop - ../src/src/lxc/mainloop.c:delete_handler:79 - Deleted 13 for "lxc_cmd_handler" handler lxc 20221002063501.585 DEBUG commands - ../src/src/lxc/commands.c:lxc_cmd_rsp_recv_fds:148 - Command "get_state" received response lxc 20221002063501.585 TRACE commands - ../src/src/lxc/commands.c:lxc_cmd:521 - Opened new command socket connection fd 32 for command "get_state" lxc 20221002063501.585 DEBUG commands - ../src/src/lxc/commands.c:lxc_cmd_get_state:1063 - Container "ct01" is in "RUNNING" state lxc ct01 20221002063501.585 TRACE commands - ../src/src/lxc/commands.c:lxc_cmd_accept:2102 - Accepted new client as fd 13 on command server fd 9 lxc ct01 20221002063501.585 TRACE commands - ../src/src/lxc/commands.c:lxc_cmd_handler:2065 - Processed command get_limit_cgroup_fd; cleaning up client fd 13 lxc ct01 20221002063501.585 TRACE mainloop - ../src/src/lxc/mainloop.c:disarm_handler:65 - Disarmed 13 for "lxc_cmd_handler" handler lxc ct01 20221002063501.585 TRACE commands - ../src/src/lxc/commands.c:lxc_cmd_cleanup_handler:2005 - Closing client fd 13 for "lxc_cmd_cleanup_handler" lxc ct01 20221002063501.585 TRACE mainloop - ../src/src/lxc/mainloop.c:delete_handler:79 - Deleted 13 for "lxc_cmd_handler" handler lxc 20221002063501.585 DEBUG commands - ../src/src/lxc/commands.c:lxc_cmd_rsp_recv_fds:137 - Received exact number of file descriptors 1 == 1 for command "get_limit_cgroup_fd" lxc 20221002063501.585 TRACE commands - ../src/src/lxc/commands.c:lxc_cmd:521 - Opened new command socket connection fd 32 for command "get_limit_cgroup_fd" lxc 20221002063501.585 TRACE cgfsng - ../src/src/lxc/cgroups/cgfsng.c:cgroup_get:4035 - Reading memory.max from unified cgroup hierarchy lxc ct01 20221002063501.585 TRACE commands - ../src/src/lxc/commands.c:lxc_cmd_accept:2102 - Accepted new client as fd 13 on command server fd 9 lxc ct01 20221002063501.585 TRACE commands - ../src/src/lxc/commands.c:lxc_cmd_handler:2065 - Processed command get_state; cleaning up client fd 13 lxc ct01 20221002063501.585 TRACE mainloop - ../src/src/lxc/mainloop.c:disarm_handler:65 - Disarmed 13 for "lxc_cmd_handler" handler lxc ct01 20221002063501.585 TRACE commands - ../src/src/lxc/commands.c:lxc_cmd_cleanup_handler:2005 - Closing client fd 13 for "lxc_cmd_cleanup_handler" lxc ct01 20221002063501.585 TRACE mainloop - ../src/src/lxc/mainloop.c:delete_handler:79 - Deleted 13 for "lxc_cmd_handler" handler lxc 20221002063501.586 DEBUG commands - ../src/src/lxc/commands.c:lxc_cmd_rsp_recv_fds:148 - Command "get_state" received response lxc 20221002063501.586 TRACE commands - ../src/src/lxc/commands.c:lxc_cmd:521 - Opened new command socket connection fd 32 for command "get_state" lxc 20221002063501.586 DEBUG commands - ../src/src/lxc/commands.c:lxc_cmd_get_state:1063 - Container "ct01" is in "RUNNING" state lxc ct01 20221002063501.586 TRACE commands - ../src/src/lxc/commands.c:lxc_cmd_accept:2102 - Accepted new client as fd 13 on command server fd 9 lxc ct01 20221002063501.586 TRACE commands - ../src/src/lxc/commands.c:lxc_cmd_handler:2065 - Processed command get_limit_cgroup_fd; cleaning up client fd 13 lxc ct01 20221002063501.586 TRACE mainloop - ../src/src/lxc/mainloop.c:disarm_handler:65 - Disarmed 13 for "lxc_cmd_handler" handler lxc ct01 20221002063501.586 TRACE commands - ../src/src/lxc/commands.c:lxc_cmd_cleanup_handler:2005 - Closing client fd 13 for "lxc_cmd_cleanup_handler" lxc ct01 20221002063501.586 TRACE mainloop - ../src/src/lxc/mainloop.c:delete_handler:79 - Deleted 13 for "lxc_cmd_handler" handler lxc 20221002063501.586 DEBUG commands - ../src/src/lxc/commands.c:lxc_cmd_rsp_recv_fds:137 - Received exact number of file descriptors 1 == 1 for command "get_limit_cgroup_fd" lxc 20221002063501.586 TRACE commands - ../src/src/lxc/commands.c:lxc_cmd:521 - Opened new command socket connection fd 37 for command "get_limit_cgroup_fd" lxc 20221002063501.586 TRACE cgfsng - ../src/src/lxc/cgroups/cgfsng.c:cgroup_get:4035 - Reading memory.max from unified cgroup hierarchy ...
When metrics are absent cg.rw.Get(version, "memory", "memory.stat") returns ("", nil)
https://github.com/lxc/lxd/blob/master/lxd/cgroup/abstraction.go#L842 stats, err = cg.rw.Get(version, "memory", "memory.stat")
If you want I can provide ssh-access to test-server
stgraber
commented
1 year ago It would be interesting to track down the actual memory.stat file for the container and see if it's the kernel returning bad or empty data here.
You can get the container's PID with lxc info NAME, then after that look at /proc/PID/cgroup to get the cgroup path and finally look for that under /sys/fs/cgroup/memory to access the actual memory.stat file.
If that also shows up empty occasionally, then there's nothing we can do about it.
exsver
commented
1 year ago It would be interesting to track down the actual
memory.statfile for the container and see if it's the kernel returning bad or empty data here.You can get the container's PID with
lxc info NAME, then after that look at/proc/PID/cgroupto get the cgroup path and finally look for that under/sys/fs/cgroup/memoryto access the actualmemory.statfile.If that also shows up empty occasionally, then there's nothing we can do about it.
I tested this. The kernel always returns the correct memory.stat file.
exsver
commented
1 year ago Sorry, can't easily load all your logs here, can you check your
lxd.logto see if you're getting errors related to the memory cgroup?
Truncated logs: 2022-10-02_03-00_03-30_lxd.log 2022-10-02_06-30_06-40_lxd.log
2022-10-02_06-30_06-40_ct01.log 2022-10-02_03-00_03-20_ct05.log 2022-10-02_03-00_03-30_ct10.log 2022-10-02_03-00_03-20_ct19.log
MaximMonin
commented
1 year ago We tested it on 4 different servers with ubuntu 22.04 and lxd snap version with cgroup v2, and every server had metric drops. There are no messages in debug logs, so lxd didnt see any errors. On some servers there were drops in 1 container every 1-3 hours. On some - every 15-30 minutes with 1-4 drops. We spent 2 weeks to solve this, but failed,
tomponline
commented
1 year ago Hi @MaximMonin we've made some improvements to the cgroup handling for metrics in recent releases, please can you confirm if you still experience this on lxd >= 5.15
Thanks
MaximMonin
commented
1 year ago Right now we rewrited our metric collector w/o using lxd api. So we do not use it in production. Maybe we try it again later.
tomponline
commented
1 year ago OK, I'll close this for now, but do let us know if you experience the issue again and we can reopen.
Required information
root@lxd:~# lxc info config: {} api_extensions: - storage_zfs_remove_snapshots - container_host_shutdown_timeout - container_stop_priority - container_syscall_filtering - auth_pki - container_last_used_at - etag - patch - usb_devices - https_allowed_credentials - image_compression_algorithm - directory_manipulation - container_cpu_time - storage_zfs_use_refquota - storage_lvm_mount_options - network - profile_usedby - container_push - container_exec_recording - certificate_update - container_exec_signal_handling - gpu_devices - container_image_properties - migration_progress - id_map - network_firewall_filtering - network_routes - storage - file_delete - file_append - network_dhcp_expiry - storage_lvm_vg_rename - storage_lvm_thinpool_rename - network_vlan - image_create_aliases - container_stateless_copy - container_only_migration - storage_zfs_clone_copy - unix_device_rename - storage_lvm_use_thinpool - storage_rsync_bwlimit - network_vxlan_interface - storage_btrfs_mount_options - entity_description - image_force_refresh - storage_lvm_lv_resizing - id_map_base - file_symlinks - container_push_target - network_vlan_physical - storage_images_delete - container_edit_metadata - container_snapshot_stateful_migration - storage_driver_ceph - storage_ceph_user_name - resource_limits - storage_volatile_initial_source - storage_ceph_force_osd_reuse - storage_block_filesystem_btrfs - resources - kernel_limits - storage_api_volume_rename - macaroon_authentication - network_sriov - console - restrict_devlxd - migration_pre_copy - infiniband - maas_network - devlxd_events - proxy - network_dhcp_gateway - file_get_symlink - network_leases - unix_device_hotplug - storage_api_local_volume_handling - operation_description - clustering - event_lifecycle - storage_api_remote_volume_handling - nvidia_runtime - container_mount_propagation - container_backup - devlxd_images - container_local_cross_pool_handling - proxy_unix - proxy_udp - clustering_join - proxy_tcp_udp_multi_port_handling - network_state - proxy_unix_dac_properties - container_protection_delete - unix_priv_drop - pprof_http - proxy_haproxy_protocol - network_hwaddr - proxy_nat - network_nat_order - container_full - candid_authentication - backup_compression - candid_config - nvidia_runtime_config - storage_api_volume_snapshots - storage_unmapped - projects - candid_config_key - network_vxlan_ttl - container_incremental_copy - usb_optional_vendorid - snapshot_scheduling - snapshot_schedule_aliases - container_copy_project - clustering_server_address - clustering_image_replication - container_protection_shift - snapshot_expiry - container_backup_override_pool - snapshot_expiry_creation - network_leases_location - resources_cpu_socket - resources_gpu - resources_numa - kernel_features - id_map_current - event_location - storage_api_remote_volume_snapshots - network_nat_address - container_nic_routes - rbac - cluster_internal_copy - seccomp_notify - lxc_features - container_nic_ipvlan - network_vlan_sriov - storage_cephfs - container_nic_ipfilter - resources_v2 - container_exec_user_group_cwd - container_syscall_intercept - container_disk_shift - storage_shifted - resources_infiniband - daemon_storage - instances - image_types - resources_disk_sata - clustering_roles - images_expiry - resources_network_firmware - backup_compression_algorithm - ceph_data_pool_name - container_syscall_intercept_mount - compression_squashfs - container_raw_mount - container_nic_routed - container_syscall_intercept_mount_fuse - container_disk_ceph - virtual-machines - image_profiles - clustering_architecture - resources_disk_id - storage_lvm_stripes - vm_boot_priority - unix_hotplug_devices - api_filtering - instance_nic_network - clustering_sizing - firewall_driver - projects_limits - container_syscall_intercept_hugetlbfs - limits_hugepages - container_nic_routed_gateway - projects_restrictions - custom_volume_snapshot_expiry - volume_snapshot_scheduling - trust_ca_certificates - snapshot_disk_usage - clustering_edit_roles - container_nic_routed_host_address - container_nic_ipvlan_gateway - resources_usb_pci - resources_cpu_threads_numa - resources_cpu_core_die - api_os - container_nic_routed_host_table - container_nic_ipvlan_host_table - container_nic_ipvlan_mode - resources_system - images_push_relay - network_dns_search - container_nic_routed_limits - instance_nic_bridged_vlan - network_state_bond_bridge - usedby_consistency - custom_block_volumes - clustering_failure_domains - resources_gpu_mdev - console_vga_type - projects_limits_disk - network_type_macvlan - network_type_sriov - container_syscall_intercept_bpf_devices - network_type_ovn - projects_networks - projects_networks_restricted_uplinks - custom_volume_backup - backup_override_name - storage_rsync_compression - network_type_physical - network_ovn_external_subnets - network_ovn_nat - network_ovn_external_routes_remove - tpm_device_type - storage_zfs_clone_copy_rebase - gpu_mdev - resources_pci_iommu - resources_network_usb - resources_disk_address - network_physical_ovn_ingress_mode - network_ovn_dhcp - network_physical_routes_anycast - projects_limits_instances - network_state_vlan - instance_nic_bridged_port_isolation - instance_bulk_state_change - network_gvrp - instance_pool_move - gpu_sriov - pci_device_type - storage_volume_state - network_acl - migration_stateful - disk_state_quota - storage_ceph_features - projects_compression - projects_images_remote_cache_expiry - certificate_project - network_ovn_acl - projects_images_auto_update - projects_restricted_cluster_target - images_default_architecture - network_ovn_acl_defaults - gpu_mig - project_usage - network_bridge_acl - warnings - projects_restricted_backups_and_snapshots - clustering_join_token - clustering_description - server_trusted_proxy - clustering_update_cert - storage_api_project - server_instance_driver_operational - server_supported_storage_drivers - event_lifecycle_requestor_address - resources_gpu_usb - clustering_evacuation - network_ovn_nat_address - network_bgp - network_forward - custom_volume_refresh - network_counters_errors_dropped - metrics - image_source_project - clustering_config - network_peer - linux_sysctl - network_dns - ovn_nic_acceleration - certificate_self_renewal - instance_project_move - storage_volume_project_move - cloud_init - network_dns_nat - database_leader - instance_all_projects - clustering_groups - ceph_rbd_du - instance_get_full - qemu_metrics - gpu_mig_uuid - event_project - clustering_evacuation_live - instance_allow_inconsistent_copy - network_state_ovn - storage_volume_api_filtering - image_restrictions - storage_zfs_export - network_dns_records - storage_zfs_reserve_space - network_acl_log - storage_zfs_blocksize - metrics_cpu_seconds - instance_snapshot_never - certificate_token - instance_nic_routed_neighbor_probe - event_hub - agent_nic_config - projects_restricted_intercept - metrics_authentication - images_target_project - cluster_migration_inconsistent_copy - cluster_ovn_chassis - container_syscall_intercept_sched_setscheduler - storage_lvm_thinpool_metadata_size - storage_volume_state_total - instance_file_head - instances_nic_host_name - image_copy_profile - container_syscall_intercept_sysinfo - clustering_evacuation_mode - resources_pci_vpd - qemu_raw_conf - storage_cephfs_fscache - network_load_balancer - vsock_api - instance_ready_state - network_bgp_holdtime - storage_volumes_all_projects - metrics_memory_oom_total - storage_buckets - storage_buckets_create_credentials - metrics_cpu_effective_total - projects_networks_restricted_access - storage_buckets_local - loki api_status: stable api_version: "1.0" auth: trusted public: false auth_methods: - tls environment: addresses: [] architectures: - x86_64 - i686 certificate: | -----BEGIN CERTIFICATE----- MIIB+jCCAYCgAwIBAgIQa6goHxyQubqMHkFOjmt1/DAKBggqhkjOPQQDAzAxMRww GgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMREwDwYDVQQDDAhyb290QGx4ZDAe Fw0yMjA5MjgxMDM1NDZaFw0zMjA5MjUxMDM1NDZaMDExHDAaBgNVBAoTE2xpbnV4 Y29udGFpbmVycy5vcmcxETAPBgNVBAMMCHJvb3RAbHhkMHYwEAYHKoZIzj0CAQYF K4EEACIDYgAEmmbWmH/RNV12B58FEm+AjbXqzax9jf9veuGuox/4lh10w4nKxc9E dqh3r2Vkrn4kBLdapvCI4Wv5bHnrIdhaCIBcqECkjpTr9vm1XoTtXfAMHc6Nk3HV rQ4+H/MG24ZQo10wWzAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUH AwEwDAYDVR0TAQH/BAIwADAmBgNVHREEHzAdggNseGSHBH8AAAGHEAAAAAAAAAAA AAAAAAAAAAEwCgYIKoZIzj0EAwMDaAAwZQIxAOvCDZhrvwFu96Y5X87C+6btaK7m kJvmU69cBernZu0BzMRgtO3HRMPgbz63e1hkegIwVWSeZBQ9RJ3mBdIVqBBXvvIX iZS9jlrAN2Z7Pw4PfPI8ZB7K9cRMc6a8sza+b076 -----END CERTIFICATE----- certificate_fingerprint: 90c73a79f485544e0f7c11c5006d90cf031e8007ea689b77dcc3fbf93a25c656 driver: qemu | lxc driver_version: 7.1.0 | 5.0.1 firewall: nftables kernel: Linux kernel_architecture: x86_64 kernel_features: idmapped_mounts: "true" netnsid_getifaddrs: "true" seccomp_listener: "true" seccomp_listener_continue: "true" shiftfs: "false" uevent_injection: "true" unpriv_fscaps: "true" kernel_version: 5.15.0-48-generic lxc_features: cgroup2: "true" core_scheduling: "true" devpts_fd: "true" idmapped_mounts_v2: "true" mount_injection_file: "true" network_gateway_device_route: "true" network_ipvlan: "true" network_l2proxy: "true" network_phys_macvlan_mtu: "true" network_veth_router: "true" pidfd: "true" seccomp_allow_deny_syntax: "true" seccomp_notify: "true" seccomp_proxy_send_notify_fd: "true" os_name: Ubuntu os_version: "22.04" project: default server: lxd server_clustered: false server_event_mode: full-mesh server_name: lxd server_pid: 1042 server_version: "5.6" storage: dir storage_version: "1" storage_supported_drivers: - name: cephobject version: 15.2.16 remote: true - name: dir version: "1" remote: false - name: lvm version: 2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0 remote: false - name: zfs version: 2.1.4-0ubuntu0.1 remote: false - name: btrfs version: 5.4.1 remote: false - name: ceph version: 15.2.16 remote: true - name: cephfs version: 15.2.16 remote: trueIssue description
Sometimes API call /1.0/metrics (https://linuxcontainers.org/lxd/docs/latest/api/#/metrics/metrics_get) do not return memory.stat metrics for one or more random containers.
See the output in the attachments.
in file metrics_2022-10-02_03-10 are absent the following metrics:
in file metrics_2022-10-02_03-22 are absent the following metrics:
in file metrics_2022-10-02_06:35 are absent the following metrics:
Steps to reproduce
Information to attach
metrics_2022-10-02_03-10.txt metrics_2022-10-02_03-22.txt metrics_2022-10-02_06-35.txt
lxd_logs.partaa.tar.gz lxd_logs.partab.tar.gz lxd_logs.partac.tar.gz lxd_logs.partad.tar.gz lxd_logs.partae.tar.gz lxd_logs.partaf.tar.gz lxd_logs.partag.tar.gz to join tar.gz: