search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.32k stars 926 forks source link

Failure to push files over to some VMs #10336

Closed antifob closed 2 years ago

antifob commented 2 years ago

Required information

Issue description

Using lxc file push to copy a file over to a local VM fails. The exact same operation works to another, Ubuntu-based, VM.

Steps to reproduce

lxc file push /etc/hosts centos9/
lxc exec centos9 -- shutdown -r now
# loop, no success

$ qemu-system-x86_64 --version
QEMU emulator version 6.2.0 (Debian 1:6.2+dfsg-3)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers

Information to attach

# lxc monitor
location: none
metadata:
  context:
    listener: 9a88490a-3b02-44a9-9cdb-59b56e2819dc
    local: /var/lib/lxd/unix.socket
    remote: '@'
  level: debug
  message: Event listener server handler started
timestamp: "2022-04-28T05:20:01.851062652Z"
type: logging

location: none
metadata:
  context:
    ip: '@'
    method: GET
    protocol: unix
    url: /1.0
    username: root
  level: debug
  message: Handling API request
timestamp: "2022-04-28T05:20:04.540426399Z"
type: logging

location: none
metadata:
  context:
    ip: '@'
    method: POST
    protocol: unix
    url: /1.0/instances/centos9/files?path=%2Fhosts
    username: root
  level: debug
  message: Handling API request
timestamp: "2022-04-28T05:20:04.542424017Z"
type: logging

location: none
metadata:
  context:
    http_code: "200"
  level: debug
  message: "WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\":
    200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\":
    {\n\t\t\t\"config\": {\n\t\t\t\t\"images.auto_update_interval\": \"0\"\n\t\t\t},\n\t\t\t\"api_extensions\":
    [\n\t\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\t\"container_stop_priority\",\n\t\t\t\t\"container_syscall_filtering\",\n\t\t\t\t\"auth_pki\",\n\t\t\t\t\"container_last_used_at\",\n\t\
t\t\t\"etag\",\n\t\t\t\t\"patch\",\n\t\t\t\t\"usb_devices\",\n\t\t\t\t\"https_allowed_credentials\",\n\t\t\t\t\"image_compression_algorithm\",\n\t\t\t\t\"directory_manipulation\",\n\t\t\t\t\"container_cpu_time\",\n\t\t\t\t\"storage_zfs_use
_refquota\",\n\t\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\t\"network\",\n\t\t\t\t\"profile_usedby\",\n\t\t\t\t\"container_push\",\n\t\t\t\t\"container_exec_recording\",\n\t\t\t\t\"certificate_update\",\n\t\t\t\t\"container_exec_signal_h
andling\",\n\t\t\t\t\"gpu_devices\",\n\t\t\t\t\"container_image_properties\",\n\t\t\t\t\"migration_progress\",\n\t\t\t\t\"id_map\",\n\t\t\t\t\"network_firewall_filtering\",\n\t\t\t\t\"network_routes\",\n\t\t\t\t\"storage\",\n\t\t\t\t\"file
_delete\",\n\t\t\t\t\"file_append\",\n\t\t\t\t\"network_dhcp_expiry\",\n\t\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\t\"network_vlan\",\n\t\t\t\t\"image_create_aliases\",\n\t\t\t\t\"container_statele
ss_copy\",\n\t\t\t\t\"container_only_migration\",\n\t\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\t\"unix_device_rename\",\n\t\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\t\"network_vxlan_interface\",\n\t\t\
t\t\"storage_btrfs_mount_options\",\n\t\t\t\t\"entity_description\",\n\t\t\t\t\"image_force_refresh\",\n\t\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\t\"id_map_base\",\n\t\t\t\t\"file_symlinks\",\n\t\t\t\t\"container_push_target\",\n\t\t\t\
t\"network_vlan_physical\",\n\t\t\t\t\"storage_images_delete\",\n\t\t\t\t\"container_edit_metadata\",\n\t\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\t\"storage_driver_ceph\",\n\t\t\t\t\"storage_ceph_user_name\",\n\t\t\t\t\"res
ource_limits\",\n\t\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\t\"resources\",\n\t\t\t\t\"kernel_limits\",\n\t\t\t\t\"storage_api_volume_rename\
",\n\t\t\t\t\"macaroon_authentication\",\n\t\t\t\t\"network_sriov\",\n\t\t\t\t\"console\",\n\t\t\t\t\"restrict_devlxd\",\n\t\t\t\t\"migration_pre_copy\",\n\t\t\t\t\"infiniband\",\n\t\t\t\t\"maas_network\",\n\t\t\t\t\"devlxd_events\",\n\t\t
\t\t\"proxy\",\n\t\t\t\t\"network_dhcp_gateway\",\n\t\t\t\t\"file_get_symlink\",\n\t\t\t\t\"network_leases\",\n\t\t\t\t\"unix_device_hotplug\",\n\t\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\t\"operation_description\",\n\t\t\t\t\"
clustering\",\n\t\t\t\t\"event_lifecycle\",\n\t\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\t\"nvidia_runtime\",\n\t\t\t\t\"container_mount_propagation\",\n\t\t\t\t\"container_backup\",\n\t\t\t\t\"devlxd_images\",\n\t\t\t\t\"conta
iner_local_cross_pool_handling\",\n\t\t\t\t\"proxy_unix\",\n\t\t\t\t\"proxy_udp\",\n\t\t\t\t\"clustering_join\",\n\t\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\t\"network_state\",\n\t\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\t\
"container_protection_delete\",\n\t\t\t\t\"unix_priv_drop\",\n\t\t\t\t\"pprof_http\",\n\t\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\t\"network_hwaddr\",\n\t\t\t\t\"proxy_nat\",\n\t\t\t\t\"network_nat_order\",\n\t\t\t\t\"container_full\",\n\
t\t\t\t\"candid_authentication\",\n\t\t\t\t\"backup_compression\",\n\t\t\t\t\"candid_config\",\n\t\t\t\t\"nvidia_runtime_config\",\n\t\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\t\"storage_unmapped\",\n\t\t\t\t\"projects\",\n\t\t\t\t\"
candid_config_key\",\n\t\t\t\t\"network_vxlan_ttl\",\n\t\t\t\t\"container_incremental_copy\",\n\t\t\t\t\"usb_optional_vendorid\",\n\t\t\t\t\"snapshot_scheduling\",\n\t\t\t\t\"snapshot_schedule_aliases\",\n\t\t\t\t\"container_copy_project\"
,\n\t\t\t\t\"clustering_server_address\",\n\t\t\t\t\"clustering_image_replication\",\n\t\t\t\t\"container_protection_shift\",\n\t\t\t\t\"snapshot_expiry\",\n\t\t\t\t\"container_backup_override_pool\",\n\t\t\t\t\"snapshot_expiry_creation\",
\n\t\t\t\t\"network_leases_location\",\n\t\t\t\t\"resources_cpu_socket\",\n\t\t\t\t\"resources_gpu\",\n\t\t\t\t\"resources_numa\",\n\t\t\t\t\"kernel_features\",\n\t\t\t\t\"id_map_current\",\n\t\t\t\t\"event_location\",\n\t\t\t\t\"storage_a
pi_remote_volume_snapshots\",\n\t\t\t\t\"network_nat_address\",\n\t\t\t\t\"container_nic_routes\",\n\t\t\t\t\"rbac\",\n\t\t\t\t\"cluster_internal_copy\",\n\t\t\t\t\"seccomp_notify\",\n\t\t\t\t\"lxc_features\",\n\t\t\t\t\"container_nic_ipvl
an\",\n\t\t\t\t\"network_vlan_sriov\",\n\t\t\t\t\"storage_cephfs\",\n\t\t\t\t\"container_nic_ipfilter\",\n\t\t\t\t\"resources_v2\",\n\t\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\t\"container_syscall_intercept\",\n\t\t\t\t\"container_
disk_shift\",\n\t\t\t\t\"storage_shifted\",\n\t\t\t\t\"resources_infiniband\",\n\t\t\t\t\"daemon_storage\",\n\t\t\t\t\"instances\",\n\t\t\t\t\"image_types\",\n\t\t\t\t\"resources_disk_sata\",\n\t\t\t\t\"clustering_roles\",\n\t\t\t\t\"image
s_expiry\",\n\t\t\t\t\"resources_network_firmware\",\n\t\t\t\t\"backup_compression_algorithm\",\n\t\t\t\t\"ceph_data_pool_name\",\n\t\t\t\t\"container_syscall_intercept_mount\",\n\t\t\t\t\"compression_squashfs\",\n\t\t\t\t\"container_raw_m
ount\",\n\t\t\t\t\"container_nic_routed\",\n\t\t\t\t\"container_syscall_intercept_mount_fuse\",\n\t\t\t\t\"container_disk_ceph\",\n\t\t\t\t\"virtual-machines\",\n\t\t\t\t\"image_profiles\",\n\t\t\t\t\"clustering_architecture\",\n\t\t\t\t\"
resources_disk_id\",\n\t\t\t\t\"storage_lvm_stripes\",\n\t\t\t\t\"vm_boot_priority\",\n\t\t\t\t\"unix_hotplug_devices\",\n\t\t\t\t\"api_filtering\",\n\t\t\t\t\"instance_nic_network\",\n\t\t\t\t\"clustering_sizing\",\n\t\t\t\t\"firewall_dri
ver\",\n\t\t\t\t\"projects_limits\",\n\t\t\t\t\"container_syscall_intercept_hugetlbfs\",\n\t\t\t\t\"limits_hugepages\",\n\t\t\t\t\"container_nic_routed_gateway\",\n\t\t\t\t\"projects_restrictions\",\n\t\t\t\t\"custom_volume_snapshot_expiry
\",\n\t\t\t\t\"volume_snapshot_scheduling\",\n\t\t\t\t\"trust_ca_certificates\",\n\t\t\t\t\"snapshot_disk_usage\",\n\t\t\t\t\"clustering_edit_roles\",\n\t\t\t\t\"container_nic_routed_host_address\",\n\t\t\t\t\"container_nic_ipvlan_gateway\
",\n\t\t\t\t\"resources_usb_pci\",\n\t\t\t\t\"resources_cpu_threads_numa\",\n\t\t\t\t\"resources_cpu_core_die\",\n\t\t\t\t\"api_os\",\n\t\t\t\t\"container_nic_routed_host_table\",\n\t\t\t\t\"container_nic_ipvlan_host_table\",\n\t\t\t\t\"co
ntainer_nic_ipvlan_mode\",\n\t\t\t\t\"resources_system\",\n\t\t\t\t\"images_push_relay\",\n\t\t\t\t\"network_dns_search\",\n\t\t\t\t\"container_nic_routed_limits\",\n\t\t\t\t\"instance_nic_bridged_vlan\",\n\t\t\t\t\"network_state_bond_brid
ge\",\n\t\t\t\t\"usedby_consistency\",\n\t\t\t\t\"custom_block_volumes\",\n\t\t\t\t\"clustering_failure_domains\",\n\t\t\t\t\"resources_gpu_mdev\",\n\t\t\t\t\"console_vga_type\",\n\t\t\t\t\"projects_limits_disk\",\n\t\t\t\t\"network_type_m
acvlan\",\n\t\t\t\t\"network_type_sriov\",\n\t\t\t\t\"container_syscall_intercept_bpf_devices\",\n\t\t\t\t\"network_type_ovn\",\n\t\t\t\t\"projects_networks\",\n\t\t\t\t\"projects_networks_restricted_uplinks\",\n\t\t\t\t\"custom_volume_bac
kup\",\n\t\t\t\t\"backup_override_name\",\n\t\t\t\t\"storage_rsync_compression\",\n\t\t\t\t\"network_type_physical\",\n\t\t\t\t\"network_ovn_external_subnets\",\n\t\t\t\t\"network_ovn_nat\",\n\t\t\t\t\"network_ovn_external_routes_remove\",
\n\t\t\t\t\"tpm_device_type\",\n\t\t\t\t\"storage_zfs_clone_copy_rebase\",\n\t\t\t\t\"gpu_mdev\",\n\t\t\t\t\"resources_pci_iommu\",\n\t\t\t\t\"resources_network_usb\",\n\t\t\t\t\"resources_disk_address\",\n\t\t\t\t\"network_physical_ovn_in
gress_mode\",\n\t\t\t\t\"network_ovn_dhcp\",\n\t\t\t\t\"network_physical_routes_anycast\",\n\t\t\t\t\"projects_limits_instances\",\n\t\t\t\t\"network_state_vlan\",\n\t\t\t\t\"instance_nic_bridged_port_isolation\",\n\t\t\t\t\"instance_bulk_
state_change\",\n\t\t\t\t\"network_gvrp\",\n\t\t\t\t\"instance_pool_move\",\n\t\t\t\t\"gpu_sriov\",\n\t\t\t\t\"pci_device_type\",\n\t\t\t\t\"storage_volume_state\",\n\t\t\t\t\"network_acl\",\n\t\t\t\t\"migration_stateful\",\n\t\t\t\t\"disk
_state_quota\",\n\t\t\t\t\"storage_ceph_features\",\n\t\t\t\t\"projects_compression\",\n\t\t\t\t\"projects_images_remote_cache_expiry\",\n\t\t\t\t\"certificate_project\",\n\t\t\t\t\"network_ovn_acl\",\n\t\t\t\t\"projects_images_auto_update
\",\n\t\t\t\t\"projects_restricted_cluster_target\",\n\t\t\t\t\"images_default_architecture\",\n\t\t\t\t\"network_ovn_acl_defaults\",\n\t\t\t\t\"gpu_mig\",\n\t\t\t\t\"project_usage\",\n\t\t\t\t\"network_bridge_acl\",\n\t\t\t\t\"warnings\",
\n\t\t\t\t\"projects_restricted_backups_and_snapshots\",\n\t\t\t\t\"clustering_join_token\",\n\t\t\t\t\"clustering_description\",\n\t\t\t\t\"server_trusted_proxy\",\n\t\t\t\t\"clustering_update_cert\",\n\t\t\t\t\"storage_api_project\",\n\t
\t\t\t\"server_instance_driver_operational\",\n\t\t\t\t\"server_supported_storage_drivers\",\n\t\t\t\t\"event_lifecycle_requestor_address\",\n\t\t\t\t\"resources_gpu_usb\",\n\t\t\t\t\"clustering_evacuation\",\n\t\t\t\t\"network_ovn_nat_add
ress\",\n\t\t\t\t\"network_bgp\",\n\t\t\t\t\"network_forward\",\n\t\t\t\t\"custom_volume_refresh\",\n\t\t\t\t\"network_counters_errors_dropped\",\n\t\t\t\t\"metrics\",\n\t\t\t\t\"image_source_project\",\n\t\t\t\t\"clustering_config\",\n\t\
t\t\t\"network_peer\",\n\t\t\t\t\"linux_sysctl\",\n\t\t\t\t\"network_dns\",\n\t\t\t\t\"ovn_nic_acceleration\",\n\t\t\t\t\"certificate_self_renewal\",\n\t\t\t\t\"instance_project_move\",\n\t\t\t\t\"storage_volume_project_move\",\n\t\t\t\t\"
cloud_init\",\n\t\t\t\t\"network_dns_nat\",\n\t\t\t\t\"database_leader\",\n\t\t\t\t\"instance_all_projects\",\n\t\t\t\t\"clustering_groups\",\n\t\t\t\t\"ceph_rbd_du\",\n\t\t\t\t\"instance_get_full\",\n\t\t\t\t\"qemu_metrics\",\n\t\t\t\t\"g
pu_mig_uuid\",\n\t\t\t\t\"event_project\",\n\t\t\t\t\"clustering_evacuation_live\",\n\t\t\t\t\"instance_allow_inconsistent_copy\",\n\t\t\t\t\"network_state_ovn\",\n\t\t\t\t\"storage_volume_api_filtering\",\n\t\t\t\t\"image_restrictions\",\
n\t\t\t\t\"storage_zfs_export\",\n\t\t\t\t\"network_dns_records\",\n\t\t\t\t\"storage_zfs_reserve_space\",\n\t\t\t\t\"network_acl_log\",\n\t\t\t\t\"storage_zfs_blocksize\",\n\t\t\t\t\"metrics_cpu_seconds\",\n\t\t\t\t\"instance_snapshot_nev
er\",\n\t\t\t\t\"certificate_token\",\n\t\t\t\t\"instance_nic_routed_neighbor_probe\",\n\t\t\t\t\"event_hub\",\n\t\t\t\t\"agent_nic_config\",\n\t\t\t\t\"projects_restricted_intercept\",\n\t\t\t\t\"metrics_authentication\",\n\t\t\t\t\"image
s_target_project\",\n\t\t\t\t\"cluster_migration_inconsistent_copy\",\n\t\t\t\t\"cluster_ovn_chassis\",\n\t\t\t\t\"container_syscall_intercept_sched_setscheduler\",\n\t\t\t\t\"storage_lvm_thinpool_metadata_size\"\n\t\t\t],\n\t\t\t\"api_sta
tus\":
    \"stable\",\n\t\t\t\"api_version\": \"1.0\",\n\t\t\t\"auth\": \"trusted\",\n\t\t\t\"public\":
    false,\n\t\t\t\"auth_methods\": [\n\t\t\t\t\"tls\"\n\t\t\t],\n\t\t\t\"environment\":
    {\n\t\t\t\t\"addresses\": [],\n\t\t\t\t\"architectures\": [\n\t\t\t\t\t\"x86_64\",\n\t\t\t\t\t\"i686\"\n\t\t\t\t],\n\t\t\t\t\"certificate\":
    \"-----BEGIN CERTIFICATE-----\\nREDACTED\\n-----END
    CERTIFICATE-----\\n\",\n\t\t\t\t\"certificate_fingerprint\": \"52df3b4f248b8f9bc350ded35c8a553ebe9f0551930eb83052656040a558ee75\",\n\t\t\t\t\"driver\":
    \"lxc | qemu\",\n\t\t\t\t\"driver_version\": \"4.0.11 | 6.2.0\",\n\t\t\t\t\"firewall\":
    \"xtables\",\n\t\t\t\t\"kernel\": \"Linux\",\n\t\t\t\t\"kernel_architecture\":
    \"x86_64\",\n\t\t\t\t\"kernel_features\": {\n\t\t\t\t\t\"idmapped_mounts\": \"true\",\n\t\t\t\t\t\"netnsid_getifaddrs\":
    \"true\",\n\t\t\t\t\t\"seccomp_listener\": \"true\",\n\t\t\t\t\t\"seccomp_listener_continue\":
    \"true\",\n\t\t\t\t\t\"shiftfs\": \"false\",\n\t\t\t\t\t\"uevent_injection\":
    \"true\",\n\t\t\t\t\t\"unpriv_fscaps\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"kernel_version\":
    \"5.15.0-3-amd64\",\n\t\t\t\t\"lxc_features\": {\n\t\t\t\t\t\"cgroup2\": \"true\",\n\t\t\t\t\t\"core_scheduling\":
    \"true\",\n\t\t\t\t\t\"devpts_fd\": \"true\",\n\t\t\t\t\t\"idmapped_mounts_v2\":
    \"true\",\n\t\t\t\t\t\"mount_injection_file\": \"true\",\n\t\t\t\t\t\"network_gateway_device_route\":
    \"true\",\n\t\t\t\t\t\"network_ipvlan\": \"true\",\n\t\t\t\t\t\"network_l2proxy\":
    \"true\",\n\t\t\t\t\t\"network_phys_macvlan_mtu\": \"true\",\n\t\t\t\t\t\"network_veth_router\":
    \"true\",\n\t\t\t\t\t\"pidfd\": \"true\",\n\t\t\t\t\t\"seccomp_allow_deny_syntax\":
    \"true\",\n\t\t\t\t\t\"seccomp_notify\": \"true\",\n\t\t\t\t\t\"seccomp_proxy_send_notify_fd\":
    \"true\"\n\t\t\t\t},\n\t\t\t\t\"os_name\": \"Debian GNU/Linux\",\n\t\t\t\t\"os_version\":
    \"\",\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"server\": \"lxd\",\n\t\t\t\t\"server_clustered\":
    false,\n\t\t\t\t\"server_event_mode\": \"full-mesh\",\n\t\t\t\t\"server_name\":
    \"x1g9\",\n\t\t\t\t\"server_pid\": 7650,\n\t\t\t\t\"server_version\": \"5.0.0\",\n\t\t\t\t\"storage\":
    \"dir\",\n\t\t\t\t\"storage_version\": \"1\",\n\t\t\t\t\"storage_supported_drivers\":
    [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"lvm\",\n\t\t\t\t\t\t\"Version\": \"2.03.11(2)
    (2021-01-08) / 1.02.175 (2021-01-08) / 4.45.0\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\":
    \"dir\",\n\t\t\t\t\t\t\"Version\": \"1\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t}\n\t\t\t\t]\n\t\t\t}\n\t\t}\n\t}"
timestamp: "2022-04-28T05:20:04.541991281Z"
type: logging

location: none
metadata:
  context:
    instance: centos9
    project: default
  level: debug
  message: MountInstance started
timestamp: "2022-04-28T05:20:04.545818619Z"
type: logging

location: none
metadata:
  context:
    instance: centos9
    project: default
  level: debug
  message: MountInstance finished
timestamp: "2022-04-28T05:20:04.546673616Z"
type: logging

location: none
metadata:
  context:
    instance: centos9
    project: default
  level: debug
  message: UnmountInstance started
timestamp: "2022-04-28T05:20:04.546798985Z"
type: logging

location: none
metadata:
  context:
    driver: dir
    pool: default
    refCount: "1"
    volName: centos9
  level: debug
  message: Skipping unmount as in use
timestamp: "2022-04-28T05:20:04.547552339Z"
type: logging

location: none
metadata:
  context:
    instance: centos9
    project: default
  level: debug
  message: UnmountInstance finished
timestamp: "2022-04-28T05:20:04.547577764Z"
type: logging

location: none
metadata:
  context:
    http_code: "500"
  level: debug
  message: "Error Response\n\t{\n\t\t\"type\": \"error\",\n\t\t\"status\": \"\",\n\t\t\"status_code\":
    0,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 500,\n\t\t\"error\": \"Dialing
    failed: expected status code 101 got 404\",\n\t\t\"metadata\": null\n\t}"
timestamp: "2022-04-28T05:20:04.597089693Z"
type: logging
stgraber commented 2 years ago

For one thing, you probably meant to push to /etc/hosts and not /hosts :)

The error showed here would make sense if the agent in the VM was outdated which shouldn't be possible if the VM got restarted since the host was moved to 5.0. That's unless the VM agent binary was somehow manually installed and is running an old version?

stgraber@dakara:~$ lxc launch images:centos/9-Stream centos9 --vm
Creating centos9
Starting centos9                            
stgraber@dakara:~$ lxc file push /etc/hosts centos9/
stgraber@dakara:~$ lxc exec centos9 bash                           
[root@centos9 ~]# cat /hosts
127.0.0.1   localhost
127.0.1.1   dakara

172.17.250.151 ceph01
172.17.250.103 ceph02
172.17.250.162 ceph03

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
[root@centos9 ~]#
antifob commented 2 years ago

I had to triple (I already had) check what I wrote, my command and its output. Unless I'm missing the point, lxc file push /etc/hosts centos9/ does, indeed, produce /1.0/instances/centos9/files?path=%2Fhosts. I didn't check the API, but I would assume the %2fhosts is the target. :)

Anyway, I had discarded the possibility of an outdated lxd-agent already. The image dates from March 1st from images: and the agent has not been tampered with in any way. In fact, its hash were/is the same.

# lxc info centos9
Name: centos9
Status: RUNNING
Type: virtual-machine
Architecture: x86_64
PID: 8404
Created: 2022/03/01 06:24 UTC
Last Used: 2022/04/28 05:17 UTC
[...]

# lxc exec centos9 -- sha1sum /run/lxd_agent/lxd-agent
acadfe9ee6ab461c5bd39ef29182ea60d89bb240  /run/lxd_agent/lxd-agent
# lxc exec bullseye -- sha1sum /run/lxd_agent/lxd-agent
acadfe9ee6ab461c5bd39ef29182ea60d89bb240  /run/lxd_agent/lxd-agent

I also ruled out the following:

# lxc exec centos9 -- sestatus
SELinux status:                 disabled
stgraber commented 2 years ago

@antifob what happens if you run the same commands I did?

antifob commented 2 years ago 
# lxc launch --vm images:centos/9-Stream c9
Creating c9
Starting c9
# lxc file push /etc/hosts c9/
# lxc exec c9 -- cat /hosts
127.0.0.1       localhost
::1             localhost ip6-localhost ip6-loopback
ff02::1         ip6-allnodes

# lxc exec c9 -- sha1sum /run/lxd_agent/lxd-agent
acadfe9ee6ab461c5bd39ef29182ea60d89bb240  /run/lxd_agent/lxd-agent
stgraber commented 2 years ago

And on the broken VM, you've confirmed that sha256sum /proc/$(pgrep lxd-agent)/exe shows the same hash too?

antifob commented 2 years ago

Good catch @stgraber

lxd-agent.service was running /usr/local/bin/lxd-agent. If I may, could you tell me if it was ever stored there? I'm undecisively tracing this back to a potential issue with SELinux at the time and wondering if I should investigate more than destroying the VM.

[root@localhost ~]# stat /usr/local/bin/lxd-agent
  File: /usr/local/bin/lxd-agent
  Size: 15764335        Blocks: 30792      IO Block: 4096   regular file
Device: 802h/2050d      Inode: 20522       Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-04-29 14:23:25.333000000 +0000
Modify: 2022-03-01 06:31:50.717000000 +0000
Change: 2022-03-01 06:31:50.721000000 +0000
 Birth: 2022-03-01 06:31:50.701000000 +0000
[root@localhost ~]# sha256sum /usr/local/bin/lxd-agent
89bbe4e5607e99df52eaf36d9fea75010d546d47c6e981a8bed204d53f1a7e9f  /usr/local/bin/lxd-agent
stgraber commented 2 years ago

We never put the agent at that location. Our scripts have always loaded it from the 9p or virtiofs share specifically to avoid such issues :)