muzammil-iftikhar commented 2 years ago

Issue description

I am unable to launch/start any container with a same error every time.

Error: Failed to run: /usr/bin/lxd forkstart selected-panda /var/lib/lxd/containers /var/log/lxd/selected-panda/lxc.conf:
Try `lxc info --show-log local:selected-panda` for more info

Required information

Distribution: Manjaro
Distribution version: 21.2.6
Kernel version: 5.17.9-1
LXC version: 5.1
LXD version: 5.1

The output of "lxc info" or if that fails:


config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- instances_nic_host_name
- image_copy_profile
- container_syscall_intercept_sysinfo
- clustering_evacuation_mode
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
addresses: []
architectures:
- x86_64
- i686
certificate: |
-----BEGIN CERTIFICATE-----
MIICEjCCAZigAwIBAgIQe/cUUBOJgdkhE/F/cjofOzAKBggqhkjOPQQDAzA5MRww
GgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRkwFwYDVQQDDBByb290QG1hbmph
cm8ta2RlMB4XDTIyMDYwMjIyMDAzMFoXDTMyMDUzMDIyMDAzMFowOTEcMBoGA1UE
ChMTbGludXhjb250YWluZXJzLm9yZzEZMBcGA1UEAwwQcm9vdEBtYW5qYXJvLWtk
ZTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAiKAK4EaMCCUsjW8KumrQO9R7uAjiIU
o69lBkdZknqRHd2E099GMo2qJsoLOpQ+UNzMfEctNJ1/UGUvBr0mlrrrSxiCeZ0l
URRhHqGt7CFDHw3Ti2KZwVrW/l4iMt0Y0qNlMGMwDgYDVR0PAQH/BAQDAgWgMBMG
A1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwLgYDVR0RBCcwJYILbWFu
amFyby1rZGWHBH8AAAGHEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDaAAw
ZQIwfn/NbFAwT/5+o+nUl2ToDvYtEee/IpG9/T63225lg2kCdJDfkjeOSpJYuPpj
QyZVAjEA9TrYIJdZBeTV1gUkqWsoqljAqldUnPH0HQgwOjgoEnU8IKHw/reB4SlH
2Rx8miMI
-----END CERTIFICATE-----
certificate_fingerprint: e11619c780c0f074acc0487957641e74a3e9acbf16d843c1c7d1f13cdaccaac4
driver: lxc
driver_version: 4.0.12
firewall: nftables
kernel: Linux
kernel_architecture: x86_64
kernel_features:
idmapped_mounts: "true"
netnsid_getifaddrs: "true"
seccomp_listener: "true"
seccomp_listener_continue: "true"
shiftfs: "false"
uevent_injection: "true"
unpriv_fscaps: "true"
kernel_version: 5.17.9-1-MANJARO
lxc_features:
cgroup2: "true"
core_scheduling: "true"
devpts_fd: "true"
idmapped_mounts_v2: "true"
mount_injection_file: "true"
network_gateway_device_route: "true"
network_ipvlan: "true"
network_l2proxy: "true"
network_phys_macvlan_mtu: "true"
network_veth_router: "true"
pidfd: "true"
seccomp_allow_deny_syntax: "true"
seccomp_notify: "true"
seccomp_proxy_send_notify_fd: "true"
os_name: Manjaro Linux
os_version: ""
project: default
server: lxd
server_clustered: false
server_event_mode: full-mesh
server_name: manjaro-kde
server_pid: 60663
server_version: "5.1"
storage: dir
storage_version: "1"
storage_supported_drivers:
- name: btrfs
version: "5.17"
remote: false
- name: dir
version: "1"
remote: false
- name: lvm
version: 2.03.15(2) (2022-02-07) / 1.02.183 (2022-02-07) / 4.45.0
remote: false

# Steps to reproduce

` lxc launch ubuntu:20.04`
`lxc start selected-panda`

# Information to attach

lxc info --show-log selected-panda

Name: selected-panda Status: STOPPED Type: container Architecture: x86_64 Created: 2022/06/03 03:48 PKT Last Used: 2022/06/03 04:01 PKT

Log:

lxc selected-panda 20220602230156.839 ERROR conf - conf.c:lxc_map_ids:3668 - newuidmap failed to write mapping "": newuidmap 70171 0 1000000 1000000000 lxc selected-panda 20220602230156.839 ERROR start - start.c:lxc_spawn:1791 - Failed to set up id mapping. lxc selected-panda 20220602230156.839 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:877 - Received container state "ABORTING" instead of "RUNNING" lxc selected-panda 20220602230156.840 ERROR start - start.c:__lxc_start:2074 - Failed to spawn container "selected-panda" lxc selected-panda 20220602230156.841 WARN start - start.c:lxc_abort:1039 - No such process - Failed to send SIGKILL via pidfd 17 for process 70171 lxc 20220602230201.871 ERROR af_unix - af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response lxc 20220602230201.871 ERROR commands - commands.c:lxc_cmd_rsp_recv_fds:127 - Failed to receive file descriptors for command "get_state"

cat /var/log/lxd/lxd.log

time="2022-06-03T03:00:30+05:00" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored" time="2022-06-03T03:00:30+05:00" level=warning msg=" - Couldn't find the CGroup network priority controller, network priority will be ignored" time="2022-06-03T03:00:30+05:00" level=warning msg="Instance type not operational" driver=qemu err="QEMU command not available for CPU architecture" type=virtual-machine time="2022-06-03T03:25:24+05:00" level=error msg="Failed starting container" action=start created="2022-06-02 22:25:14.836441533 +0000 UTC" ephemeral=false instance=meet-fowl instanceType=container project=default stateful=false used="1970-01-01 00:00:00 +0000 UTC" time="2022-06-03T03:33:52+05:00" level=error msg="Failed starting container" action=start created="2022-06-02 22:25:14.836441533 +0000 UTC" ephemeral=false instance=meet-fowl instanceType=container project=default stateful=false used="2022-06-02 22:25:19.420341917 +0000 UTC" time="2022-06-03T03:36:05+05:00" level=error msg="Failed starting container" action=start created="2022-06-02 22:25:14.836441533 +0000 UTC" ephemeral=false instance=meet-fowl instanceType=container project=default stateful=false used="2022-06-02 22:33:47.113256035 +0000 UTC" time="2022-06-03T03:48:11+05:00" level=error msg="Failed starting container" action=start created="2022-06-02 22:48:01.346918967 +0000 UTC" ephemeral=false instance=selected-panda instanceType=container project=default stateful=false used="1970-01-01 00:00:00 +0000 UTC" time="2022-06-03T04:02:01+05:00" level=error msg="Failed starting container" action=start created="2022-06-02 22:48:01.346918967 +0000 UTC" ephemeral=false instance=selected-panda instanceType=container project=default stateful=false used="2022-06-02 22:48:06.321352824 +0000 UTC"

stgraber commented 2 years ago

lxc selected-panda 20220602230156.839 ERROR conf - conf.c:lxc_map_ids:3668 - newuidmap failed to write mapping "": newuidmap 70171 0 1000000 1000000000

This suggests you're using a native package (not the snap) and that your system has misconfigured /etc/subuid or /etc/subgid files (those aren't processed when using the snap).

Make sure that both /etc/subuid and /etc/subgid have a single map for the root user, that the map is the same in uid and gid and that it is made of at least 65536 uids and gids. Then restart the LXD daemon (it reads the files at startup) and things should work better.

muzammil-iftikhar commented 2 years ago

thanks @stgraber. For anyone stumbling upon this thread in future with similar issue, please follow this thread on how to exactly achieve what's suggested by stgraber. https://wiki.archlinux.org/title/Linux_Containers#Enable_support_to_run_unprivilegedcontainers(optional)

canonical / lxd

Fail to launch any container #10503

Issue description

Required information