mcondarelli commented 8 months ago

Required information

Distribution: Debian
Distribution version: Debian GNU/Linux 12 (bookworm)

The output of "incus info" :

root@incus:~# incus info
config:
core.https_address: :8443
images.auto_update_interval: "15"
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- network_sriov
- console
- restrict_dev_incus
- migration_pre_copy
- infiniband
- dev_incus_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- dev_incus_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- backup_compression
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- instances_nic_host_name
- image_copy_profile
- container_syscall_intercept_sysinfo
- clustering_evacuation_mode
- resources_pci_vpd
- qemu_raw_conf
- storage_cephfs_fscache
- network_load_balancer
- vsock_api
- instance_ready_state
- network_bgp_holdtime
- storage_volumes_all_projects
- metrics_memory_oom_total
- storage_buckets
- storage_buckets_create_credentials
- metrics_cpu_effective_total
- projects_networks_restricted_access
- storage_buckets_local
- loki
- acme
- internal_metrics
- cluster_join_token_expiry
- remote_token_expiry
- init_preseed
- storage_volumes_created_at
- cpu_hotplug
- projects_networks_zones
- network_txqueuelen
- cluster_member_state
- instances_placement_scriptlet
- storage_pool_source_wipe
- zfs_block_mode
- instance_generation_id
- disk_io_cache
- amd_sev
- storage_pool_loop_resize
- migration_vm_live
- ovn_nic_nesting
- oidc
- network_ovn_l3only
- ovn_nic_acceleration_vdpa
- cluster_healing
- instances_state_total
- auth_user
- security_csm
- instances_rebuild
- numa_cpu_placement
- custom_volume_iso
- network_allocations
- zfs_delegate
- storage_api_remote_volume_snapshot_copy
- operations_get_query_all_projects
- metadata_configuration
- syslog_socket
- event_lifecycle_name_and_project
- instances_nic_limits_priority
- disk_initial_volume_configuration
- operation_wait
- image_restriction_privileged
- cluster_internal_custom_volume_copy
- disk_io_bus
- storage_cephfs_create_missing
- instance_move_config
- ovn_ssl_config
- certificate_description
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
auth_user_name: root
auth_user_method: unix
environment:
addresses:
- 192.168.7.213:8443
- 192.168.9.1:8443
- 10.127.15.1:8443
architectures:
- x86_64
- i686
certificate: |
-----BEGIN CERTIFICATE-----
MIIB/DCCAYGgAwIBAgIRAI2fyqKGVJOVIB5kEamH/xEwCgYIKoZIzj0EAwMwMDEZ
MBcGA1UEChMQTGludXggQ29udGFpbmVyczETMBEGA1UEAwwKcm9vdEBpbmN1czAe
Fw0yNDAxMDcyMzE4NDNaFw0zNDAxMDQyMzE4NDNaMDAxGTAXBgNVBAoTEExpbnV4
IENvbnRhaW5lcnMxEzARBgNVBAMMCnJvb3RAaW5jdXMwdjAQBgcqhkjOPQIBBgUr
gQQAIgNiAASvfB/Je7qfBacrjc3bUxXwmWxyr/mzRmNXjP9/ldqpMfZ4dIPwNlNO
7QGN4Kdikmo7tpE878J/LK5Mu8N6tEepXCb/sln/yKSyd2UbsekX3Fo0fd9S6PFC
jaDWgbND+bmjXzBdMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcD
ATAMBgNVHRMBAf8EAjAAMCgGA1UdEQQhMB+CBWluY3VzhwR/AAABhxAAAAAAAAAA
AAAAAAAAAAABMAoGCCqGSM49BAMDA2kAMGYCMQCMEn4TRrq13d8RIP7m0jzak19C
0TwGLZOe2dBVJoqRPlHrH9pE45wDX3Ml6YIeQY8CMQCcD3744szsGPsUJeRNULOj
qxVYLmIuaPtRcfBCnbelpJ5+MDU3llGbAjiDwuHKoq4=
-----END CERTIFICATE-----
certificate_fingerprint: 9351459b6c8ba62d3e29b8b9ef050d82920af55d47bfadb1144eff6ed6e93472
driver: lxc | qemu
driver_version: 5.0.3 | 8.1.3
firewall: nftables
kernel: Linux
kernel_architecture: x86_64
kernel_features:
idmapped_mounts: "true"
netnsid_getifaddrs: "true"
seccomp_listener: "true"
seccomp_listener_continue: "true"
uevent_injection: "true"
unpriv_fscaps: "true"
kernel_version: 6.1.0-17-amd64
lxc_features:
cgroup2: "true"
core_scheduling: "true"
devpts_fd: "true"
idmapped_mounts_v2: "true"
mount_injection_file: "true"
network_gateway_device_route: "true"
network_ipvlan: "true"
network_l2proxy: "true"
network_phys_macvlan_mtu: "true"
network_veth_router: "true"
pidfd: "true"
seccomp_allow_deny_syntax: "true"
seccomp_notify: "true"
seccomp_proxy_send_notify_fd: "true"
os_name: Debian GNU/Linux
os_version: "12"
project: default
server: incus
server_clustered: false
server_event_mode: full-mesh
server_name: incus
server_pid: 754
server_version: "0.4"
storage: zfs
storage_version: 2.1.11-1
storage_supported_drivers:
- name: zfs
version: 2.1.11-1
remote: false
- name: btrfs
version: "6.2"
remote: false
- name: ceph
version: 16.2.11
remote: true
- name: cephfs
version: 16.2.11
remote: true
- name: cephobject
version: 16.2.11
remote: true
- name: dir
version: "1"
remote: false
- name: lvm
version: 2.03.16(2) (2022-05-18) / 1.02.185 (2022-05-18) / 4.47.0
remote: false

Issue description

I have a WiFi adapter (USB) I pass over to a container (standard images:openwrt/23.05). First time everything is fine but if I stop container adapter is not freed and any attempt to reuse it fails. See below for a commented log. Attatched incusd.log reports it was unable to detach adapter.

Steps to reproduce

## server is rebooted
mcon@cinderella:~$ ssh root@incus reboot

## WiFi card (wlx00c0cab2bd76) is visible
mcon@cinderella:~$ ssh root@incus -- ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 60:02:92:57:66:c1 brd ff:ff:ff:ff:ff:ff
3: enxa0cec8b43055: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether a0:ce:c8:b4:30:55 brd ff:ff:ff:ff:ff:ff
4: wlx00c0cab2bd76: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:c0:ca:b2:bd:76 brd ff:ff:ff:ff:ff:ff
5: ORANGE: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 00:16:3e:ed:dd:b3 brd ff:ff:ff:ff:ff:ff
6: incusbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 00:16:3e:54:c8:5c brd ff:ff:ff:ff:ff:ff
8: vethee6b197a@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master incusbr0 state UP mode DEFAULT group default qlen 1000
    link/ether de:d4:99:67:26:1f brd ff:ff:ff:ff:ff:ff link-netnsid 0
10: vethae9bd16d@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master incusbr0 state UP mode DEFAULT group default qlen 1000
    link/ether ae:89:33:8b:97:07 brd ff:ff:ff:ff:ff:ff link-netnsid 1
12: vethe3470c1f@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ORANGE state UP mode DEFAULT group default qlen 1000
    link/ether ba:4b:6d:be:79:49 brd ff:ff:ff:ff:ff:ff link-netnsid 2

## this is instance configuration, notice `wlan0` device
mcon@cinderella:~$ incus config show incus:openwrt --expanded
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Openwrt 23.05 amd64 (20240113_11:57)
  image.os: Openwrt
  image.release: "23.05"
  image.serial: "20240113_11:57"
  image.type: squashfs
  image.variant: default
  volatile.base_image: 632ab890e8c9ec28ccd47b971f64e0dda98c235687f5cfc2486a30a6b231139f
  volatile.cloud-init.instance-id: cd96c3f7-dee9-4d38-9006-927fa8cd0fca
  volatile.eth0.hwaddr: 00:16:3e:f2:5a:6f
  volatile.eth0.name: eth0
  volatile.eth1.hwaddr: 00:16:3e:57:99:42
  volatile.eth1.name: eth1
  volatile.eth2.hwaddr: 00:16:3e:ed:92:d6
  volatile.eth2.name: eth2
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"IsUID":true,"IsGID":false,"HostID":1000000,"NSID":0,"MapRange":1000000000},{"IsUID":false,"IsGID":true,"HostID":1000000,"NSID":0,"MapRange":1000000000}]'
  volatile.idmap.next: '[{"IsUID":true,"IsGID":false,"HostID":1000000,"NSID":0,"MapRange":1000000000},{"IsUID":false,"IsGID":true,"HostID":1000000,"NSID":0,"MapRange":1000000000}]'
  volatile.last_state.idmap: '[{"IsUID":true,"IsGID":false,"HostID":1000000,"NSID":0,"MapRange":1000000000},{"IsUID":false,"IsGID":true,"HostID":1000000,"NSID":0,"MapRange":1000000000}]'
  volatile.last_state.power: STOPPED
  volatile.uuid: 9bfb30dd-4ac8-4d48-83f3-acb208038603
  volatile.uuid.generation: 9bfb30dd-4ac8-4d48-83f3-acb208038603
  volatile.wlan0.host_name: wlx00c0cab2bd76
  volatile.wlan0.last_state.created: "false"
  volatile.wlan0.last_state.hwaddr: 00:c0:ca:b2:bd:76
  volatile.wlan0.last_state.mtu: "1500"
devices:
  eth0:
    nictype: macvlan
    parent: enp2s0
    type: nic
  eth1:
    nictype: macvlan
    parent: enxa0cec8b43055
    type: nic
  eth2:
    network: ORANGE
    type: nic
  root:
    path: /
    pool: default
    type: disk
  wlan0:
    name: wlan0
    nictype: physical
    parent: wlx00c0cab2bd76
    type: nic
ephemeral: false
profiles:
- default
stateful: false
description: OpenWrt firewall/router

## instance starts normally
mcon@cinderella:~$ incus start incus:openwrt

## after a while I stop it
mcon@cinderella:~$ incus stop incus:openwrt

## and after a few minutes I try to restrt it
mcon@cinderella:~$ incus start incus:openwrt
Error: Failed to start device "wlan0": Parent device 'wlx00c0cab2bd76' doesn't exist
Try `incus info --show-log incus:openwrt` for more info

## It fails and it is right because device is not visible anymore (presumably it was not dettched on `incus stop`
mcon@cinderella:~$ ssh root@incus -- ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 60:02:92:57:66:c1 brd ff:ff:ff:ff:ff:ff
3: enxa0cec8b43055: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether a0:ce:c8:b4:30:55 brd ff:ff:ff:ff:ff:ff
5: ORANGE: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 00:16:3e:ed:dd:b3 brd ff:ff:ff:ff:ff:ff
6: incusbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 00:16:3e:54:c8:5c brd ff:ff:ff:ff:ff:ff
8: vethee6b197a@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master incusbr0 state UP mode DEFAULT group default qlen 1000
    link/ether de:d4:99:67:26:1f brd ff:ff:ff:ff:ff:ff link-netnsid 0
10: vethae9bd16d@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master incusbr0 state UP mode DEFAULT group default qlen 1000
    link/ether ae:89:33:8b:97:07 brd ff:ff:ff:ff:ff:ff link-netnsid 1
12: vethe3470c1f@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ORANGE state UP mode DEFAULT group default qlen 1000
    link/ether ba:4b:6d:be:79:49 brd ff:ff:ff:ff:ff:ff link-netnsid 2
mcon@cinderella:~$

Information to attach

[x] Any relevant kernel output (dmesg) full-dmesg.log
[x] Container log (incus info NAME --show-log) info-show-log.log
[x] Container configuration (incus config show NAME --expanded) config.log
[x] Main daemon log (at /var/log/incus/incusd.log) incusd.log
[ ] Output of the client with --debug
[ ] Output of the daemon with --debug (alternatively output of incus monitor --pretty while reproducing the issue)

stgraber commented 8 months ago

Wireless devices are a bit weird and are comprised of two devices, the phy and the interface. Most users only ever notice the interface which is what you interact with.

When moving wireless devices into a container, both the phy and interface are moved in. When the instance dies, the Linux kernel attempts to delete whatever can be deleted and move back the rest. In this case, it likely means that the interface gets deleted and the phy is moved back, which doesn't really help you.

I need to find a USB wifi adapter or something that I can use to poke at this, hopefully it'd be possible to add pre-stop logic to relocate both the phy and interface to the host system prior to the kernel deleting everything.

mcondarelli commented 8 months ago

Let me know if I can help in any way.

stgraber commented 8 months ago

I've not been able to reproduce the issue here:

stgraber@dakara:~$ ip link | grep wl
3: wlp57s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DORMANT group default qlen 1000
31: wlxb44bd62a1d17: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DORMANT group default qlen 1000
stgraber@dakara:~$ incus config device add u1 wlan0 nic nictype=physical name=wlan0 parent=wlxb44bd62a1d17
Device wlan0 added to u1
stgraber@dakara:~$ incus start u1
stgraber@dakara:~$ ip link | grep wl
3: wlp57s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DORMANT group default qlen 1000
stgraber@dakara:~$ incus exec u1 -- ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
31: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DORMANT group default qlen 1000
    link/ether 9e:cb:30:7c:da:62 brd ff:ff:ff:ff:ff:ff permaddr b4:4b:d6:2a:1d:17
47: eth0@if48: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 00:16:3e:4d:51:51 brd ff:ff:ff:ff:ff:ff link-netnsid 0
stgraber@dakara:~$ incus stop u1
stgraber@dakara:~$ ip link | grep wl
3: wlp57s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DORMANT group default qlen 1000
31: wlxb44bd62a1d17: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DORMANT group default qlen 1000
stgraber@dakara:~$

stgraber commented 8 months ago

Can you confirm that the interface inside the container hasn't been renamed? I could see how having the interface renamed from Incus' expected name (wlan0) to something else could prevent it from being "rescued" ahead of shutdown.

stgraber commented 8 months ago

Also can you confirm that you have the iw tool installed on your system?

mcondarelli commented 8 months ago

I can confirm I have iw installed. Interface seems to have been renamed somehow; this is what I see in container:

~ # ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: phy4-ap0: <BROADCAST,MULTICAST> mtu 1500 qdisc noqueue state DOWN qlen 1000
    link/ether 00:c0:ca:b2:bd:76 brd ff:ff:ff:ff:ff:ff
29: eth0@phy4-ap0: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 00:16:3e:f2:5a:6f brd ff:ff:ff:ff:ff:ff
30: eth1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 00:16:3e:57:99:42 brd ff:ff:ff:ff:ff:ff
31: eth2@if32: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 00:16:3e:ed:92:d6 brd ff:ff:ff:ff:ff:ff
~ #

Output of iw list (if useful) is:

~ # iw list
Wiphy phy4
    wiphy index: 4
    max # scan SSIDs: 4
    max scan IEs length: 2243 bytes
    max # sched scan SSIDs: 0
    max # match sets: 0
    Retry short limit: 7
    Retry long limit: 4
    Coverage class: 0 (up to 0m)
    Device supports AP-side u-APSD.
    Device supports T-DLS.
    Available Antennas: TX 0x1 RX 0x1
    Configured Antennas: TX 0x1 RX 0x1
    Supported interface modes:
         * IBSS
         * managed
         * AP
         * AP/VLAN
         * monitor
         * mesh point
         * P2P-client
         * P2P-GO
    Band 1:
        Capabilities: 0x17e
            HT20/HT40
            SM Power Save disabled
            RX Greenfield
            RX HT20 SGI
            RX HT40 SGI
            RX STBC 1-stream
            Max AMSDU length: 3839 bytes
            No DSSS/CCK HT40
        Maximum RX AMPDU length 65535 bytes (exponent: 0x003)
        Minimum RX AMPDU time spacing: No restriction (0x00)
        HT TX/RX MCS rate indexes supported: 0-7
        Frequencies:
            * 2412 MHz [1] (1.0 dBm)
            * 2417 MHz [2] (1.0 dBm)
            * 2422 MHz [3] (1.0 dBm)
            * 2427 MHz [4] (2.0 dBm)
            * 2432 MHz [5] (2.0 dBm)
            * 2437 MHz [6] (2.0 dBm)
            * 2442 MHz [7] (2.0 dBm)
            * 2447 MHz [8] (2.0 dBm)
            * 2452 MHz [9] (2.0 dBm)
            * 2457 MHz [10] (2.0 dBm)
            * 2462 MHz [11] (2.0 dBm)
            * 2467 MHz [12] (2.0 dBm) (no IR)
            * 2472 MHz [13] (2.0 dBm) (no IR)
            * 2484 MHz [14] (2.0 dBm) (no IR)
    Band 2:
        Capabilities: 0x17e
            HT20/HT40
            SM Power Save disabled
            RX Greenfield
            RX HT20 SGI
            RX HT40 SGI
            RX STBC 1-stream
            Max AMSDU length: 3839 bytes
            No DSSS/CCK HT40
        Maximum RX AMPDU length 65535 bytes (exponent: 0x003)
        Minimum RX AMPDU time spacing: No restriction (0x00)
        HT TX/RX MCS rate indexes supported: 0-7
        VHT Capabilities (0x31800120):
            Max MPDU length: 3895
            Supported Channel Width: neither 160 nor 80+80
            short GI (80 MHz)
            RX antenna pattern consistency
            TX antenna pattern consistency
        VHT RX MCS set:
            1 streams: MCS 0-9
            2 streams: not supported
            3 streams: not supported
            4 streams: not supported
            5 streams: not supported
            6 streams: not supported
            7 streams: not supported
            8 streams: not supported
        VHT RX highest supported: 0 Mbps
        VHT TX MCS set:
            1 streams: MCS 0-9
            2 streams: not supported
            3 streams: not supported
            4 streams: not supported
            5 streams: not supported
            6 streams: not supported
            7 streams: not supported
            8 streams: not supported
        VHT TX highest supported: 0 Mbps
        VHT extended NSS: not supported
        Frequencies:
            * 5180 MHz [36] (15.0 dBm) (no IR)
            * 5200 MHz [40] (15.0 dBm) (no IR)
            * 5220 MHz [44] (15.0 dBm) (no IR)
            * 5240 MHz [48] (15.0 dBm) (no IR)
            * 5260 MHz [52] (14.0 dBm) (no IR, radar detection)
            * 5280 MHz [56] (14.0 dBm) (no IR, radar detection)
            * 5300 MHz [60] (15.0 dBm) (no IR, radar detection)
            * 5320 MHz [64] (15.0 dBm) (no IR, radar detection)
            * 5500 MHz [100] (13.0 dBm) (no IR, radar detection)
            * 5520 MHz [104] (12.0 dBm) (no IR, radar detection)
            * 5540 MHz [108] (11.0 dBm) (no IR, radar detection)
            * 5560 MHz [112] (11.0 dBm) (no IR, radar detection)
            * 5580 MHz [116] (11.0 dBm) (no IR, radar detection)
            * 5600 MHz [120] (11.0 dBm) (no IR, radar detection)
            * 5620 MHz [124] (11.0 dBm) (no IR, radar detection)
            * 5640 MHz [128] (11.0 dBm) (no IR, radar detection)
            * 5660 MHz [132] (11.0 dBm) (no IR, radar detection)
            * 5680 MHz [136] (11.0 dBm) (no IR, radar detection)
            * 5700 MHz [140] (11.0 dBm) (no IR, radar detection)
            * 5720 MHz [144] (12.0 dBm) (no IR, radar detection)
            * 5745 MHz [149] (12.0 dBm) (no IR)
            * 5765 MHz [153] (12.0 dBm) (no IR)
            * 5785 MHz [157] (12.0 dBm) (no IR)
            * 5805 MHz [161] (12.0 dBm) (no IR)
            * 5825 MHz [165] (12.0 dBm) (no IR)
            * 5845 MHz [169] (disabled)
            * 5865 MHz [173] (disabled)
    valid interface combinations:
         * #{ IBSS } <= 1, #{ managed, AP, mesh point, P2P-client, P2P-GO } <= 2,
           total <= 2, #channels <= 1, STA/AP BI must match
    HT Capability overrides:
         * MCS: ff ff ff ff ff ff ff ff ff ff
         * maximum A-MSDU length
         * supported channel width
         * short GI for 40 MHz
         * max A-MPDU length exponent
         * min MPDU start spacing
    max # scan plans: 1
    max scan plan interval: -1
    max scan plan iterations: 0
    Supported extended features:
        * [ VHT_IBSS ]: VHT-IBSS
        * [ RRM ]: RRM
        * [ FILS_STA ]: STA FILS (Fast Initial Link Setup)
        * [ CQM_RSSI_LIST ]: multiple CQM_RSSI_THOLD records
        * [ CONTROL_PORT_OVER_NL80211 ]: control port over nl80211
        * [ TXQS ]: FQ-CoDel-enabled intermediate TXQs
        * [ SCAN_RANDOM_SN ]: use random sequence numbers in scans
        * [ SCAN_MIN_PREQ_CONTENT ]: use probe request with only rate IEs in scans
        * [ AIRTIME_FAIRNESS ]: airtime fairness scheduling
        * [ AQL ]: Airtime Queue Limits (AQL)
        * [ CONTROL_PORT_NO_PREAUTH ]: disable pre-auth over nl80211 control port support
        * [ DEL_IBSS_STA ]: deletion of IBSS station support
        * [ SCAN_FREQ_KHZ ]: scan on kHz frequency support
        * [ CONTROL_PORT_OVER_NL80211_TX_STATUS ]: tx status for nl80211 control port support
~ #

stgraber commented 8 months ago

Okay, so that's likely the problem then.

If it always gets renamed to the same thing in the container, adjusting the name value in the container config to match that name should prevent the deletion on stop.

So based on the above, that would be phy4-ap0.

stgraber commented 8 months ago

If you can confirm that things work properly when the name lines up between your configuration and the name inside of the container prior to it being stopped, then we can close this issue.

mcondarelli commented 8 months ago

Thanks @stgraber, unfortunately I cannot confirm your analysis. There's something more going on on my side:

even with "same name" it doesn't seem possible to correctly detach adapter: Listing-1.txt
name inside Instance does not stay the same (right after reboot it starts at phy0-ap0).
I have two almost identical instances, one VM and a Container:
- Container is exhibiting problem.
- VM seems to be able to detach adapter normally.
- Each time I start VM the "internal name" gets bumped up phy(n+1)-ap0, even if adapter is not touched in any way: Listing-2.txt

It seems to me renaming is done by Incus machinery and not by any artifact in my VM/Container, but I might be very wrong.

In my specific case this is not a huge problem since this Incus server is meant to handle just 3 instances and to stay up 24/7; "inconvenience" is just in the present setup/trimming stage where I need to reboot the whole server between tries, but it seems to point to some non-trivial inconsistency inside Incus.

Just tell me if you want to further pursue the issue or if I have to live with it.

stgraber commented 8 months ago

@mcondarelli can you try passing the wireless card to a basic Ubuntu container instead? I'd like to see if the same issue happens when using a container and configuration similar to the one I tested here.

mcondarelli commented 8 months ago

@stgraber I have somewhat mixed results with plain Ubuntu:

Interface is released
I see the same message in /var/log/incus/incusd.log (time of last entry matches time of Container shutdown). Listing-3.txt

Should I try adding also all other NICs?

stgraber commented 8 months ago

Ah, interesting, so yeah, it does look like with Ubuntu (so without things getting renamed in the container) that things get restored properly.

The failure messages are interesting though, I'll have to look at that logic a bit.

mcondarelli commented 8 months ago

I will test on my "sensitive" setup as soon as it propagates to "daily".

lxc / incus

`incus stop <instance>` does not seem to release `nictype=physical` devices #385

Required information

Issue description

Steps to reproduce

Information to attach