LXD snap fails to load xtables firewall driver on system without nftables

[antonc42]

Required information

• Distribution: Kubuntu
• Distribution version: 22.04
• The output of "lxc info":

  config: {}
  api_extensions:
  - storage_zfs_remove_snapshots
  - container_host_shutdown_timeout
  - container_stop_priority
  - container_syscall_filtering
  - auth_pki
  - container_last_used_at
  - etag
  - patch
  - usb_devices
  - https_allowed_credentials
  - image_compression_algorithm
  - directory_manipulation
  - container_cpu_time
  - storage_zfs_use_refquota
  - storage_lvm_mount_options
  - network
  - profile_usedby
  - container_push
  - container_exec_recording
  - certificate_update
  - container_exec_signal_handling
  - gpu_devices
  - container_image_properties
  - migration_progress
  - id_map
  - network_firewall_filtering
  - network_routes
  - storage
  - file_delete
  - file_append
  - network_dhcp_expiry
  - storage_lvm_vg_rename
  - storage_lvm_thinpool_rename
  - network_vlan
  - image_create_aliases
  - container_stateless_copy
  - container_only_migration
  - storage_zfs_clone_copy
  - unix_device_rename
  - storage_lvm_use_thinpool
  - storage_rsync_bwlimit
  - network_vxlan_interface
  - storage_btrfs_mount_options
  - entity_description
  - image_force_refresh
  - storage_lvm_lv_resizing
  - id_map_base
  - file_symlinks
  - container_push_target
  - network_vlan_physical
  - storage_images_delete
  - container_edit_metadata
  - container_snapshot_stateful_migration
  - storage_driver_ceph
  - storage_ceph_user_name
  - resource_limits
  - storage_volatile_initial_source
  - storage_ceph_force_osd_reuse
  - storage_block_filesystem_btrfs
  - resources
  - kernel_limits
  - storage_api_volume_rename
  - macaroon_authentication
  - network_sriov
  - console
  - restrict_devlxd
  - migration_pre_copy
  - infiniband
  - maas_network
  - devlxd_events
  - proxy
  - network_dhcp_gateway
  - file_get_symlink
  - network_leases
  - unix_device_hotplug
  - storage_api_local_volume_handling
  - operation_description
  - clustering
  - event_lifecycle
  - storage_api_remote_volume_handling
  - nvidia_runtime
  - container_mount_propagation
  - container_backup
  - devlxd_images
  - container_local_cross_pool_handling
  - proxy_unix
  - proxy_udp
  - clustering_join
  - proxy_tcp_udp_multi_port_handling
  - network_state
  - proxy_unix_dac_properties
  - container_protection_delete
  - unix_priv_drop
  - pprof_http
  - proxy_haproxy_protocol
  - network_hwaddr
  - proxy_nat
  - network_nat_order
  - container_full
  - candid_authentication
  - backup_compression
  - candid_config
  - nvidia_runtime_config
  - storage_api_volume_snapshots
  - storage_unmapped
  - projects
  - candid_config_key
  - network_vxlan_ttl
  - container_incremental_copy
  - usb_optional_vendorid
  - snapshot_scheduling
  - snapshot_schedule_aliases
  - container_copy_project
  - clustering_server_address
  - clustering_image_replication
  - container_protection_shift
  - snapshot_expiry
  - container_backup_override_pool
  - snapshot_expiry_creation
  - network_leases_location
  - resources_cpu_socket
  - resources_gpu
  - resources_numa
  - kernel_features
  - id_map_current
  - event_location
  - storage_api_remote_volume_snapshots
  - network_nat_address
  - container_nic_routes
  - rbac
  - cluster_internal_copy
  - seccomp_notify
  - lxc_features
  - container_nic_ipvlan
  - network_vlan_sriov
  - storage_cephfs
  - container_nic_ipfilter
  - resources_v2
  - container_exec_user_group_cwd
  - container_syscall_intercept
  - container_disk_shift
  - storage_shifted
  - resources_infiniband
  - daemon_storage
  - instances
  - image_types
  - resources_disk_sata
  - clustering_roles
  - images_expiry
  - resources_network_firmware
  - backup_compression_algorithm
  - ceph_data_pool_name
  - container_syscall_intercept_mount
  - compression_squashfs
  - container_raw_mount
  - container_nic_routed
  - container_syscall_intercept_mount_fuse
  - container_disk_ceph
  - virtual-machines
  - image_profiles
  - clustering_architecture
  - resources_disk_id
  - storage_lvm_stripes
  - vm_boot_priority
  - unix_hotplug_devices
  - api_filtering
  - instance_nic_network
  - clustering_sizing
  - firewall_driver
  - projects_limits
  - container_syscall_intercept_hugetlbfs
  - limits_hugepages
  - container_nic_routed_gateway
  - projects_restrictions
  - custom_volume_snapshot_expiry
  - volume_snapshot_scheduling
  - trust_ca_certificates
  - snapshot_disk_usage
  - clustering_edit_roles
  - container_nic_routed_host_address
  - container_nic_ipvlan_gateway
  - resources_usb_pci
  - resources_cpu_threads_numa
  - resources_cpu_core_die
  - api_os
  - container_nic_routed_host_table
  - container_nic_ipvlan_host_table
  - container_nic_ipvlan_mode
  - resources_system
  - images_push_relay
  - network_dns_search
  - container_nic_routed_limits
  - instance_nic_bridged_vlan
  - network_state_bond_bridge
  - usedby_consistency
  - custom_block_volumes
  - clustering_failure_domains
  - resources_gpu_mdev
  - console_vga_type
  - projects_limits_disk
  - network_type_macvlan
  - network_type_sriov
  - container_syscall_intercept_bpf_devices
  - network_type_ovn
  - projects_networks
  - projects_networks_restricted_uplinks
  - custom_volume_backup
  - backup_override_name
  - storage_rsync_compression
  - network_type_physical
  - network_ovn_external_subnets
  - network_ovn_nat
  - network_ovn_external_routes_remove
  - tpm_device_type
  - storage_zfs_clone_copy_rebase
  - gpu_mdev
  - resources_pci_iommu
  - resources_network_usb
  - resources_disk_address
  - network_physical_ovn_ingress_mode
  - network_ovn_dhcp
  - network_physical_routes_anycast
  - projects_limits_instances
  - network_state_vlan
  - instance_nic_bridged_port_isolation
  - instance_bulk_state_change
  - network_gvrp
  - instance_pool_move
  - gpu_sriov
  - pci_device_type
  - storage_volume_state
  - network_acl
  - migration_stateful
  - disk_state_quota
  - storage_ceph_features
  - projects_compression
  - projects_images_remote_cache_expiry
  - certificate_project
  - network_ovn_acl
  - projects_images_auto_update
  - projects_restricted_cluster_target
  - images_default_architecture
  - network_ovn_acl_defaults
  - gpu_mig
  - project_usage
  - network_bridge_acl
  - warnings
  - projects_restricted_backups_and_snapshots
  - clustering_join_token
  - clustering_description
  - server_trusted_proxy
  - clustering_update_cert
  - storage_api_project
  - server_instance_driver_operational
  - server_supported_storage_drivers
  - event_lifecycle_requestor_address
  - resources_gpu_usb
  - clustering_evacuation
  - network_ovn_nat_address
  - network_bgp
  - network_forward
  - custom_volume_refresh
  - network_counters_errors_dropped
  - metrics
  - image_source_project
  - clustering_config
  - network_peer
  - linux_sysctl
  - network_dns
  - ovn_nic_acceleration
  - certificate_self_renewal
  - instance_project_move
  - storage_volume_project_move
  - cloud_init
  - network_dns_nat
  - database_leader
  - instance_all_projects
  - clustering_groups
  - ceph_rbd_du
  - instance_get_full
  - qemu_metrics
  - gpu_mig_uuid
  - event_project
  - clustering_evacuation_live
  - instance_allow_inconsistent_copy
  - network_state_ovn
  - storage_volume_api_filtering
  - image_restrictions
  - storage_zfs_export
  - network_dns_records
  - storage_zfs_reserve_space
  - network_acl_log
  - storage_zfs_blocksize
  - metrics_cpu_seconds
  - instance_snapshot_never
  - certificate_token
  - instance_nic_routed_neighbor_probe
  - event_hub
  - agent_nic_config
  - projects_restricted_intercept
  - metrics_authentication
  - images_target_project
  - cluster_migration_inconsistent_copy
  - cluster_ovn_chassis
  - container_syscall_intercept_sched_setscheduler
  - storage_lvm_thinpool_metadata_size
  - storage_volume_state_total
  - instance_file_head
  - instances_nic_host_name
  - image_copy_profile
  - container_syscall_intercept_sysinfo
  - clustering_evacuation_mode
  - resources_pci_vpd
  - qemu_raw_conf
  - storage_cephfs_fscache
  - network_load_balancer
  - vsock_api
  - instance_ready_state
  - network_bgp_holdtime
  - storage_volumes_all_projects
  - metrics_memory_oom_total
  - storage_buckets
  - storage_buckets_create_credentials
  - metrics_cpu_effective_total
  - projects_networks_restricted_access
  - storage_buckets_local
  - loki
  - acme
  - internal_metrics
  - cluster_join_token_expiry
  - remote_token_expiry
  - init_preseed
  - storage_volumes_created_at
  - cpu_hotplug
  - projects_networks_zones
  - network_txqueuelen
  - cluster_member_state
  - instances_placement_scriptlet
  - storage_pool_source_wipe
  - zfs_block_mode
  - instance_generation_id
  - disk_io_cache
  - amd_sev
  - storage_pool_loop_resize
  - migration_vm_live
  - ovn_nic_nesting
  - oidc
  - network_ovn_l3only
  - ovn_nic_acceleration_vdpa
  api_status: stable
  api_version: "1.0"
  auth: trusted
  public: false
  auth_methods:
  - tls
  environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: redacted
  certificate_fingerprint: redacted
  driver: lxc | qemu
  driver_version: 5.0.2 | 8.0.0
  firewall: nftables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
  idmapped_mounts: "true"
  netnsid_getifaddrs: "true"
  seccomp_listener: "true"
  seccomp_listener_continue: "true"
  shiftfs: "false"
  uevent_injection: "true"
  unpriv_fscaps: "true"
  kernel_version: 5.15.0-72-generic
  lxc_features:
  cgroup2: "true"
  core_scheduling: "true"
  devpts_fd: "true"
  idmapped_mounts_v2: "true"
  mount_injection_file: "true"
  network_gateway_device_route: "true"
  network_ipvlan: "true"
  network_l2proxy: "true"
  network_phys_macvlan_mtu: "true"
  network_veth_router: "true"
  pidfd: "true"
  seccomp_allow_deny_syntax: "true"
  seccomp_notify: "true"
  seccomp_proxy_send_notify_fd: "true"
  os_name: Ubuntu
  os_version: "22.04"
  project: default
  server: lxd
  server_clustered: false
  server_event_mode: full-mesh
  server_name: redacted
  server_pid: 78125
  server_version: "5.13"
  storage: zfs
  storage_version: 2.1.5-1ubuntu6~22.04.1
  storage_supported_drivers:
  - name: ceph
  version: 17.2.5
  remote: true
  - name: cephfs
  version: 17.2.5
  remote: true
  - name: cephobject
  version: 17.2.5
  remote: true
  - name: dir
  version: "1"
  remote: false
  - name: lvm
  version: 2.03.11(2) (2021-01-08) / 1.02.175 (2021-01-08) / 4.45.0
  remote: false
  - name: zfs
  version: 2.1.5-1ubuntu6~22.04.1
  remote: false
  - name: btrfs
  version: 5.16.2
  remote: false

  • Snapd version:

    snap    2.59.2
    snapd   2.59.2
    series  16
    ubuntu  22.04
    kernel  5.15.0-72-generic

  • LXD snap version:

    lxd                     5.13-8e2d7eb     24846  latest/stable  canonical✓   -

Issue description

On a new Kubuntu 22.04 install, I disabled and removed nftables and ufw. I'm using plain iptables. After installing the LXD snap, it is not implementing any automatic iptables rules. Therefor the containers do not get DHCP addresses or get NATed and cannot communicate on the network. It appears that for some reason LXD is chosing the 'nftables' firewall driver instead of 'xtables'. From looking at some other discussion threads and the driver code, it appears that it should not do this if the 'nft' command is not available on the system, which it is not.

Steps to reproduce

1. Ensure nftables is not installed.

   nft
   bash: nft: command not found

   sudo update-alternatives --display iptables
   iptables - manual mode
   link best version is /usr/sbin/iptables-nft
   link currently points to /usr/sbin/iptables-legacy
   link iptables is /usr/sbin/iptables
   slave iptables-restore is /usr/sbin/iptables-restore
   slave iptables-save is /usr/sbin/iptables-save
   /usr/sbin/iptables-legacy - priority 10
   slave iptables-restore: /usr/sbin/iptables-legacy-restore
   slave iptables-save: /usr/sbin/iptables-legacy-save
   /usr/sbin/iptables-nft - priority 20
   slave iptables-restore: /usr/sbin/iptables-nft-restore
   slave iptables-save: /usr/sbin/iptables-nft-save

2. There are iptables-legacy rules present

   
   sudo iptables -nvL
   Chain INPUT (policy DROP 0 packets, 0 bytes)
   pkts bytes target     prot opt in     out     source               destination         
   122K  108M PREEMT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           
   122K  108M PREPROC    all  --  *      *       0.0.0.0/0            0.0.0.0/0           
   724  142K MAIN       all  --  *      *       0.0.0.0/0            0.0.0.0/0           
   706  140K REJECT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* reject all other UDP */ reject-with icmp-port-unreachable
   18  1899 REJECT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* reject all other TCP */ reject-with tcp-reset
   0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* reject all other packets */ reject-with icmp-proto-unreachable

Chain FORWARD (policy DROP 0 packets, 0 bytes) pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 112K packets, 31M bytes) pkts bytes target prot opt in out source destination

(rest of tables redacted)

 5. LXD still chooses 'nftables' firewall driver and fails to put automatic rules in iptables.

lxc info | grep firewall: firewall: nftables

 6. It appears that the 'nft' command exists inside the snap. Could this be affecting how the driver is detected?

sudo find / -name nft /snap/lxd/24846/bin/nft

# Information to attach

sudo snap set lxd daemon.debug=true sudo systemctl reload snap.lxd.daemon

[/var/snap/lxd/common/lxd/logs/lxd.log](https://github.com/lxc/lxd/files/11500817/lxd.log)

[tomponline]

What does iptables-save show and what does iptables -v show?

[tomponline]

Also if you reinstall nft temporarily what does "nft list ruleset" show? I suspect you have nft rules still active (either from nft or from the iptables-nft shim)

[antonc42]

You were correct. There were some IPv6 rules in the nft ruleset. I forgot to do update-alternatives on ip6tables. Everything appears to be working now. Sorry for the trouble, and thanks for all the great work on LXD!