Fred78290 commented 2 months ago

Required information

Distribution: Ubuntu
Distribution version: 24.04

The output of "snap list --all lxd core20 core22 core24 snapd":

core22  20240419        1439   latest/stable  canonical✓  base
lxd     5.21.2-34459c8  29568  6.1/stable     canonical✓  disabled
lxd     6.1-c14927a     29551  6.1/stable     canonical✓  -
snapd   2.63            21759  latest/stable  canonical✓  snapd

The output of "lxc info" or if that fails:

config:
core.https_address: '[::]:8443'
network.ovn.northbound_connection: unix:/var/run/ovn/ovnsb_db.sock
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- backup_compression
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- instances_nic_host_name
- image_copy_profile
- container_syscall_intercept_sysinfo
- clustering_evacuation_mode
- resources_pci_vpd
- qemu_raw_conf
- storage_cephfs_fscache
- network_load_balancer
- vsock_api
- instance_ready_state
- network_bgp_holdtime
- storage_volumes_all_projects
- metrics_memory_oom_total
- storage_buckets
- storage_buckets_create_credentials
- metrics_cpu_effective_total
- projects_networks_restricted_access
- storage_buckets_local
- loki
- acme
- internal_metrics
- cluster_join_token_expiry
- remote_token_expiry
- init_preseed
- storage_volumes_created_at
- cpu_hotplug
- projects_networks_zones
- network_txqueuelen
- cluster_member_state
- instances_placement_scriptlet
- storage_pool_source_wipe
- zfs_block_mode
- instance_generation_id
- disk_io_cache
- amd_sev
- storage_pool_loop_resize
- migration_vm_live
- ovn_nic_nesting
- oidc
- network_ovn_l3only
- ovn_nic_acceleration_vdpa
- cluster_healing
- instances_state_total
- auth_user
- security_csm
- instances_rebuild
- numa_cpu_placement
- custom_volume_iso
- network_allocations
- storage_api_remote_volume_snapshot_copy
- zfs_delegate
- operations_get_query_all_projects
- metadata_configuration
- syslog_socket
- event_lifecycle_name_and_project
- instances_nic_limits_priority
- disk_initial_volume_configuration
- operation_wait
- cluster_internal_custom_volume_copy
- disk_io_bus
- storage_cephfs_create_missing
- instance_move_config
- ovn_ssl_config
- init_preseed_storage_volumes
- metrics_instances_count
- server_instance_type_info
- resources_disk_mounted
- server_version_lts
- oidc_groups_claim
- loki_config_instance
- storage_volatile_uuid
- import_instance_devices
- instances_uefi_vars
- instances_migration_stateful
- container_syscall_filtering_allow_deny_syntax
- access_management
- vm_disk_io_limits
- storage_volumes_all
- instances_files_modify_permissions
- image_restriction_nesting
- container_syscall_intercept_finit_module
- device_usb_serial
- network_allocate_external_ips
- explicit_trust_token
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
auth_user_name: ubuntu
auth_user_method: unix
environment:
addresses:
- 10.81.38.205:8443
- '[fd42:7087:5459:1d57:5054:ff:fe98:955e]:8443'
- 10.15.224.1:8443
- '[fd42:1f75:8193:5aa8::1]:8443'
architectures:
- x86_64
- i686
certificate: |
-----BEGIN CERTIFICATE-----
MIIB2jCCAWCgAwIBAgIQSt2SseC3pRIXj5vdOGWznDAKBggqhkjOPQQDAzAhMQww
CgYDVQQKEwNMWEQxETAPBgNVBAMMCHJvb3RAbHhkMB4XDTI0MDgyMDA5MzY0MFoX
DTM0MDgxODA5MzY0MFowITEMMAoGA1UEChMDTFhEMREwDwYDVQQDDAhyb290QGx4
ZDB2MBAGByqGSM49AgEGBSuBBAAiA2IABKp5bkwDA936O87fJ5lcu7GL2XuBTdDR
p6UF0Qt3mMx2GwHqXnK2Av3WFaOgMrUL04PSyxTHjl/O6ROE8R3qD0VQ1LXbRFbw
kIswSg45DQ9W7XBSxjvsapR+yEm06DimQKNdMFswDgYDVR0PAQH/BAQDAgWgMBMG
A1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwJgYDVR0RBB8wHYIDbHhk
hwR/AAABhxAAAAAAAAAAAAAAAAAAAAABMAoGCCqGSM49BAMDA2gAMGUCMBYUzgHx
nHU2FcqGJhdWaD0aVf7YpAloGBeIPMugrvpDhQalcn1W5zuCx7K3RSox5QIxAPxD
ws8TjPDsBu6SqQ6Zncja7JCpvFuKew1vzUL3Xoscd1N64U+KyW5sHWxbRH/Txg==
-----END CERTIFICATE-----
certificate_fingerprint: 434f42f92a487533d67449ba715097c1c4a9f160890eb2388bb78ca9d1aaadf5
driver: lxc | qemu
driver_version: 6.0.0 | 8.2.1
instance_types:
- container
- virtual-machine
firewall: nftables
kernel: Linux
kernel_architecture: x86_64
kernel_features:
idmapped_mounts: "true"
netnsid_getifaddrs: "true"
seccomp_listener: "true"
seccomp_listener_continue: "true"
uevent_injection: "true"
unpriv_fscaps: "true"
kernel_version: 6.8.0-40-generic
lxc_features:
cgroup2: "true"
core_scheduling: "true"
devpts_fd: "true"
idmapped_mounts_v2: "true"
mount_injection_file: "true"
network_gateway_device_route: "true"
network_ipvlan: "true"
network_l2proxy: "true"
network_phys_macvlan_mtu: "true"
network_veth_router: "true"
pidfd: "true"
seccomp_allow_deny_syntax: "true"
seccomp_notify: "true"
seccomp_proxy_send_notify_fd: "true"
os_name: Ubuntu
os_version: "24.04"
project: default
server: lxd
server_clustered: false
server_event_mode: full-mesh
server_name: lxd
server_pid: 9985
server_version: "6.1"
server_lts: false
storage: dir
storage_version: "1"
storage_supported_drivers:
- name: ceph
version: 17.2.7
remote: true
- name: cephfs
version: 17.2.7
remote: true
- name: cephobject
version: 17.2.7
remote: true
- name: dir
version: "1"
remote: false
- name: lvm
version: 2.03.11(2) (2021-01-08) / 1.02.175 (2021-01-08) / 4.48.0
remote: false
- name: powerflex
version: 1.16 (nvme-cli)
remote: true
- name: zfs
version: 2.2.2-0ubuntu9
remote: false
- name: btrfs
version: 5.16.2
remote: false

Issue description

Unable to setup OVN network, the command lxc network create ovntest --type=ovn network=lxdbr0 fail returning

Error: Failed to run: ovn-nbctl --timeout=10 --db unix:/var/lib/snapd/hostfs/run/ovn/ovnsb_db.sock --wait=sb ha-chassis-group-add lxd-net7: exit status 1 (ovn-nbctl: unix:/var/lib/snapd/hostfs/run/ovn/ovnsb_db.sock: database connection failed ()) The reason is snapped lxd append /var/lib/snapd/hostfs to ovn socket path

Steps to reproduce

create an instance with multipass: multipass lxd-test launch -c 4 -m 8G -d 40G
install snap lxd: sudo snap install lxd --channel=6.1/stable
lxd init all default except: pool use dir storage
install ovn as described in the documentation Set up a standalone OVN network
set lxd config: lxc config set network.ovn.northbound_connection=unix:/var/run/ovn/ovnnb_db.sock
create ovn network: lxc network create ovntest --type=ovn network=lxdbr0

Problem in the source location

File: ./lxd/network/openvswitch/ovn.go Line: 305 Method: func (o *OVN) xbctl(southbound bool, extraArgs ...string) (string, error) { Wrong code: if strings.HasPrefix(dbAddr, "unix:") { dbAddr = fmt.Sprintf("unix:%s", shared.HostPathFollow(strings.TrimPrefix(dbAddr, "unix:"))) }

shared.HostPathFollow(...) detect lxd snapped and append /var/lib/snapd/hostfs path

tomponline commented 2 months ago

@Fred78290 out of interest, why are you setting network.ovn.northbund_connection, as it defaults to unix:/var/run/ovn/ovnnb_db.sock anyway: https://documentation.ubuntu.com/lxd/en/latest/server/#server-miscellaneous:network.ovn.northbound_connection

tomponline commented 2 months ago

LXD appending /var/lib/snapd/hostfs/ to the supplied path is correct in order to access files outside of the snap package.

tomponline commented 2 months ago

This part is confusing me:

ovn-nbctl --timeout=10 --db unix:/var/lib/snapd/hostfs/run/ovn/ovnsb_db.sock

Why is ovn-nbctl (which manages the northbound database) being passed the path for the OVN southbound database (ovnsb_db.sock)?

Can you try lxc config unset network.ovn.northbound_connection and then also provide the output of:

sudo ovs-vsctl get open_vswitch . external_ids:ovn-remote

Which is where LXD gets the address of the OVN southbound database.

tomponline commented 2 months ago

Have you followed the setup guide here btw?

https://documentation.ubuntu.com/lxd/en/latest/howto/network_ovn_setup/#set-up-a-standalone-ovn-network

Fred78290 commented 2 months ago

@tomponline

After exec lxc config unset network.ovn.northbound_connection i'm able to run lxc network create ovntest --type=ovn network=lxdbr0 succesfully.

https://documentation.ubuntu.com/lxd/en/latest/howto/network_ovn_setup/#set-up-a-standalone-ovn-network

Yes I followed the setup guide.

Fred78290 commented 2 months ago

sudo ovs-vsctl get open_vswitch . external_ids:ovn-remote

got: unix:/var/run/ovn/ovnsb_db.sock

Fred78290 commented 2 months ago

@tomponline

This part is confusing me:

ovn-nbctl --timeout=10 --db unix:/var/lib/snapd/hostfs/run/ovn/ovnsb_db.sock

It's returned as error message the executed sub command.

I have setted the network.ovn.northbound_connection explicitly to network.ovn.northbound_connection because in my real environment I share ovn system beetwen openstack and LXd on the same machine and the connection is SSL and lxd won't connect to ovn when I fill SSL infos and remote IP but it connect thru the unix socket (default value). and I'm looking what is the problem.

It's mean actually setting the network.ovn.northbound_connection to something is not working.

tomponline commented 2 months ago

because in my real environment I share ovn system beetwen openstack and LXd on the same machine

I've never tried that, but its not supported, so you might be unexpected conflicts/problems doing that.

Fred78290 commented 2 months ago

I've never tried that, but its not supported, so you might be unexpected conflicts/problems doing that.

Has unexpected it's work but without setting network.ovn.northbound_connection.

On the same machine I run multipass, vmware workstation pro, openstack, devstack and lxd without conflicts.

tomponline commented 2 months ago

@boltmark as you're going to start looking into OVN uplink VLAN feature, please could you take a look at recreating this issue first as you setup your OVN environment? Thanks

canonical / lxd

Snapped LXD 6.1 fail to create ovn network (snap path appending /var/lib/snapd/hostfs to network.ovn.northbound_connection) #13949

Required information

Issue description

Steps to reproduce

Problem in the source location