lxd: Unsetting `core.metric_address` does not stop from serving metrics

[gabrielmougard]

Required information

• Distribution: Ubuntu
• Distribution version: 20.04
• The output of "lxc info" or if that fails:

  
  config:
  core.metrics_address: :8444
  api_extensions:
  - storage_zfs_remove_snapshots
  - container_host_shutdown_timeout
  - container_stop_priority
  - container_syscall_filtering
  - auth_pki
  - container_last_used_at
  - etag
  - patch
  - usb_devices
  - https_allowed_credentials
  - image_compression_algorithm
  - directory_manipulation
  - container_cpu_time
  - storage_zfs_use_refquota
  - storage_lvm_mount_options
  - network
  - profile_usedby
  - container_push
  - container_exec_recording
  - certificate_update
  - container_exec_signal_handling
  - gpu_devices
  - container_image_properties
  - migration_progress
  - id_map
  - network_firewall_filtering
  - network_routes
  - storage
  - file_delete
  - file_append
  - network_dhcp_expiry
  - storage_lvm_vg_rename
  - storage_lvm_thinpool_rename
  - network_vlan
  - image_create_aliases
  - container_stateless_copy
  - container_only_migration
  - storage_zfs_clone_copy
  - unix_device_rename
  - storage_lvm_use_thinpool
  - storage_rsync_bwlimit
  - network_vxlan_interface
  - storage_btrfs_mount_options
  - entity_description
  - image_force_refresh
  - storage_lvm_lv_resizing
  - id_map_base
  - file_symlinks
  - container_push_target
  - network_vlan_physical
  - storage_images_delete
  - container_edit_metadata
  - container_snapshot_stateful_migration
  - storage_driver_ceph
  - storage_ceph_user_name
  - resource_limits
  - storage_volatile_initial_source
  - storage_ceph_force_osd_reuse
  - storage_block_filesystem_btrfs
  - resources
  - kernel_limits
  - storage_api_volume_rename
  - macaroon_authentication
  - network_sriov
  - console
  - restrict_devlxd
  - migration_pre_copy
  - infiniband
  - maas_network
  - devlxd_events
  - proxy
  - network_dhcp_gateway
  - file_get_symlink
  - network_leases
  - unix_device_hotplug
  - storage_api_local_volume_handling
  - operation_description
  - clustering
  - event_lifecycle
  - storage_api_remote_volume_handling
  - nvidia_runtime
  - container_mount_propagation
  - container_backup
  - devlxd_images
  - container_local_cross_pool_handling
  - proxy_unix
  - proxy_udp
  - clustering_join
  - proxy_tcp_udp_multi_port_handling
  - network_state
  - proxy_unix_dac_properties
  - container_protection_delete
  - unix_priv_drop
  - pprof_http
  - proxy_haproxy_protocol
  - network_hwaddr
  - proxy_nat
  - network_nat_order
  - container_full
  - candid_authentication
  - backup_compression
  - candid_config
  - nvidia_runtime_config
  - storage_api_volume_snapshots
  - storage_unmapped
  - projects
  - candid_config_key
  - network_vxlan_ttl
  - container_incremental_copy
  - usb_optional_vendorid
  - snapshot_scheduling
  - snapshot_schedule_aliases
  - container_copy_project
  - clustering_server_address
  - clustering_image_replication
  - container_protection_shift
  - snapshot_expiry
  - container_backup_override_pool
  - snapshot_expiry_creation
  - network_leases_location
  - resources_cpu_socket
  - resources_gpu
  - resources_numa
  - kernel_features
  - id_map_current
  - event_location
  - storage_api_remote_volume_snapshots
  - network_nat_address
  - container_nic_routes
  - rbac
  - cluster_internal_copy
  - seccomp_notify
  - lxc_features
  - container_nic_ipvlan
  - network_vlan_sriov
  - storage_cephfs
  - container_nic_ipfilter
  - resources_v2
  - container_exec_user_group_cwd
  - container_syscall_intercept
  - container_disk_shift
  - storage_shifted
  - resources_infiniband
  - daemon_storage
  - instances
  - image_types
  - resources_disk_sata
  - clustering_roles
  - images_expiry
  - resources_network_firmware
  - backup_compression_algorithm
  - ceph_data_pool_name
  - container_syscall_intercept_mount
  - compression_squashfs
  - container_raw_mount
  - container_nic_routed
  - container_syscall_intercept_mount_fuse
  - container_disk_ceph
  - virtual-machines
  - image_profiles
  - clustering_architecture
  - resources_disk_id
  - storage_lvm_stripes
  - vm_boot_priority
  - unix_hotplug_devices
  - api_filtering
  - instance_nic_network
  - clustering_sizing
  - firewall_driver
  - projects_limits
  - container_syscall_intercept_hugetlbfs
  - limits_hugepages
  - container_nic_routed_gateway
  - projects_restrictions
  - custom_volume_snapshot_expiry
  - volume_snapshot_scheduling
  - trust_ca_certificates
  - snapshot_disk_usage
  - clustering_edit_roles
  - container_nic_routed_host_address
  - container_nic_ipvlan_gateway
  - resources_usb_pci
  - resources_cpu_threads_numa
  - resources_cpu_core_die
  - api_os
  - container_nic_routed_host_table
  - container_nic_ipvlan_host_table
  - container_nic_ipvlan_mode
  - resources_system
  - images_push_relay
  - network_dns_search
  - container_nic_routed_limits
  - instance_nic_bridged_vlan
  - network_state_bond_bridge
  - usedby_consistency
  - custom_block_volumes
  - clustering_failure_domains
  - resources_gpu_mdev
  - console_vga_type
  - projects_limits_disk
  - network_type_macvlan
  - network_type_sriov
  - container_syscall_intercept_bpf_devices
  - network_type_ovn
  - projects_networks
  - projects_networks_restricted_uplinks
  - custom_volume_backup
  - backup_override_name
  - storage_rsync_compression
  - network_type_physical
  - network_ovn_external_subnets
  - network_ovn_nat
  - network_ovn_external_routes_remove
  - tpm_device_type
  - storage_zfs_clone_copy_rebase
  - gpu_mdev
  - resources_pci_iommu
  - resources_network_usb
  - resources_disk_address
  - network_physical_ovn_ingress_mode
  - network_ovn_dhcp
  - network_physical_routes_anycast
  - projects_limits_instances
  - network_state_vlan
  - instance_nic_bridged_port_isolation
  - instance_bulk_state_change
  - network_gvrp
  - instance_pool_move
  - gpu_sriov
  - pci_device_type
  - storage_volume_state
  - network_acl
  - migration_stateful
  - disk_state_quota
  - storage_ceph_features
  - projects_compression
  - projects_images_remote_cache_expiry
  - certificate_project
  - network_ovn_acl
  - projects_images_auto_update
  - projects_restricted_cluster_target
  - images_default_architecture
  - network_ovn_acl_defaults
  - gpu_mig
  - project_usage
  - network_bridge_acl
  - warnings
  - projects_restricted_backups_and_snapshots
  - clustering_join_token
  - clustering_description
  - server_trusted_proxy
  - clustering_update_cert
  - storage_api_project
  - server_instance_driver_operational
  - server_supported_storage_drivers
  - event_lifecycle_requestor_address
  - resources_gpu_usb
  - clustering_evacuation
  - network_ovn_nat_address
  - network_bgp
  - network_forward
  - custom_volume_refresh
  - network_counters_errors_dropped
  - metrics
  - image_source_project
  - clustering_config
  - network_peer
  - linux_sysctl
  - network_dns
  - ovn_nic_acceleration
  - certificate_self_renewal
  - instance_project_move
  - storage_volume_project_move
  - cloud_init
  - network_dns_nat
  - database_leader
  - instance_all_projects
  - clustering_groups
  - ceph_rbd_du
  - instance_get_full
  - qemu_metrics
  - gpu_mig_uuid
  - event_project
  - clustering_evacuation_live
  - instance_allow_inconsistent_copy
  - network_state_ovn
  - storage_volume_api_filtering
  - image_restrictions
  - storage_zfs_export
  - network_dns_records
  - storage_zfs_reserve_space
  - network_acl_log
  - storage_zfs_blocksize
  - metrics_cpu_seconds
  - instance_snapshot_never
  - certificate_token
  - instance_nic_routed_neighbor_probe
  - event_hub
  - agent_nic_config
  - projects_restricted_intercept
  - metrics_authentication
  - images_target_project
  - cluster_migration_inconsistent_copy
  - cluster_ovn_chassis
  - container_syscall_intercept_sched_setscheduler
  - storage_lvm_thinpool_metadata_size
  - storage_volume_state_total
  - instance_file_head
  - instances_nic_host_name
  - image_copy_profile
  - container_syscall_intercept_sysinfo
  - clustering_evacuation_mode
  - resources_pci_vpd
  - qemu_raw_conf
  - storage_cephfs_fscache
  - network_load_balancer
  - vsock_api
  - instance_ready_state
  - network_bgp_holdtime
  - storage_volumes_all_projects
  - metrics_memory_oom_total
  - storage_buckets
  - storage_buckets_create_credentials
  - metrics_cpu_effective_total
  - projects_networks_restricted_access
  - storage_buckets_local
  - loki
  - acme
  - internal_metrics
  - cluster_join_token_expiry
  - remote_token_expiry
  - init_preseed
  - storage_volumes_created_at
  - cpu_hotplug
  - projects_networks_zones
  - network_txqueuelen
  - cluster_member_state
  - instances_placement_scriptlet
  - storage_pool_source_wipe
  - zfs_block_mode
  - instance_generation_id
  - disk_io_cache
  - amd_sev
  - storage_pool_loop_resize
  - migration_vm_live
  - ovn_nic_nesting
  - oidc
  - network_ovn_l3only
  - ovn_nic_acceleration_vdpa
  - cluster_healing
  - instances_state_total
  - auth_user
  - security_csm
  - instances_rebuild
  - numa_cpu_placement
  - custom_volume_iso
  - network_allocations
  - storage_api_remote_volume_snapshot_copy
  - zfs_delegate
  api_status: stable
  api_version: "1.0"
  auth: trusted
  public: false
  auth_methods:
  - tls
  auth_user_name: gab
  auth_user_method: unix
  environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
  -----BEGIN CERTIFICATE-----
  MIICEjCCAZigAwIBAgIQQAXihKt7wvKisLzRSrlH4zAKBggqhkjOPQQDAzA5MRww
  GgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRkwFwYDVQQDDBByb290QGdhYi1k
  ZXNrdG9wMB4XDTIzMDgwMTE0MDUzOVoXDTMzMDcyOTE0MDUzOVowOTEcMBoGA1UE
  ChMTbGludXhjb250YWluZXJzLm9yZzEZMBcGA1UEAwwQcm9vdEBnYWItZGVza3Rv
  cDB2MBAGByqGSM49AgEGBSuBBAAiA2IABEuygJQItQcJJxMg8GcbP1gZ59cuBQHr
  /8FPG4F0GWUXLK2evlnGVTb3OeuUXqyRlMwgmh7KU6bwI357Nlt5bIxyaCHVRIvR
  6NINijuYx7Mw/sDlxD3qlHZTVYS2UfN1BKNlMGMwDgYDVR0PAQH/BAQDAgWgMBMG
  A1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwLgYDVR0RBCcwJYILZ2Fi
  LWRlc2t0b3CHBH8AAAGHEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDaAAw
  ZQIxANOFEO0q0Magb7jICa0opDb6l2i26KCI7zuLM7a9LGOSthss5L3XTloGDM6f
  YFR1MgIwGvn4JfE+EmKXqRRKCqNSb7GZL1mfV4VImJ5vqrNJ5OG3Mp+ZdQFw3rNt
  qT1hX3fk
  -----END CERTIFICATE-----
  certificate_fingerprint: 515f857536086789a098cb83f3bd915efd3c4e9ef799a55d1b95b2399d376b73
  driver: lxc | qemu
  driver_version: 5.0.2 | 8.0.0
  firewall: xtables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
  idmapped_mounts: "true"
  netnsid_getifaddrs: "true"
  seccomp_listener: "true"
  seccomp_listener_continue: "true"
  shiftfs: "false"
  uevent_injection: "true"
  unpriv_fscaps: "true"
  kernel_version: 5.15.0-78-generic
  lxc_features:
  cgroup2: "true"
  core_scheduling: "true"
  devpts_fd: "true"
  idmapped_mounts_v2: "true"
  mount_injection_file: "true"
  network_gateway_device_route: "true"
  network_ipvlan: "true"
  network_l2proxy: "true"
  network_phys_macvlan_mtu: "true"
  network_veth_router: "true"
  pidfd: "true"
  seccomp_allow_deny_syntax: "true"
  seccomp_notify: "true"
  seccomp_proxy_send_notify_fd: "true"
  os_name: Ubuntu
  os_version: "20.04"
  project: default
  server: lxd
  server_clustered: false
  server_event_mode: full-mesh
  server_name: gab-desktop
  server_pid: 16218
  server_version: "5.16"
  storage: dir
  storage_version: "1"
  storage_supported_drivers:
  - name: dir
  version: "1"
  remote: false
  - name: lvm
  version: 2.03.11(2) (2021-01-08) / 1.02.175 (2021-01-08) / 4.45.0
  remote: false
  - name: zfs
  version: 2.1.5-1ubuntu6~22.04.1
  remote: false
  - name: btrfs
  version: 5.16.2
  remote: false
  - name: ceph
  version: 17.2.5
  remote: true
  - name: cephfs
  version: 17.2.5
  remote: true
  - name: cephobject
  version: 17.2.5
  remote: true

   * Kernel version: `5.15.0-78-generic`
   * LXC version: `5.16`
   * LXD version: `5.16`
   * Storage backend in use: `zfs`

# Issue description

After a metric server is started (`lxc config set core.metric_address <ADDR>`), unsetting the property (`lxc config unset core.metric_address`) does not stop the server (the metric page is still available). Looking at the logs, it seems that the underlying listener is closed but I suspect that the server shutdown is not triggered by the listener shutting down. Looking in `lxd/endpoint/endpoints.go` in `func (e *Endpoints) serve(kind kind)`, I see a final call to: 

```go
...

e.tomb.Go(func() error {
   return server.Serve(listener)
})

With no signal handling whatsoever (i.e, when the listener closes, is this goroutine terminated ? Because it seems I still can access the endpoint eventhough this one is not listening for new incoming data anymore)

I didn't check if this happens for other existing endpoints but I noticed the same behaviour here with this new endpoint I recently introduced.

Is it a bug or a feature (a.k.a is it intended to keep the endpoint reachable eventhough the property defining the endpoint address is not in the server core config) ?

Steps to reproduce

1. lxc config set core.metric_address :8444
2. Go to http://0.0.0.0:8444/1.0 in a browser and see the metrics.
3. lxc config unset core.metric_address
4. lxc config show , now core.metric_address should have disappeared which is as expected.
5. See a trace of closing socket in the log, proof that func (e *Endpoints) closeListener(kind kind) error has been called successfully.
6. Go to http://0.0.0.0:8444/1.0 in a browser and witness that the metric server is still reachable (we don't want that).

[gabrielmougard]

@monstermunchkin @tomponline have you witnessed this already ?

[tomponline]

Please show "lxc config show"

[gabrielmougard]

$ lxc config show
config: {}

[monstermunchkin]

This is not a bug.

I can reproduce what you're seeing when using the browser. That must have to do with caching. If you try the same thing with curl, you cannot connect to the server. Also, the output of ss -tln doesn't show that it's listening on the port anymore.

[tomponline]

Thanks!