search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.38k stars 931 forks source link

404 on /1.0/certificates #10376

Closed vosdev closed 2 years ago

vosdev commented 2 years ago

Required information

Issue description

A brief description of the problem. Should include what you were attempting to do, what you did, what happened and what you expected to see happen.

Adding my server as a remote on my local client (Powershell LXD 5.1 installed via choco) the server returns a 404

root @ epyc # lxc config set core.trust_password=QUyLH9mReRueF2k
PS C:\WINDOWS\system32> lxc remote add epyc 192.168.178.5 --protocol lxd --password QUyLH9mReRueF2k
Error: not found
PS C:\WINDOWS\system32> lxc remote add epyc 192.168.178.5 --protocol lxd --password QUyLH9mReRueF2k
Error: not found
PS C:\WINDOWS\system32> lxc --version
5.1

I executed snap set lxd daemon.debug=true but the logs in journalctl are pretty unreadable as every special character is escaped :(

I replaced \", \t with their respective characters and stripped the \n for your convenience

Also the GET on /1.0 is done twice so I only included it once

May 07 09:33:38 epyc lxd.daemon[1105813]: time="2022-05-07T09:33:38+02:00" level=debug msg="Allowing untrusted GET" ip="192.168.178.29:58333" url=/1.0
May 07 09:33:38 epyc lxd.daemon[1105813]: time="2022-05-07T09:33:38+02:00" level=debug msg="WriteJSON  {   "type": "sync",   "status": "Success",   "status_code": 200,   "operation": "",   "error_code": 0,   "error": "",   "metadata": {    "api_extensions": [     "storage_zfs_remove_snapshots",     "container_host_shutdown_timeout",     "container_stop_priority",     "container_syscall_filtering",     "auth_pki",     "container_last_used_at",     "etag",     "patch",     "usb_devices",     "https_allowed_credentials",     "image_compression_algorithm",     "directory_manipulation",     "container_cpu_time",     "storage_zfs_use_refquota",     "storage_lvm_mount_options",     "network",     "profile_usedby",     "container_push",     "container_exec_recording",     "certificate_update",     "container_exec_signal_handling",     "gpu_devices",     "container_image_properties",     "migration_progress",     "id_map",     "network_firewall_filtering",     "network_routes",     "storage",     "file_delete",     "file_append",     "network_dhcp_expiry",     "storage_lvm_vg_rename",     "storage_lvm_thinpool_rename",     "network_vlan",     "image_create_aliases",     "container_stateless_copy",     "container_only_migration",     "storage_zfs_clone_copy",     "unix_device_rename",     "storage_lvm_use_thinpool",     "storage_rsync_bwlimit",     "network_vxlan_interface",     "storage_btrfs_mount_options",     "entity_description",     "image_force_refresh",     "storage_lvm_lv_resizing",     "id_map_base",     "file_symlinks",     "container_push_target",     "network_vlan_physical",     "storage_images_delete",     "container_edit_metadata",     "container_snapshot_stateful_migration",     "storage_driver_ceph",     "storage_ceph_user_name",     "resource_limits",     "storage_volatile_initial_source",     "storage_ceph_force_osd_reuse",     "storage_block_filesystem_btrfs",     "resources",     "kernel_limits",     "storage_api_volume_rename",     "macaroon_authentication",     "network_sriov",     "console",     "restrict_devlxd",     "migration_pre_copy",     "infiniband",     "maas_network",     "devlxd_events",     "proxy",     "network_dhcp_gateway",     "file_get_symlink",     "network_leases",     "unix_device_hotplug",     "storage_api_local_volume_handling",     "operation_description",     "clustering",     "event_lifecycle",     "storage_api_remote_volume_handling",     "nvidia_runtime",     "container_mount_propagation",     "container_backup",     "devlxd_images",     "container_local_cross_pool_handling",     "proxy_unix",     "proxy_udp",     "clustering_join",     "proxy_tcp_udp_multi_port_handling",     "network_state",     "proxy_unix_dac_properties",     "container_protection_delete",     "unix_priv_drop",     "pprof_http",     "proxy_haproxy_protocol",     "network_hwaddr",     "proxy_nat",     "network_nat_order",     "container_full",     "candid_authentication",     "backup_compression",     "candid_config",     "nvidia_runtime_config",     "storage_api_volume_snapshots",     "storage_unmapped",     "projects",     "candid_config_key",     "network_vxlan_ttl",     "container_incremental_copy",     "usb_optional_vendorid",     "snapshot_scheduling",     "snapshot_schedule_aliases",     "container_copy_project",     "clustering_server_address",     "clustering_image_replication",     "container_protection_shift",     "snapshot_expiry",     "container_backup_override_pool",     "snapshot_expiry_creation",     "network_leases_location",     "resources_cpu_socket",     "resources_gpu",     "resources_numa",     "kernel_features",     "id_map_current",     "event_location",     "storage_api_remote_volume_snapshots",     "network_nat_address",     "container_nic_routes",     "rbac",     "cluster_internal_copy",     "seccomp_notify",     "lxc_features",     "container_nic_ipvlan",     "network_vlan_sriov",     "storage_cephfs",     "container_nic_ipfilter",     "resources_v2",     "container_exec_user_group_cwd",     "container_syscall_intercept",     "container_disk_shift",     "storage_shifted",     "resources_infiniband",     "daemon_storage",     "instances",     "image_types",     "resources_disk_sata",     "clustering_roles",     "images_expiry",     "resources_network_firmware",     "backup_compression_algorithm",     "ceph_data_pool_name",     "container_syscall_intercept_mount",     "compression_squashfs",     "container_raw_mount",     "container_nic_routed",     "container_syscall_intercept_mount_fuse",     "container_disk_ceph",     "virtual-machines",     "image_profiles",     "clustering_architecture",     "resources_disk_id",     "storage_lvm_stripes",     "vm_boot_priority",     "unix_hotplug_devices",     "api_filtering",     "instance_nic_network",     "clustering_sizing",     "firewall_driver",     "projects_limits",     "container_syscall_intercept_hugetlbfs",     "limits_hugepages",     "container_nic_routed_gateway",     "projects_restrictions",     "custom_volume_snapshot_expiry",     "volume_snapshot_scheduling",     "trust_ca_certificates",     "snapshot_disk_usage",     "clustering_edit_roles",     "container_nic_routed_host_address",     "container_nic_ipvlan_gateway",     "resources_usb_pci",     "resources_cpu_threads_numa",     "resources_cpu_core_die",     "api_os",     "container_nic_routed_host_table",     "container_nic_ipvlan_host_table",     "container_nic_ipvlan_mode",     "resources_system",     "images_push_relay",     "network_dns_search",     "container_nic_routed_limits",     "instance_nic_bridged_vlan",     "network_state_bond_bridge",     "usedby_consistency",     "custom_block_volumes",     "clustering_failure_domains",     "resources_gpu_mdev",     "console_vga_type",     "projects_limits_disk",     "network_type_macvlan",     "network_type_sriov",     "container_syscall_intercept_bpf_devices",     "network_type_ovn",     "projects_networks",     "projects_networks_restricted_uplinks",     "custom_volume_backup",     "backup_override_name",     "storage_rsync_compression",     "network_type_physical",     "network_ovn_external_subnets",     "network_ovn_nat",     "network_ovn_external_routes_remove",     "tpm_device_type",     "storage_zfs_clone_copy_rebase",     "gpu_mdev",     "resources_pci_iommu",     "resources_network_usb",     "resources_disk_address",     "network_physical_ovn_ingress_mode",     "network_ovn_dhcp",     "network_physical_routes_anycast",     "projects_limits_instances",     "network_state_vlan",     "instance_nic_bridged_port_isolation",     "instance_bulk_state_change",     "network_gvrp",     "instance_pool_move",     "gpu_sriov",     "pci_device_type",     "storage_volume_state",     "network_acl",     "migration_stateful",     "disk_state_quota",     "storage_ceph_features",     "projects_compression",     "projects_images_remote_cache_expiry",     "certificate_project",     "network_ovn_acl",     "projects_images_auto_update",     "projects_restricted_cluster_target",     "images_default_architecture",     "network_ovn_acl_defaults",     "gpu_mig",     "project_usage",     "network_bridge_acl",     "warnings",     "projects_restricted_backups_and_snapshots",     "clustering_join_token",     "clustering_description",     "server_trusted_proxy",     "clustering_update_cert",     "storage_api_project",     "server_instance_driver_operational",     "server_supported_storage_drivers",     "event_lifecycle_requestor_address",     "resources_gpu_usb",     "clustering_evacuation",     "network_ovn_nat_address",     "network_bgp",     "network_forward",     "custom_volume_refresh",     "network_counters_errors_dropped",     "metrics",     "image_source_project",     "clustering_config",     "network_peer",     "linux_sysctl",     "network_dns",     "ovn_nic_acceleration",     "certificate_self_renewal",     "instance_project_move",     "storage_volume_project_move",     "cloud_init",     "network_dns_nat",     "database_leader",     "instance_all_projects",     "clustering_groups",     "ceph_rbd_du",     "instance_get_full",     "qemu_metrics",     "gpu_mig_uuid",     "event_project",     "clustering_evacuation_live",     "instance_allow_inconsistent_copy",     "network_state_ovn",     "storage_volume_api_filtering",     "image_restrictions",     "storage_zfs_export",     "network_dns_records",     "storage_zfs_reserve_space",     "network_acl_log",     "storage_zfs_blocksize",     "metrics_cpu_seconds",     "instance_snapshot_never",     "certificate_token",     "instance_nic_routed_neighbor_probe",     "event_hub",     "agent_nic_config",     "projects_restricted_intercept",     "metrics_authentication",     "images_target_project",     "cluster_migration_inconsistent_copy",     "cluster_ovn_chassis",     "container_syscall_intercept_sched_setscheduler",     "storage_lvm_thinpool_metadata_size",     "storage_volume_state_total",     "instance_file_head",     "instances_nic_host_name",     "image_copy_profile",     "container_syscall_intercept_sysinfo",     "clustering_evacuation_mode"    ],    "api_status": "stable",    "api_version": "1.0",    "auth": "untrusted",    "public": false,    "auth_methods": [     "tls"    ]   }  }" http_code=200
May 07 09:33:38 epyc lxd.daemon[1105813]: time="2022-05-07T09:33:38+02:00" level=info msg="Sending top level 404" url=/1.0/certificates
May 07 09:33:38 epyc lxd.daemon[1105813]: time="2022-05-07T09:33:38+02:00" level=debug msg="Error Response  {   "type": "error",   "status": "",   "status_code": 0,   "operation": "",   "error_code": 404,   "error": "not found",   "metadata": null  }" http_code=404

To make sure my LXD install for Windows is not the issue here I created a Focal VM, installed the latest LXD snap and tried to add my server as a remote:

~
root @ lxd1 # snap install lxd --channel=latest
2022-05-07T09:45:54+02:00 INFO Waiting for automatic snapd restart...
lxd 5.1-4ae3604 from Canonical✓ installed

~
root @ lxd1 # lxc remote add epyc 192.168.178.5 --protocol lxd --password QUyLH9mReRueF2k
If this is your first time running LXD on this machine, you should also run: lxd init
To start your first container, try: lxc launch ubuntu:22.04
Or for a virtual machine: lxc launch ubuntu:22.04 --vm

Generating a client certificate. This may take a minute...
Certificate fingerprint: 8cce00621d1cfd14ccfad6726183439c6c7ab1315f9e8f9a1db0c3d528b9f3d9
ok (y/n/[fingerprint])? y
Error: not found

Server logs are the same as with my lxd client on windows

tomponline commented 2 years ago

What version is the LXD server running on 192.168.178.5?

Can you go to it locally and run:

lxc query /1.0/certificates
vosdev commented 2 years ago

What version is the LXD server running on 192.168.178.5?

Can you go to it locally and run:

lxc query /1.0/certificates

The server on 192.168.178.5 runs the latest snap, 5.1-4ae3604. It's an Ubuntu 20.04 server.

~
root @ epyc # lxc query /1.0/certificates
[]

~
root @ epyc # lxc version
Client version: 5.1
Server version: 5.1

~
root @ epyc # snap list
Name    Version      Rev    Tracking       Publisher   Notes
core18  20220309     2344   latest/stable  canonical✓  base
core20  20220329     1434   latest/stable  canonical✓  base
lxd     5.1-4ae3604  23001  latest/stable  canonical✓  -
snapd   2.55.3       15534  latest/stable  canonical✓  snapd

Just to be sure I just rebooted the server but the "Error: Not Found" remains

stgraber commented 2 years ago

Can you run remote add with --debug added?

vosdev commented 2 years ago

Ha I found the issue!

The HTTPS listener was not configured. I only configured the metrics listener and used :8443 for it...

I changed the metrics to :8444 and added the config for core.https_listener for :8443 and everything works.

Apologies!

stgraber commented 2 years ago

That would explain it :)