BUG: unable to initialize proxmox api client not authorized to access endpoint

[dnviti]

What steps did you take and what happened: After installing all prerequisites and running the command

clusterctl init --infrastructure proxmox --ipam in-cluster --core cluster-api:v1.5.3

when watching pods creating i notice that the capmox-controller is in CrashLoopback. Pod Logs:

I0124 23:54:17.007521       1 main.go:87] "setup: starting capmox"
I0124 23:54:17.008105       1 listener.go:44] "controller-runtime/metrics: Metrics server is starting to listen" addr="localhost:8080"
I0124 23:54:17.008556       1 main.go:126] "setup: feature gates: ClusterTopology=false\n"
E0124 23:54:20.031775       1 main.go:133] "setup: unable to setup proxmox API client" err="unable to initialize proxmox api client: not authorized to access endpoint"

clusterctl.yaml

PROXMOX_URL: "https://pve.dev.local/api2/json"
PROXMOX_TOKEN: "capi@pve!token1"
PROXMOX_SECRET: "REDACTED"
PROXMOX_SOURCENODE: "pve01"
TEMPLATE_VMID: "9000"
ALLOWED_NODES: "[pve01,pve02,pve03]"
VM_SSH_KEYS: "ssh-rsa ..."
CONTROL_PLANE_ENDPOINT_IP: "192.168.254.40"
NODE_IP_RANGES: "[192.168.254.50-192.168.254.80]"
GATEWAY: "192.168.254.1"
IP_PREFIX: "24"
DNS_SERVERS: "[192.168.254.1]"
BRIDGE: "vmbr0"
BOOT_VOLUME_DEVICE: "scsi0"
BOOT_VOLUME_SIZE: "64"
NUM_SOCKETS: "1"
NUM_CORES: "4"
MEMORY_MIB: "4096"
EXP_CLUSTER_RESOURCE_SET: "false"
CONTROL_PLANE_MACHINE_COUNT: "3"
WORKER_MACHINE_COUNT: "3"

providers:
  - name: in-cluster
    url: https://github.com/kubernetes-sigs/cluster-api-ipam-provider-in-cluster/releases/latest/ipam-components.yaml
    type: IPAMProvider

capi@pve!token1 is proxmox Administrator on / so it's cluster admin. Yesyed the user on terraform and packer and ansible and everything works fine there.

What did you expect to happen: The capmox-controller should connect to proxmox and continue provisioning

Anything else you would like to add: Using kind as local kubernetes provisioner and docker rootless as driver, everything it at default configuration.

Environment:

• Cluster-api-provider-proxmox version: 0.1.1
• Kubernetes version: (use kubectl version):

  Client Version: v1.29.1
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: v1.27.3

• OS (e.g. from /etc/os-release):

  
  NAME="AlmaLinux"
  VERSION="9.3 (Shamrock Pampas Cat)"
  ID="almalinux"
  ID_LIKE="rhel centos fedora"
  VERSION_ID="9.3"
  PLATFORM_ID="platform:el9"
  PRETTY_NAME="AlmaLinux 9.3 (Shamrock Pampas Cat)"
  ANSI_COLOR="0;34"
  LOGO="fedora-logo-icon"
  CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
  HOME_URL="https://almalinux.org/"
  DOCUMENTATION_URL="https://wiki.almalinux.org/"
  BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9" ALMALINUX_MANTISBT_PROJECT_VERSION="9.3" REDHAT_SUPPORT_PRODUCT="AlmaLinux" REDHAT_SUPPORT_PRODUCT_VERSION="9.3"

It's systematic so i must be doing something wrong...

[mcbenjemaa]

@dnviti the PROXMOX_URL: "https://pve.dev.local/api2/json" is wrong,

Please just use PROXMOX_URL: "https://pve.dev.local"

[dnviti]

It didn't work either way.

BTW this morning it gives me the error:

[default@capi-bootstrapper ~]$ clusterctl init --infrastructure proxmox --ipam in-cluster --core cluster-api:v1.5.3
Fetching providers
Error: failed to get provider components for the "proxmox" provider: failed to get repository client for the InfrastructureProvider with name proxmox: error creating the GitHub repository client: failed to get latest release: release not found for version v0.2.0, please retry later or set "GOPROXY=off" to get the current stable release: 404 Not Found

if i only run clusterctl init it works but it does not initialize the proxmox provider.

[mcbenjemaa]

yeah, we are currently releasing v0.2.0.

you can enforce the version by

clusterctl init --infrastructure proxmox:v0.1.1 --ipam in-cluster --core cluster-api:v1.5.3
Fetching providers

[dnviti]

yeah, we are currently releasing v0.2.0.

you can enforce the version by

clusterctl init --infrastructure proxmox:v0.1.1 --ipam in-cluster --core cluster-api:v1.5.3
Fetching providers

Nope, not working either, i'm going to wait for the v0.2.0 and try again. Now it just hangs on Fetching providers... Anyway, where can i check available providers/infrastructures? can't find the list anywhere...

[mcbenjemaa]

please retry or put this in your clusterctl

providers:
  - name: proxmox
    url: https://github.com/ionos-cloud/cluster-api-provider-proxmox/releases/download/v0.1.1/infrastructure-components.yaml
    type: InfrastructureProvider

[dnviti]

It starts but gives the same error

I0124 23:54:17.007521       1 main.go:87] "setup: starting capmox"
I0124 23:54:17.008105       1 listener.go:44] "controller-runtime/metrics: Metrics server is starting to listen" addr="localhost:8080"
I0124 23:54:17.008556       1 main.go:126] "setup: feature gates: ClusterTopology=false\n"
E0124 23:54:20.031775       1 main.go:133] "setup: unable to setup proxmox API client" err="unable to initialize proxmox api client: not authorized to access endpoint"

Can't really figure out why, i strictly followed documentation steps.

[mcbenjemaa]

@dnviti Do you think the token is correct?

OR maybe you should add the port PROXMOX_URL: "https://pve.dev.local:8006"

[dnviti]

@dnviti you're sure the token is correct?

it worked using a token from user root@pam could it be the pve realm has different capabilities?

[dnviti]

@dnviti Do you think the token is correct?

OR maybe you should add the port PROXMOX_URL: "https://pve.dev.local:8006"

nope that, my access is granted and managed by an HAProxy so i use another domain, that was a random example

[mcbenjemaa]

I think I remember this issue:

Please disable privilege separation or use root token,

[dnviti]

I think I remember this issue:

Please disable privilege separation or use root token,

yes, i disabled previously "Privilege Separation" also on the other user and it worked using ansible.

Now all started and that is the log.

NAME                                                                   READY  SEVERITY  REASON                       SINCE  MESSAGE
Cluster/proxmox-quickstart                                             False  Warning   ScalingUp                    4m16s  Scaling up control plane to 3 replicas (actual 1)
├─ClusterInfrastructure - ProxmoxCluster/proxmox-quickstart            True                                          4m24s
├─ControlPlane - KubeadmControlPlane/proxmox-quickstart-control-plane  False  Warning   ScalingUp                    4m16s  Scaling up control plane to 3 replicas (actual 1)
│ └─Machine/proxmox-quickstart-control-plane-xhdfp                     False  Warning   CloningFailed                4m21s  1 of 2 completed
└─Workers
  └─MachineDeployment/proxmox-quickstart-workers                       False  Warning   WaitingForAvailableMachines  4m24s  Minimum availability requires 3 replicas, current 0 available
    └─3 Machines...                                                    False  Info      WaitingForBootstrapData      4m23s  See proxmox-quickstart-workers-qllcb-ds486, proxmox-quickstart-workers-qllcb-twctl, ...

Should the VM template be created using a specific method? i used a standard almalinux cloud init image (Generic Cloud)...

[mcbenjemaa]

Does your image have kubeadm, kubelet, kubectl, containerd installed?

[dnviti]

Does your image have kubeadm, kubelet, kubectl, containerd installed?

nope, if i only need those dependencies i'll cook a new one right away!

[mcbenjemaa]

Does your image have kubeadm, kubelet, kubectl, containerd installed?

nope, if i only need those dependencies i'll cook a new one right away!

you can add this almalinux to image-builder otherwise, you can build ubuntu images.

[mcbenjemaa]

v0.2.0 is now released https://github.com/ionos-cloud/cluster-api-provider-proxmox/releases/tag/v0.2.0

Which means that the issue that you had before, should be fixed.

[dnviti]

v0.2.0 is now released https://github.com/ionos-cloud/cluster-api-provider-proxmox/releases/tag/v0.2.0

Which means that the issue that you had before, should be fixed.

I also developed this script: https://gitlab.com/dnviti/Bash-Scripts/-/blob/master/capi-proxmox-deps.sh its purpose it's to prepare an image from scratch in au automated way. it references another script of mine, to install docker and all its deps.

[dnviti]

Installed all dependencies into a new VM tempate and still CloningFailed message appears upon cluster deploy. It's not even trying to clone it anyway.

[mcbenjemaa]

join this channel to get support: https://kubernetes.slack.com/archives/C06FC9P0FK7