[BUG] Nodes name are different in RKE2 downstream cluster

[YunSangJun]

Rancher Server Setup

• Rancher version: 2.6.6
• Installation option (Docker install/Helm Chart): Helm Chart
  • If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): RKE2 / v1.22.9
• Proxy/Cert Details: Proxy(None) / Cert(Sectigo)

Information about the Cluster

• Kubernetes version: RKE2 / v1.23.6+rke2r2
• Cluster Type (Local/Downstream): Downstream
  • If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider): Infrastructure Provider / Openstack and Custom node driver

    Nodes name are different in RKE2 downstream cluster.

    To Reproduce

    1) Create a RKE2 downstream cluster 2) Confirm nodes name

    Result A first node name is different from other two nodes.

    kubectl get nodes NAME STATUS ROLES AGE VERSION ksy-rke2-d262-m00-2ab63e53-cd87r Ready control-plane,etcd,master 6d21h v1.23.6+rke2r2 ksy-rke2-d262-w00-c67d39cb-hh8wm.ap-northeast-arc.stg.h53 Ready worker 6d21h v1.23.6+rke2r2 ksy-rke2-d262-w01-2d51ab56-t47k9.ap-northeast-arc.stg.h53 Ready worker 20h v1.23.6+rke2r2

    Expected Result

    I expect that all nodes name are same such as example 1 or 2. (I prefer example 1) ex 1)

    kubectl get nodes NAME STATUS ROLES AGE VERSION ksy-rke2-d262-m00-2ab63e53-cd87r Ready control-plane,etcd,master 6d21h v1.23.6+rke2r2 ksy-rke2-d262-w00-c67d39cb-hh8wm Ready worker 6d21h v1.23.6+rke2r2 ksy-rke2-d262-w01-2d51ab56-t47k9 Ready worker 20h v1.23.6+rke2r2

    ex 2)

    kubectl get nodes NAME STATUS ROLES AGE VERSION ksy-rke2-d262-m00-2ab63e53-cd87r.ap-northeast-arc.stg.h53 Ready control-plane,etcd,master 6d21h v1.23.6+rke2r2 ksy-rke2-d262-w00-c67d39cb-hh8wm.ap-northeast-arc.stg.h53 Ready worker 6d21h v1.23.6+rke2r2 ksy-rke2-d262-w01-2d51ab56-t47k9.ap-northeast-arc.stg.h53 Ready worker 20h v1.23.6+rke2r2

    Screenshots

    kubectl get nodes NAME STATUS ROLES AGE VERSION ksy-rke2-d262-m00-2ab63e53-cd87r Ready control-plane,etcd,master 6d21h v1.23.6+rke2r2 ksy-rke2-d262-w00-c67d39cb-hh8wm.ap-northeast-arc.stg.h53 Ready worker 6d21h v1.23.6+rke2r2 ksy-rke2-d262-w01-2d51ab56-t47k9.ap-northeast-arc.stg.h53 Ready worker 20h v1.23.6+rke2r2

    Additional context

    A node's hostname change after a vm created. I pass a post scripts in Openstack userdata and the script add postfix to the hostname. This is a nessary process in our environment. e.g) sj-test-33c9-m01-af2fe3df-nk9bq => sj-test-33c9-m01-af2fe3df-nk9bq.ap-northeast-arc.prd.h53 In my thought, it seems that Rancher refer the hostname when set node name. So there might be timing issue. e.g) sj-test-33c9-m00-29d3778c-gq2m8 => Rancher refer hostname before running the post scripts. sj-test-33c9-m00-776d4bfb7-5qg2t.ap-northeast-arc.prd.h53 => Rancher refer hostname after running the post scriptss.

    === userdata === $ curl http://169.254.169.254/openstack/latest/user_data

    cloud-config

    hostname: ksy-rke2-d262-w00-c67d39cb-hh8wm runcmd: -sh /usr/local/custom_script/install.sh -curl -sfL https://.../post-scripts/ap-northeast-arc/vm.sh | ZONE_TYPE=stg sh - => Change hostname in this script set_hostname:

    • ksy-rke2-d262-w00-c67d39cb-hh8wm write_files: ...

    After remove the script the issue has gone. All node's name are same.

    kubectl get nodes NAME STATUS ROLES AGE VERSION ksy-rke2-d262-m00-2ab63e53-cd87r Ready control-plane,etcd,master 6d21h v1.23.6+rke2r2 ksy-rke2-d262-w00-c67d39cb-hh8wm Ready worker 6d21h v1.23.6+rke2r2 ksy-rke2-d262-w01-2d51ab56-t47k9 Ready worker 20h v1.23.6+rke2r2

[YunSangJun]

If change the location of the script that changes the hostname as below, all nodes will have the same name. In my thought, it seems that Rancher refer the hostname when set node name. Is it by design?.

===== [centos@test-f82e-w00-9ab86e06-lxldm ~]$ curl http://169.254.169.254/openstack/latest/user_data

cloud-config

hostname: test-f82e-w00-9ab86e06-lxldm runcmd:

• sh /usr/local/custom_script/install.sh
• hostnamectl set-hostname "$(cat /etc/hostname).ap-northeast-arc.stg.h53" => Change a hostname at here
• curl -sfL https://.../post-scripts/ap-northeast-arc/vm.sh => Comment the script which changes a hostname in postscript | ZONE_TYPE=stg sh - set_hostname:
• test-f82e-w00-9ab86e06-lxldm write_files: ...

//host name [centos@test-f82e-w00-9ab86e06-lxldm ~]$ hostname test-f82e-w00-9ab86e06-lxldm.ap-northeast-arc.stg.h53

//node name 1) test1

kubectl get nodes NAME STATUS ROLES AGE VERSION test-f82e-m00-a6513c24-sqzzc.ap-northeast-arc.stg.h53 Ready control-plane,etcd,master 17m v1.23.6+rke2r2 test-f82e-w00-9ab86e06-5pphd.ap-northeast-arc.stg.h53 Ready worker 15m v1.23.6+rke2r2 test-f82e-w00-9ab86e06-ftkts.ap-northeast-arc.stg.h53 Ready worker 15m v1.23.6+rke2r2 test-f82e-w00-9ab86e06-lbt2p.ap-northeast-arc.stg.h53 Ready worker 15m v1.23.6+rke2r2 test-f82e-w00-9ab86e06-lxldm.ap-northeast-arc.stg.h53 Ready worker 15m v1.23.6+rke2r2

2) test2

kubectl get nodes NAME STATUS ROLES AGE VERSION test2-c9f9-m00-94bbe3c9-vlmmg.ap-northeast-arc.stg.h53 Ready control-plane,etcd,master 7m20s v1.23.6+rke2r2 test2-c9f9-w00-0dcca191-hb7n6.ap-northeast-arc.stg.h53 Ready worker 5m7s v1.23.6+rke2r2

3) test3

kubectl get nodes NAME STATUS ROLES AGE VERSION test3-0938-m00-aac97a25-22cdg.ap-northeast-arc.stg.h53 Ready control-plane,etcd,master 5m17s v1.23.6+rke2r2 test3-0938-m00-aac97a25-ngzv9.ap-northeast-arc.stg.h53 Ready control-plane,etcd,master 5m54s v1.23.6+rke2r2 test3-0938-m00-aac97a25-nmtp6.ap-northeast-arc.stg.h53 Ready control-plane,etcd,master 8m44s v1.23.6+rke2r2 test3-0938-w00-636dc298-qx7rl.ap-northeast-arc.stg.h53 Ready worker 4m35s v1.23.6+rke2r2 test3-0938-w00-636dc298-r5phr.ap-northeast-arc.stg.h53 Ready worker 4m35s v1.23.6+rke2r2