RKE2 installation with Cilium on RHEL9

[ObieBent]

Environmental Info: RKE2 Version: v1.26.5+rke2r1

Node(s) CPU architecture, OS, and Version: # uname -a Linux dwst-cp-1 5.14.0-162.23.1.el9_1.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 23 20:08:28 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

• OS Version

  
  # cat /etc/os-release
  NAME="Red Hat Enterprise Linux"
  VERSION="9.1 (Plow)"
  ID="rhel"
  ID_LIKE="fedora"
  VERSION_ID="9.1"
  PLATFORM_ID="platform:el9"
  PRETTY_NAME="Red Hat Enterprise Linux 9.1 (Plow)"
  ANSI_COLOR="0;31"
  LOGO="fedora-logo-icon"
  CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
  HOME_URL="https://www.redhat.com/"
  DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/9/"
  BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9" REDHAT_BUGZILLA_PRODUCT_VERSION=9.1 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="9.1"

- SELinux is activated 

getenforce

Enforcing

rpm -qav | grep selinux

libselinux-3.4-3.el9.x86_64 libselinux-utils-3.4-3.el9.x86_64 rpm-plugin-selinux-4.16.1.3-19.el9_1.x86_64 selinux-policy-34.1.43-1.el9_1.2.noarch selinux-policy-targeted-34.1.43-1.el9_1.2.noarch python3-libselinux-3.4-3.el9.x86_64 container-selinux-2.189.0-1.el9.noarch rke2-selinux-0.16-1.el8.noarch

- Add-on config file

MTU: 0 affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution:

• labelSelector: matchLabels: k8s-app: cilium topologyKey: kubernetes.io/hostname agent: true agentNotReadyTaintKey: node.cilium.io/agent-not-ready aksbyocni: enabled: false alibabacloud: enabled: false annotateK8sNode: false autoDirectNodeRoutes: false azure: enabled: false bandwidthManager: bbr: false enabled: false bgp: announce: loadbalancerIP: false podCIDR: false enabled: false bgpControlPlane: enabled: false bpf: clockProbe: false ctAnyMax: null ctTcpMax: null hostLegacyRouting: null lbExternalClusterIP: false lbMapMax: 65536 mapDynamicSizeRatio: null masquerade: null monitorAggregation: medium monitorFlags: all monitorInterval: 5s natMax: null neighMax: null policyMapMax: 16384 preallocateMaps: false root: /sys/fs/bpf tproxy: null vlanBypass: null certgen: extraVolumeMounts: [] extraVolumes: [] image: override: null pullPolicy: IfNotPresent repository: rancher/mirrored-cilium-certgen tag: v0.1.8 podLabels: {} tolerations: [] ttlSecondsAfterFinished: 1800 cgroup: autoMount: enabled: true resources: {} hostRoot: /run/cilium/cgroupv2 cleanBpfState: false cleanState: false cluster: id: 0 name: default clustermesh: apiserver: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution:
  • labelSelector: matchLabels: k8s-app: clustermesh-apiserver topologyKey: kubernetes.io/hostname etcd: image: override: null pullPolicy: IfNotPresent repository: rancher/mirrored-coreos-etcd tag: v3.5.4 init: resources: {} resources: {} securityContext: {} extraEnv: [] extraVolumeMounts: [] extraVolumes: [] image: override: null pullPolicy: IfNotPresent repository: rancher/mirrored-cilium-clustermesh-apiserver tag: v1.13.2 useDigest: false nodeSelector: kubernetes.io/os: linux podAnnotations: {} podDisruptionBudget: enabled: false maxUnavailable: 1 minAvailable: null podLabels: {} podSecurityContext: {} priorityClassName: '' replicas: 1 resources: {} securityContext: {} service: annotations: {} nodePort: 32379 type: NodePort tls: admin: cert: '' key: '' auto: certManagerIssuerRef: {} certValidityDuration: 1095 enabled: true method: helm ca: cert: '' key: '' client: cert: '' key: '' remote: cert: '' key: '' server: cert: '' extraDnsNames: [] extraIpAddresses: [] key: '' tolerations: [] topologySpreadConstraints: [] updateStrategy: rollingUpdate: maxUnavailable: 1 type: RollingUpdate config: clusters: [] domain: mesh.cilium.io enabled: false useAPIServer: false cni: binPath: /opt/cni/bin chainingMode: portmap confFileMountPath: /tmp/cni-configuration confPath: /etc/cni/net.d configMapKey: cni-config customConf: false exclusive: true hostConfDirMountPath: /host/etc/cni/net.d install: true logFile: /var/run/cilium/cilium-cni.log uninstall: true conntrackGCInterval: '' containerRuntime: integration: none crdWaitTimeout: '' customCalls: enabled: false daemon: allowedConfigOverrides: null blockedConfigOverrides: null configSources: null runPath: /var/run/cilium debug: enabled: false verbose: null disableEndpointCRD: 'false' dnsPolicy: '' dnsProxy: dnsRejectResponseCode: refused enableDnsCompression: true endpointMaxIpPerHostname: 50 idleConnectionGracePeriod: 0s maxDeferredConnectionDeletes: 10000 minTtl: 3600 preCache: '' proxyPort: 0 proxyResponseMaxDelay: 100ms egressGateway: enabled: false installRoutes: false enableCiliumEndpointSlice: false enableCnpStatusUpdates: false enableCriticalPriorityClass: true enableIPv4Masquerade: true enableIPv6BIGTCP: false enableIPv6Masquerade: true enableK8sEventHandover: false enableK8sTerminatingEndpoint: true enableRuntimeDeviceDetection: false enableXTSocketFallback: true encryption: enabled: false interface: '' ipsec: interface: '' keyFile: '' mountPath: '' secretName: '' keyFile: keys mountPath: /etc/ipsec nodeEncryption: false secretName: cilium-ipsec-keys type: ipsec wireguard: userspaceFallback: false endpointHealthChecking: enabled: true endpointRoutes: enabled: false endpointStatus: enabled: false status: '' eni: awsEnablePrefixDelegation: false awsReleaseExcessIPs: false ec2APIEndpoint: '' enabled: false eniTags: {} gcInterval: '' gcTags: {} iamRole: '' instanceTagsFilter: [] subnetIDsFilter: [] subnetTagsFilter: [] updateEC2AdapterLimitViaAPI: false etcd: clusterDomain: cluster.local enabled: false endpoints:
    • https://CHANGE-ME:2379 extraArgs: [] extraVolumeMounts: [] extraVolumes: [] image: override: null pullPolicy: IfNotPresent repository: rancher/mirrored-cilium-cilium-etcd-operator tag: v2.0.7 k8sService: false nodeSelector: kubernetes.io/os: linux podAnnotations: {} podDisruptionBudget: enabled: false maxUnavailable: 1 minAvailable: null podLabels: {} podSecurityContext: {} priorityClassName: '' resources: {} securityContext: {} ssl: false tolerations:
    • operator: Exists topologySpreadConstraints: [] updateStrategy: rollingUpdate: maxSurge: 1 maxUnavailable: 1 type: RollingUpdate externalIPs: enabled: false externalWorkloads: enabled: false extraArgs: [] extraConfig: {} extraContainers: [] extraEnv: [] extraHostPathMounts: [] extraVolumeMounts: [] extraVolumes: [] gatewayAPI: enabled: false secretsNamespace: create: true name: cilium-secrets sync: true gke: enabled: false global: systemDefaultRegistry: '' healthChecking: true healthPort: 9879 hostFirewall: enabled: false hostPort: enabled: false hubble: enabled: false listenAddress: ':4244' metrics: dashboards: annotations: {} enabled: false label: grafana_dashboard labelValue: '1' namespace: null enableOpenMetrics: false enabled: null port: 9965 serviceAnnotations: {} serviceMonitor: annotations: {} enabled: false interval: 10s labels: {} metricRelabelings: null relabelings:
    • replacement: ${1} sourceLabels:
      • __meta_kubernetes_pod_node_name targetLabel: node peerService: clusterDomain: cluster.local enabled: true targetPort: 4244 preferIpv6: false relay: affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution:
  • labelSelector: matchLabels: k8s-app: cilium topologyKey: kubernetes.io/hostname dialTimeout: null enabled: false extraEnv: [] image: override: null pullPolicy: IfNotPresent repository: rancher/mirrored-cilium-hubble-relay tag: v1.13.2 useDigest: false listenHost: '' listenPort: '4245' nodeSelector: kubernetes.io/os: linux podAnnotations: {} podDisruptionBudget: enabled: false maxUnavailable: 1 minAvailable: null podLabels: {} pprof: address: localhost enabled: false port: 6062 priorityClassName: '' prometheus: enabled: false port: 9966 serviceMonitor: annotations: {} enabled: false interval: 10s labels: {} metricRelabelings: null relabelings: null replicas: 1 resources: {} retryTimeout: null rollOutPods: false securityContext: {} service: nodePort: 31234 type: ClusterIP sortBufferDrainTimeout: null sortBufferLenMax: null terminationGracePeriodSeconds: 1 tls: client: cert: '' key: '' server: cert: '' enabled: false extraDnsNames: [] extraIpAddresses: [] key: '' tolerations: [] topologySpreadConstraints: [] updateStrategy: rollingUpdate: maxUnavailable: 1 type: RollingUpdate skipUnknownCGroupIDs: null socketPath: /var/run/cilium/hubble.sock tls: auto: certManagerIssuerRef: {} certValidityDuration: 1095 enabled: true method: helm schedule: 0 0 1 /4 ca: cert: '' key: '' enabled: true server: cert: '' extraDnsNames: [] extraIpAddresses: [] key: '' ui: affinity: {} backend: extraEnv: [] extraVolumeMounts: [] extraVolumes: [] image: override: null pullPolicy: IfNotPresent repository: rancher/mirrored-cilium-hubble-ui-backend tag: v0.11.0 resources: {} securityContext: {} baseUrl: / enabled: false frontend: extraEnv: [] extraVolumeMounts: [] extraVolumes: [] image: override: null pullPolicy: IfNotPresent repository: rancher/mirrored-cilium-hubble-ui tag: v0.11.0 resources: {} securityContext: {} server: ipv6: enabled: true ingress: annotations: {} className: '' enabled: false hosts:
    • chart-example.local tls: [] nodeSelector: kubernetes.io/os: linux podAnnotations: {} podDisruptionBudget: enabled: false maxUnavailable: 1 minAvailable: null podLabels: {} priorityClassName: '' replicas: 1 rollOutPods: false securityContext: enabled: true fsGroup: 1001 runAsGroup: 1001 runAsUser: 1001 service: annotations: {} nodePort: 31235 type: ClusterIP standalone: enabled: false tls: certsVolume: {} tls: client: cert: '' key: '' tolerations: [] topologySpreadConstraints: [] updateStrategy: rollingUpdate: maxUnavailable: 1 type: RollingUpdate identityAllocationMode: crd identityChangeGracePeriod: '' image: override: null pullPolicy: IfNotPresent repository: rancher/mirrored-cilium-cilium tag: v1.13.2 useDigest: false imagePullSecrets: null ingressController: enabled: false enforceHttps: true ingressLBAnnotationPrefixes:
    • service.beta.kubernetes.io
    • service.kubernetes.io
    • cloud.google.com loadbalancerMode: dedicated secretsNamespace: create: true name: cilium-secrets sync: true service: annotations: {} insecureNodePort: null labels: {} name: cilium-ingress secureNodePort: null type: LoadBalancer installNoConntrackIptablesRules: false ipMasqAgent: enabled: false ipam: mode: kubernetes operator: clusterPoolIPv4MaskSize: 24 clusterPoolIPv4PodCIDR: 10.0.0.0/8 clusterPoolIPv4PodCIDRList: [] clusterPoolIPv6MaskSize: 120 clusterPoolIPv6PodCIDR: fd00::/104 clusterPoolIPv6PodCIDRList: [] externalAPILimitBurstSize: null externalAPILimitQPS: null ipv4: enabled: true ipv4NativeRoutingCIDR: '' ipv6: enabled: false ipv6NativeRoutingCIDR: '' k8s: {} k8sServiceHost: '' k8sServicePort: '' keepDeprecatedLabels: false keepDeprecatedProbes: false kubeConfigPath: '' kubeProxyReplacementHealthzBindAddr: '' l2NeighDiscovery: enabled: true refreshPeriod: 30s l7Proxy: true livenessProbe: failureThreshold: 10 periodSeconds: 30 loadBalancer: l7: algorithm: round_robin backend: disabled ports: [] localRedirectPolicy: false logSystemLoad: false maglev: {} monitor: enabled: false name: cilium nat46x64Gateway: enabled: false nodePort: autoProtectPortRange: true bindProtection: true enableHealthCheck: true enabled: false nodeSelector: kubernetes.io/os: linux nodeinit: affinity: {} bootstrapFile: /tmp/cilium-bootstrap.d/cilium-bootstrap-time enabled: false extraEnv: [] image: override: null pullPolicy: IfNotPresent repository: quay.io/cilium/startup-script tag: d69851597ea019af980891a4628fb36b7880ec26 nodeSelector: kubernetes.io/os: linux podAnnotations: {} podLabels: {} priorityClassName: '' resources: requests: cpu: 100m memory: 100Mi securityContext: capabilities: add:
    • SYS_MODULE
    • NET_ADMIN
    • SYS_ADMIN
    • SYS_CHROOT
    • SYS_PTRACE privileged: false seLinuxOptions: level: s0 type: spc_t tolerations:
    • operator: Exists updateStrategy: type: RollingUpdate operator: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution:
    • labelSelector: matchLabels: io.cilium/app: operator topologyKey: kubernetes.io/hostname dnsPolicy: '' enabled: true endpointGCInterval: 5m0s extraArgs: [] extraEnv: [] extraHostPathMounts: [] extraVolumeMounts: [] extraVolumes: [] identityGCInterval: 15m0s identityHeartbeatTimeout: 30m0s image: override: null pullPolicy: IfNotPresent repository: rancher/mirrored-cilium-operator suffix: '' tag: v1.13.2 useDigest: false nodeGCInterval: 5m0s nodeSelector: kubernetes.io/os: linux podAnnotations: {} podDisruptionBudget: enabled: false maxUnavailable: 1 minAvailable: null podLabels: {} podSecurityContext: {} pprof: address: localhost enabled: false port: 6061 priorityClassName: '' prometheus: enabled: true port: 9963 serviceMonitor: annotations: {} enabled: false interval: 10s labels: {} metricRelabelings: null relabelings: null removeNodeTaints: true replicas: 2 resources: {} rollOutPods: false securityContext: {} setNodeNetworkStatus: true skipCNPStatusStartupClean: false skipCRDCreation: false tolerations:
    • operator: Exists topologySpreadConstraints: [] unmanagedPodWatcher: intervalSeconds: 15 restart: true updateStrategy: rollingUpdate: maxSurge: 1 maxUnavailable: 1 type: RollingUpdate pmtuDiscovery: enabled: false podAnnotations: {} podLabels: {} podSecurityContext: {} policyEnforcementMode: default portmapPlugin: image: repository: rancher/hardened-cni-plugins tag: v1.0.1-build20221011 pprof: address: localhost enabled: false port: 6060 preflight: affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution:
    • labelSelector: matchLabels: k8s-app: cilium topologyKey: kubernetes.io/hostname enabled: false extraEnv: [] extraVolumeMounts: [] extraVolumes: [] image: override: null pullPolicy: IfNotPresent repository: rancher/mirrored-cilium-cilium tag: v1.13.2 useDigest: false nodeSelector: kubernetes.io/os: linux podAnnotations: {} podDisruptionBudget: enabled: false maxUnavailable: 1 minAvailable: null podLabels: {} podSecurityContext: {} priorityClassName: '' resources: {} securityContext: {} terminationGracePeriodSeconds: 1 tofqdnsPreCache: '' tolerations:
    • effect: NoSchedule key: node.kubernetes.io/not-ready
    • effect: NoSchedule key: node-role.kubernetes.io/master
    • effect: NoSchedule key: node-role.kubernetes.io/control-plane
    • effect: NoSchedule key: node.cloudprovider.kubernetes.io/uninitialized value: 'true'
    • key: CriticalAddonsOnly operator: Exists updateStrategy: type: RollingUpdate validateCNPs: true priorityClassName: '' prometheus: enabled: true metrics: null port: 9962 serviceMonitor: annotations: {} enabled: false interval: 10s labels: {} metricRelabelings: null relabelings:
• replacement: ${1} sourceLabels:
  • __meta_kubernetes_pod_node_name targetLabel: node proxy: prometheus: enabled: true port: '9964' sidecarImageRegex: cilium/istio_proxy rbac: create: true readinessProbe: failureThreshold: 3 periodSeconds: 30 remoteNodeIdentity: true resourceQuotas: cilium: hard: pods: 10k enabled: false operator: hard: pods: '15' resources: {} rollOutCiliumPods: false sctp: enabled: false securityContext: capabilities: applySysctlOverwrites:
• SYS_ADMIN
• SYS_CHROOT
• SYS_PTRACE ciliumAgent:
• CHOWN
• KILL
• NET_ADMIN
• NET_RAW
• IPC_LOCK
• SYS_MODULE
• SYS_ADMIN
• SYS_RESOURCE
• DAC_OVERRIDE
• FOWNER
• SETGID
• SETUID cleanCiliumState:
• NET_ADMIN
• SYS_MODULE
• SYS_ADMIN
• SYS_RESOURCE mountCgroup:
• SYS_ADMIN
• SYS_CHROOT
• SYS_PTRACE privileged: false seLinuxOptions: level: s0 type: spc_t serviceAccounts: cilium: annotations: {} automount: true create: true name: cilium clustermeshApiserver: annotations: {} automount: true create: true name: clustermesh-apiserver clustermeshcertgen: annotations: {} automount: true create: true name: clustermesh-apiserver-generate-certs etcd: annotations: {} automount: true create: true name: cilium-etcd-operator hubblecertgen: annotations: {} automount: true create: true name: hubble-generate-certs operator: annotations: {} automount: true create: true name: cilium-operator preflight: annotations: {} automount: true create: true name: cilium-pre-flight relay: annotations: {} automount: false create: true name: hubble-relay ui: annotations: {} automount: true create: true name: hubble-ui sleepAfterInit: false socketLB: enabled: false sockops: enabled: false startupProbe: failureThreshold: 105 periodSeconds: 2 svcSourceRangeCheck: true synchronizeK8sNodes: true terminationGracePeriodSeconds: 1 tls: ca: cert: '' certValidityDuration: 1095 key: '' secretsBackend: local tolerations:
  • operator: Exists tunnel: vxlan tunnelPort: 0 updateStrategy: rollingUpdate: maxUnavailable: 2 type: RollingUpdate vtep: cidr: '' enabled: false endpoint: '' mac: '' mask: '' waitForKubeProxy: false wellKnownIdentities: enabled: false cilium: ipv6: enabled: true

Cluster Configuration:
1 server and 2 agents

**Describe the bug:**
I'm trying to setup a downstream cluster RKE2 with Cilium on RHEL9.
I run the curl command provided from Rancher UI on the first server dedicated to hosting control plane services, but it seems like the rancher-system-agent
doesn't perform anything. It stays at this stage forever. 

**Steps To Reproduce:**

- Registration of the  server 

curl -fL https://rms.buzz.lab/system-agent-install.sh | sudo sh -s - --server https://rms.buzz.lab --label 'cattle.io/os=linux' --token vpwx4vjrtgk9qc9cvrv2tb8xph7dfttcpqqwvnxnjbdq8q2mn2s5kf --ca-checksum e25afd5076b49a2a5bac4f2669999b33779b1de85d6e1c7c77e81f975a1112db --etcd --controlplane

% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 30889 0 30889 0 0 257k 0 --:--:-- --:--:-- --:--:-- 257k [INFO] Label: cattle.io/os=linux [INFO] Role requested: etcd [INFO] Role requested: controlplane [INFO] Using default agent configuration directory /etc/rancher/agent [INFO] Using default agent var directory /var/lib/rancher/agent [INFO] Determined CA is not necessary to connect to Rancher [INFO] Successfully tested Rancher connection [INFO] Downloading rancher-system-agent binary from https://rms.buzz.lab/assets/rancher-system-agent-amd64 [INFO] Successfully downloaded the rancher-system-agent binary. [INFO] Downloading rancher-system-agent-uninstall.sh script from https://rms.buzz.lab/assets/system-agent-uninstall.sh [INFO] Successfully downloaded the rancher-system-agent-uninstall.sh script. [INFO] Generating Cattle ID [INFO] Successfully downloaded Rancher connection information [INFO] systemd: Creating service file [INFO] Creating environment file /etc/systemd/system/rancher-system-agent.env [INFO] Enabling rancher-system-agent.service Created symlink /etc/systemd/system/multi-user.target.wants/rancher-system-agent.service → /etc/systemd/system/rancher-system-agent.service. [INFO] Starting/restarting rancher-system-agent.service

- Fails to start the rke2-server through the rancher-system-agent service 

journalctl -u rancher-system-agent --no-pager

Jan 03 03:41:34 dwst-cp-1 systemd[1]: Started Rancher System Agent. Jan 03 03:41:34 dwst-cp-1 rancher-system-agent[12207]: time="2024-01-03T03:41:34+01:00" level=info msg="Rancher System Agent version v0.3.3 (9e827a5) is starting" Jan 03 03:41:34 dwst-cp-1 rancher-system-agent[12207]: time="2024-01-03T03:41:34+01:00" level=info msg="Using directory /var/lib/rancher/agent/work for work" Jan 03 03:41:34 dwst-cp-1 rancher-system-agent[12207]: time="2024-01-03T03:41:34+01:00" level=info msg="Starting remote watch of plans" Jan 03 03:41:35 dwst-cp-1 rancher-system-agent[12207]: E0103 03:41:35.164424 12207 memcache.go:206] couldn't get resource list for management.cattle.io/v3: Jan 03 03:41:35 dwst-cp-1 rancher-system-agent[12207]: time="2024-01-03T03:41:35+01:00" level=info msg="Starting /v1, Kind=Secret controller"

**Expected behavior:**
It should install correctly.

**Actual behavior:**
Fails to deploy the rke2-server process

**Additional context / logs:**

[brandond]

I don't believe this is an RKE2 issue. If the rancher system agent doesn't attempt to install and start rke2, then the issue is on the Rancher side. Check the logs over there.

[ObieBent]

Thanks for your reply. After perusing the entire server system logging and performing some strace, I conclude that the rancher system agent doesn't invoke rke2. I have submitted an issue for this on Rancher Github Issues page. I'll let you know in case of any updates.

[caroline-suse-rancher]

I'm converting this to a discussion to keep open for updates