Closed brandond closed 1 week ago
helm-controller version was not bumped as intended when I updated the k3s version, so RKE2 is still on https://github.com/k3s-io/helm-controller/releases/tag/v0.16.1.
Moving back to working, will open a new PR to fix.
Infrastructure
Node(s) CPU architecture, OS, and Version:
Linux ip-172-31-9-207 5.15.0-1019-aws #23-Ubuntu SMP Wed Aug 17 18:33:13 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
Cluster Configuration:
1 server (config doesn't matter)
Config.yaml:
N/A
Additional files
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
name: apache
namespace: kube-system
spec:
repo: https://charts.bitnami.com/bitnami
chart: apache
targetNamespace: web
createNamespace: true
helmVersion: v2
$ sudo mkdir -p /etc/rancher/rke2 && sudo cp config.yaml /etc/rancher/rke2
kubectl describe
for the podReplication Results:
/usr/local/bin/rke2 -v
rke2 version v1.30.3+rke2r1 (e742dc53b463d205773cfb25633309671ce6777c)
go version go1.22.5 X:boringcrypto
kubectl get nodes,pods -A -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
node/ip-172-31-12-30.us-east-2.compute.internal Ready control-plane,etcd,master 57m v1.30.3+rke2r1 172.31.12.30 [REDACTED] Ubuntu 22.04.1 LTS 5.15.0-1019-aws containerd://1.7.17-k3s1
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system pod/cloud-controller-manager-ip-172-31-12-30.us-east-2.compute.internal 1/1 Running 0 57m 172.31.12.30 ip-172-31-12-30.us-east-2.compute.internal <none> <none>
kube-system pod/etcd-ip-172-31-12-30.us-east-2.compute.internal 1/1 Running 0 57m 172.31.12.30 ip-172-31-12-30.us-east-2.compute.internal <none> <none>
kube-system pod/helm-install-apache-pw88l 0/1 CrashLoopBackOff 9 (75s ago) 26m 10.42.0.23 ip-172-31-12-30.us-east-2.compute.internal <none> <none>
kube-system pod/helm-install-rke2-canal-c8f6j 0/1 Completed 0 57m 172.31.12.30 ip-172-31-12-30.us-east-2.compute.internal <none> <none>
kube-system pod/helm-install-rke2-coredns-wm7mf 0/1 Completed 0 57m 172.31.12.30 ip-172-31-12-30.us-east-2.compute.internal <none> <none>
kube-system pod/helm-install-rke2-ingress-nginx-bwc59 0/1 Completed 0 57m 10.42.0.3 ip-172-31-12-30.us-east-2.compute.internal <none> <none>
kube-system pod/helm-install-rke2-metrics-server-bwlx6 0/1 Completed 0 57m 10.42.0.8 ip-172-31-12-30.us-east-2.compute.internal <none> <none>
kube-system pod/helm-install-rke2-snapshot-controller-crd-f9l5h 0/1 Completed 0 57m 10.42.0.2 ip-172-31-12-30.us-east-2.compute.internal <none> <none>
kube-system pod/helm-install-rke2-snapshot-controller-g7jrg 0/1 Completed 0 57m 10.42.0.7 ip-172-31-12-30.us-east-2.compute.internal <none> <none>
kube-system pod/helm-install-rke2-snapshot-validation-webhook-wvgzl 0/1 Completed 0 57m 10.42.0.5 ip-172-31-12-30.us-east-2.compute.internal <none> <none>
kube-system pod/kube-apiserver-ip-172-31-12-30.us-east-2.compute.internal 1/1 Running 0 56m 172.31.12.30 ip-172-31-12-30.us-east-2.compute.internal <none> <none>
kube-system pod/kube-controller-manager-ip-172-31-12-30.us-east-2.compute.internal 1/1 Running 0 57m 172.31.12.30 ip-172-31-12-30.us-east-2.compute.internal <none> <none>
kube-system pod/kube-proxy-ip-172-31-12-30.us-east-2.compute.internal 1/1 Running 0 56m 172.31.12.30 ip-172-31-12-30.us-east-2.compute.internal <none> <none>
kube-system pod/kube-scheduler-ip-172-31-12-30.us-east-2.compute.internal 1/1 Running 0 57m 172.31.12.30 ip-172-31-12-30.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-canal-hkgnp 2/2 Running 0 56m 172.31.12.30 ip-172-31-12-30.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-coredns-rke2-coredns-64dcf4f58b-zn7hb 1/1 Running 0 57m 10.42.0.4 ip-172-31-12-30.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-coredns-rke2-coredns-autoscaler-5749cd7b8b-bx7wk 1/1 Running 0 57m 10.42.0.6 ip-172-31-12-30.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-ingress-nginx-controller-wkqdq 1/1 Running 0 55m 10.42.0.13 ip-172-31-12-30.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-metrics-server-868fc8795f-ttpnc 1/1 Running 0 55m 10.42.0.10 ip-172-31-12-30.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-snapshot-controller-7dcf5d5b46-wfz87 1/1 Running 0 56m 10.42.0.9 ip-172-31-12-30.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-snapshot-validation-webhook-bf7bbd6fc-m58zs 1/1 Running 0 55m 10.42.0.11 ip-172-31-12-30.us-east-2.compute.internal <none> <none>
kubectl describe pod/helm-install-apache-pw88l -n kube-system
Name: helm-install-apache-pw88l
Namespace: kube-system
Priority: 0
Service Account: helm-apache
Node: ip-172-31-12-30.us-east-2.compute.internal/172.31.12.30
Start Time: Thu, 22 Aug 2024 05:35:04 +0000
Labels: batch.kubernetes.io/controller-uid=bfa039c8-4073-4041-bded-00cc49b60309
batch.kubernetes.io/job-name=helm-install-apache
controller-uid=bfa039c8-4073-4041-bded-00cc49b60309
helmcharts.helm.cattle.io/chart=apache
job-name=helm-install-apache
Annotations: cni.projectcalico.org/containerID: e4743e3170fcf5de1ab9e65dc90f1273e4671ef2c1ba8564d06efaf395186236
cni.projectcalico.org/podIP: 10.42.0.23/32
cni.projectcalico.org/podIPs: 10.42.0.23/32
helmcharts.helm.cattle.io/configHash: SHA256=E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
Status: Running
SeccompProfile: RuntimeDefault
IP: 10.42.0.23
IPs:
IP: 10.42.0.23
Controlled By: Job/helm-install-apache
Containers:
helm:
Container ID: containerd://11dd242c4b522514433bab732d294e2b79e96e625b695e02851d5cb8b2d1f23a
Image: rancher/klipper-helm:v0.8.4-build20240523
Image ID: docker.io/rancher/klipper-helm@sha256:c2fd922a9a361ac5ec7ef225a46aaaad1e79ec3acc3cf176f60cd09a11683dd5
Port: <none>
Host Port: <none>
Args:
install
--namespace
web
--create-namespace
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Message: Installing helm_v2 chart
Exit Code: 1
Started: Thu, 22 Aug 2024 06:00:12 +0000
Finished: Thu, 22 Aug 2024 06:00:38 +0000
Ready: False
Restart Count: 9
Environment:
NAME: apache
VERSION:
REPO: https://charts.bitnami.com/bitnami
HELM_DRIVER: secret
CHART_NAMESPACE: kube-system
CHART: apache/apache
HELM_VERSION: v2
TARGET_NAMESPACE: web
AUTH_PASS_CREDENTIALS: false
NO_PROXY: .svc,.cluster.local,10.42.0.0/16,10.43.0.0/16
FAILURE_POLICY: reinstall
Mounts:
/chart from content (rw)
/config from values (rw)
/home/klipper-helm/.cache from klipper-cache (rw)
/home/klipper-helm/.config from klipper-config (rw)
/home/klipper-helm/.helm from klipper-helm (rw)
/tmp from tmp (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bf8zk (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
klipper-helm:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
klipper-cache:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
klipper-config:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
values:
Type: Secret (a volume populated by a Secret)
SecretName: chart-values-apache
Optional: false
content:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: chart-content-apache
Optional: false
kube-api-access-bf8zk:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 25m default-scheduler Successfully assigned kube-system/helm-install-apache-pw88l to ip-172-31-12-30.us-east-2.compute.internal
Normal Created 22m (x5 over 25m) kubelet Created container helm
Normal Started 22m (x5 over 25m) kubelet Started container helm
Warning BackOff 5m42s (x75 over 24m) kubelet Back-off restarting failed container helm in pod helm-install-apache-pw88l_kube-system(85d46dec-bb4c-4bfb-a1b2-783b27a979eb)
Normal Pulled 47s (x10 over 25m) kubelet Container image "rancher/klipper-helm:v0.8.4-build20240523" already present on machine
kubectl describe helmchart -n kube-system apache
Name: apache
Namespace: kube-system
Labels: <none>
Annotations: helmcharts.cattle.io/managed-by: helm-controller
API Version: helm.cattle.io/v1
Kind: HelmChart
Metadata:
Creation Timestamp: 2024-08-22T05:35:04Z
Finalizers:
wrangler.cattle.io/on-helm-chart-remove
Generation: 3
Resource Version: 7087
UID: a1da525f-52e1-484d-9b3f-6fdf77cfaf19
Spec:
Chart: apache
Create Namespace: true
Helm Version: v2
Repo: https://charts.bitnami.com/bitnami
Target Namespace: web
Status:
Job Name: helm-install-apache
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ApplyJob 3m4s (x24 over 28m) helm-controller Applying HelmChart using Job kube-system/helm-install-apache
Validation Results:
rke2 -v
rke2 version v1.31.0-rc1+rke2r1 (3acd10e9389fa930860896ca6ed925a15dff61a3)
go version go1.22.5 X:boringcrypto
kubectl get nodes,pods -A -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
node/ip-172-31-8-103 Ready control-plane,etcd,master 30m v1.31.0+rke2r1 172.31.8.103 <none> Ubuntu 24.04 LTS 6.8.0-1012-aws containerd://1.7.20-k3s1
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system pod/cloud-controller-manager-ip-172-31-2-129.us-east-2.compute.internal 1/1 Running 3 (145m ago) 157m 172.31.2.129 ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kube-system pod/cloud-controller-manager-ip-172-31-4-186.us-east-2.compute.internal 1/1 Running 1 (145m ago) 152m 172.31.4.186 ip-172-31-4-186.us-east-2.compute.internal <none> <none>
kube-system pod/cloud-controller-manager-ip-172-31-7-240.us-east-2.compute.internal 1/1 Running 1 (145m ago) 144m 172.31.7.240 ip-172-31-7-240.us-east-2.compute.internal <none> <none>
kube-system pod/cloud-controller-manager-ip-172-31-8-103 1/1 Running 0 30m 172.31.8.103 ip-172-31-8-103 <none> <none>
kube-system pod/etcd-ip-172-31-2-129.us-east-2.compute.internal 1/1 Running 0 145m 172.31.2.129 ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kube-system pod/etcd-ip-172-31-4-186.us-east-2.compute.internal 1/1 Running 0 144m 172.31.4.186 ip-172-31-4-186.us-east-2.compute.internal <none> <none>
kube-system pod/etcd-ip-172-31-7-240.us-east-2.compute.internal 1/1 Running 0 144m 172.31.7.240 ip-172-31-7-240.us-east-2.compute.internal <none> <none>
kube-system pod/etcd-ip-172-31-8-103 1/1 Running 0 30m 172.31.8.103 ip-172-31-8-103 <none> <none>
kube-system pod/helm-delete-rke2-multus-vwgkj 0/1 Pending 0 29m <none> ip-172-31-4-186.us-east-2.compute.internal <none> <none>
kube-system pod/helm-install-rke2-canal-6djwf 0/1 Completed 0 157m 172.31.2.129 ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kube-system pod/helm-install-rke2-coredns-nc7s4 0/1 Completed 0 157m 172.31.2.129 ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kube-system pod/helm-install-rke2-ingress-nginx-dlldt 0/1 Completed 0 157m <none> ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kube-system pod/helm-install-rke2-metrics-server-v6b4n 0/1 Completed 0 157m <none> ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kube-system pod/helm-install-rke2-multus-xlzff 0/1 Completed 0 157m 172.31.2.129 ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kube-system pod/helm-install-rke2-snapshot-controller-crd-697l2 0/1 Completed 0 157m <none> ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kube-system pod/helm-install-rke2-snapshot-controller-t46ml 0/1 Completed 1 157m <none> ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kube-system pod/helm-install-rke2-snapshot-validation-webhook-qhvlb 0/1 Completed 0 157m <none> ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kube-system pod/kube-apiserver-ip-172-31-2-129.us-east-2.compute.internal 1/1 Running 0 145m 172.31.2.129 ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kube-system pod/kube-apiserver-ip-172-31-4-186.us-east-2.compute.internal 1/1 Running 0 144m 172.31.4.186 ip-172-31-4-186.us-east-2.compute.internal <none> <none>
kube-system pod/kube-apiserver-ip-172-31-7-240.us-east-2.compute.internal 1/1 Running 0 144m 172.31.7.240 ip-172-31-7-240.us-east-2.compute.internal <none> <none>
kube-system pod/kube-apiserver-ip-172-31-8-103 1/1 Running 0 30m 172.31.8.103 ip-172-31-8-103 <none> <none>
kube-system pod/kube-controller-manager-ip-172-31-2-129.us-east-2.compute.internal 1/1 Running 0 144m 172.31.2.129 ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kube-system pod/kube-controller-manager-ip-172-31-4-186.us-east-2.compute.internal 1/1 Running 0 144m 172.31.4.186 ip-172-31-4-186.us-east-2.compute.internal <none> <none>
kube-system pod/kube-controller-manager-ip-172-31-7-240.us-east-2.compute.internal 1/1 Running 0 144m 172.31.7.240 ip-172-31-7-240.us-east-2.compute.internal <none> <none>
kube-system pod/kube-controller-manager-ip-172-31-8-103 1/1 Running 0 30m 172.31.8.103 ip-172-31-8-103 <none> <none>
kube-system pod/kube-proxy-ip-172-31-13-11.us-east-2.compute.internal 1/1 Running 0 150m 172.31.13.11 ip-172-31-13-11.us-east-2.compute.internal <none> <none>
kube-system pod/kube-proxy-ip-172-31-2-129.us-east-2.compute.internal 1/1 Running 1 (147m ago) 157m 172.31.2.129 ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kube-system pod/kube-proxy-ip-172-31-4-186.us-east-2.compute.internal 1/1 Running 1 (145m ago) 152m 172.31.4.186 ip-172-31-4-186.us-east-2.compute.internal <none> <none>
kube-system pod/kube-proxy-ip-172-31-7-240.us-east-2.compute.internal 1/1 Running 1 (145m ago) 153m 172.31.7.240 ip-172-31-7-240.us-east-2.compute.internal <none> <none>
kube-system pod/kube-proxy-ip-172-31-8-103 1/1 Running 0 30m 172.31.8.103 ip-172-31-8-103 <none> <none>
kube-system pod/kube-scheduler-ip-172-31-2-129.us-east-2.compute.internal 1/1 Running 0 144m 172.31.2.129 ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kube-system pod/kube-scheduler-ip-172-31-4-186.us-east-2.compute.internal 1/1 Running 0 144m 172.31.4.186 ip-172-31-4-186.us-east-2.compute.internal <none> <none>
kube-system pod/kube-scheduler-ip-172-31-7-240.us-east-2.compute.internal 1/1 Running 0 144m 172.31.7.240 ip-172-31-7-240.us-east-2.compute.internal <none> <none>
kube-system pod/kube-scheduler-ip-172-31-8-103 1/1 Running 0 30m 172.31.8.103 ip-172-31-8-103 <none> <none>
kube-system pod/rke2-canal-4dhjl 2/2 Running 2 (145m ago) 152m 172.31.4.186 ip-172-31-4-186.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-canal-bnvd5 2/2 Running 2 (145m ago) 154m 172.31.7.240 ip-172-31-7-240.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-canal-n74bt 2/2 Running 0 150m 172.31.13.11 ip-172-31-13-11.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-canal-r4zzs 2/2 Running 0 157m 172.31.2.129 ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-canal-vrqns 2/2 Running 0 29m 172.31.8.103 ip-172-31-8-103 <none> <none>
kube-system pod/rke2-coredns-rke2-coredns-787bc4b7b7-7t6q5 1/1 Running 1 (145m ago) 153m 10.42.1.3 ip-172-31-7-240.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-coredns-rke2-coredns-787bc4b7b7-thk9v 1/1 Running 0 157m 10.42.0.2 ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-coredns-rke2-coredns-autoscaler-6dc69d7b97-vdss7 1/1 Running 0 157m 10.42.0.7 ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-ingress-nginx-controller-lf7d5 1/1 Running 1 (145m ago) 152m 10.42.2.4 ip-172-31-4-186.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-ingress-nginx-controller-nn792 1/1 Running 0 150m 10.42.3.2 ip-172-31-13-11.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-ingress-nginx-controller-qctlx 1/1 Running 0 156m 10.42.0.11 ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-ingress-nginx-controller-s2mdx 1/1 Running 0 29m 10.42.4.2 ip-172-31-8-103 <none> <none>
kube-system pod/rke2-ingress-nginx-controller-w869h 1/1 Running 1 (145m ago) 153m 10.42.1.2 ip-172-31-7-240.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-metrics-server-6d99b6d454-ln4f6 1/1 Running 0 156m 10.42.0.10 ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-multus-5f46j 1/1 Running 2 (150m ago) 150m 172.31.13.11 ip-172-31-13-11.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-multus-c9vbf 1/1 Running 2 (157m ago) 157m 172.31.2.129 ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-multus-l2jpm 1/1 Running 3 (145m ago) 154m 172.31.7.240 ip-172-31-7-240.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-multus-qgrn4 1/1 Running 1 (145m ago) 152m 172.31.4.186 ip-172-31-4-186.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-multus-tgpv9 1/1 Running 2 (29m ago) 29m 172.31.8.103 ip-172-31-8-103 <none> <none>
kube-system pod/rke2-snapshot-controller-658d97fccc-p8xrg 1/1 Running 1 (147m ago) 156m 10.42.0.14 ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kube-system pod/rke2-snapshot-validation-webhook-784bcc6c8-bg5wz 1/1 Running 0 156m 10.42.0.13 ip-172-31-2-129.us-east-2.compute.internal <none> <none>
kubectl describe helmchart -n kube-system apache
Name: apache
Namespace: kube-system
Labels: <none>
Annotations: helmcharts.cattle.io/managed-by: helm-controller
API Version: helm.cattle.io/v1
Kind: HelmChart
Metadata:
Creation Timestamp: 2024-09-03T19:50:32Z
Finalizers:
wrangler.cattle.io/on-helm-chart-remove
Generation: 1
Resource Version: 32963
UID: 59e8690f-88d0-4bea-a026-ad916aa07d42
Spec:
Chart: apache
Create Namespace: true
Helm Version: v2
Repo: https://charts.bitnami.com/bitnami
Target Namespace: web
Status:
Conditions:
Status: False
Type: JobCreated
Message: Only Helm v3 charts are supported
Reason: Unsupported version
Status: True
Type: Failed
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning UnsupportedVersion 2m15s (x2 over 2m15s) helm-controller Unsupported Helm version v2: only v3 charts are supported
Additional context / logs:
N/A
RKE2 tracking issue for: