Closed dtledev closed 4 months ago
What does the user data log look like? /var/log/cloud-init-output.log
Looks like it ran successfully, I can confirm that it is able to download the .crt file and run the update-ca-trust
.
I can see the certificates loaded in the file /etc/ssl/certs/ca-bundle.crt
Which is technically a symlink to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
Do you restart containerd
after you update-ca-trust
?
restarting containerd after update-ca-trust
seems to resolve it, thank you!
This makes sense, perhaps something in the order or containerd between those releases.
What happened:
Images from private hosted repo with HTTPS domain no longer able to be pulled after upgrading nodes to latest versions
In order for EKS nodes to trust URLs from a private corporate domain we load a ca-certificate is loaded onto the node via userdata. At startup the node downloads the certificate
our-cert.crt
to/etc/pki/ca-trust/source/anchors
directory and executesupdate-ca-trust
command. This stops working on release v20240202 and onwardsSimilar pattern to this blog post: https://aws.amazon.com/blogs/containers/use-private-certificates-to-enable-a-container-repository-in-amazon-eks/
One of the destinations is a private hosted Quay container repo with an HTTPS URL which will be referenced as
https://quay.ourprivatedomain.com
.Sample error:
What you expected to happen: Expect the kubelet to respect the trusted CAs that's on the node.
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?: Troubleshooted by logging into the node and executing commands via systems manager
ctr image pull xxxxxx
and successfully pull an image to the node from CLI (via SSM) without passing--skip-verify
flagEnvironment:
AWS Region: ca-central-1
Instance Type(s): t3a.xlarge
EKS Platform version (use
aws eks describe-cluster --name <name> --query cluster.platformVersion
): "eks.6"Kubernetes version (use
aws eks describe-cluster --name <name> --query cluster.version
): "1.29"AMI Version: v20240202 and later
Kernel (e.g.
uname -a
):Linux ip-10-128-95-56.ca-central-1.compute.internal 5.10.205-195.807.amzn2.x86_64 #1 SMP Tue Jan 16 18:28:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Release information (run
cat /etc/eks/release
on a node):