IPAMD fails to start after reboot

[joejulian]

What happened: When I reboot a worker node the aws-cni pod for that node fails to connect:

Defaulted container "aws-node" out of: aws-node, aws-vpc-cni-init (init)
{"level":"info","ts":"2023-05-12T20:37:22.188Z","caller":"entrypoint.sh","msg":"Validating env variables ..."}
{"level":"info","ts":"2023-05-12T20:37:22.189Z","caller":"entrypoint.sh","msg":"Install CNI binaries.."}
{"level":"info","ts":"2023-05-12T20:37:22.208Z","caller":"entrypoint.sh","msg":"Starting IPAM daemon in the background ... "}
{"level":"info","ts":"2023-05-12T20:37:22.209Z","caller":"entrypoint.sh","msg":"Checking for IPAM connectivity ... "}
{"level":"info","ts":"2023-05-12T20:37:24.215Z","caller":"entrypoint.sh","msg":"Retrying waiting for IPAM-D"}
[47 duplicate lines deleted]
{"level":"info","ts":"2023-05-12T20:39:00.559Z","caller":"entrypoint.sh","msg":"Retrying waiting for IPAM-D"}

Attach logs

What you expected to happen: I expected the cni to start and function correctly

How to reproduce it (as minimally and precisely as possible):

1. Spin up an EKS 1.23 cluster using the v1.10.4 cni
2. Attach workers using the ubuntu-eks/k8s_1.23/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20230430 ami
   • the workers all come up normally, the cni is working
3. reboot a worker

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):

  Server Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.17-eks-0a21954", GitCommit:"cd5c12c51b0899612375453f7a7c2e7b6563f5e9", GitTreeState:"clean", BuildDate:"2023-04-15T00:32:27Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}

• CNI Version v1.10.4 - OS (e.g: cat /etc/os-release):

  NAME="Ubuntu"
  VERSION="20.04.6 LTS (Focal Fossa)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 20.04.6 LTS"
  VERSION_ID="20.04"
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  VERSION_CODENAME=focal
  UBUNTU_CODENAME=focal

• Kernel (e.g. uname -a):

  Linux ip-10-0-1-55 5.15.0-1035-aws #39~20.04.1-Ubuntu SMP Wed Apr 19 15:34:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

[jdn5126]

@joejulian this indicates an issue in the IPAM daemon. What do the IPAM logs on the node show: /var/log/aws-routed-eni/ipamd.log? And have you tried the latest release, v1.12.6 against this?

[joejulian]

I emailed the debug bundle. I have not tried a different cni version.

[jdn5126]

Responded via email

[joejulian]

Upgrading to v1.12.6 fixed this.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.