Open mmshin opened 1 month ago
Is there any other way to do this without splitting the EKS cluster creation and managed node group just to insert Cilium installation in the middle?
No - you would have to break this up into steps. Roughly:
You might be able to get away with:
So we're currently deploying Cilium using helm_release and wait is set to false. I ended up testing it with time_sleep
In my terraform apply, both Cilium and managed_node_groups were created simultaneously. But because Cilium was faster to deploy, nodes were ready just in time.
Just to confirm, it that the same as the 2nd option you mentioned in your previous comment?
This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days
@mmshin were you able to succeed? I am stuck on same issue, any info or better terraform you can share please?
Hi, yes. So there were 2 options that I tried.
Add time_sleep similar to this before the cilium installation
But then I realized our helm provider definition has this dependency host = module.eks.cluster_endpoint
, meaning that time_sleep i added is not needed. And it worked. As long as the helm_release.cilium doesn't wait for the whole module to finish.
So in general, cilium just needed to make sure that the cluster is ready. Then, both cilium and nodes will be created at the same time. Our cilium installation has wait set to false so from TF's perspective, it was done in 9s. So by time that the nodes are done creating, Cilium is already up and the nodes will also be Ready just in time.
I hope my explanation makes sense.
Thanks for explanation @mmshin , I was actually able to make it work without time_sleep by following above but I am facing another issue with cilium, I am able to nslookup
any domain from the pod but can't ping
or curl
any external domain, internal works fine, all my services are up and running and coredns log
show it's fine, I dont have that issue with kube-proxy
so definitely something in cilium,
do you have an idea about it or if you could please share your cilium values?
this is what I used:
kubeProxyReplacement: "true"
ipam:
mode: "eni"
egressMasqueradeInterfaces: "eth0"
eni:
enabled: true
k8sServiceHost:"http://xxxxx"
k8sServicePort: 443
tunnel: disabled
endpointRoutes:
enabled: true
hubble:
metrics:
enabled:
- dns
- drop
- tcp
- flow
- icmp
- http
relay:
enabled: true
ui:
enabled: true
envoy:
enabled: true
gatewayAPI:
enabled: true
routingMode: "native"
Hi,
So we're trying to implement the new supported argument to disable default networking add-ons. Before this, we're doing lots of local-exec to uninstall and cleanup vpc cni, kube-proxy and core-dns because we're using alternative add-ons.
Current testing: module definition is creating a cluster without any default networking add-ons+ deploying managed node group. Terraform is stuck in applying because the nodes are in Not Ready state and we're assuming because of the missing CNI add-on. In our current flow, we have a
depends_on
to wait for eks module to finish before installing Cilium.Is there any other way to do this without splitting the EKS cluster creation and managed node group just to insert Cilium installation in the middle?
Second solution we're thinking is to add the time_sleep.trigger you did with the default add_ons. So basically, Cilium doesn't depend on the whole module but just implement a time_sleep, just to have ample amount of time before the managed node groups get created.