search

pulumi / pulumi-eks

A Pulumi component for easily creating and managing an Amazon EKS Cluster
https://www.pulumi.com/registry/packages/eks/
Apache License 2.0
171 stars 81 forks source link

Core dns is unstable #1365

Closed avinassh closed 1 month ago

avinassh commented 1 month ago

What happened?

I am new to Pulumi, and infra things. So I setup a EKS cluster following the documentation and examples. However, I am finding DNS and networking within the pods to be broken. My app is simple, at the start it calls an external API and fatals if that call fails. 

func main() {
    if err := sync(); err != nil {
        log.Fatalf("Could not update: %v", err)
    }
        // more stuff
}

What I noticed was, out of 4 pods only two were crashing. The other two were able to make this call and start the pod: 

kubectl get pods -n internal
NAME                             READY   STATUS             RESTARTS        AGE
sync-app-56754c787b-9c452   0/1     CrashLoopBackOff   9 (4m56s ago)   31m
sync-app-56754c787b-kvrhw   0/1     CrashLoopBackOff   9 (4m56s ago)   31m
sync-app-56754c787b-n2tfg   1/1     Running            0               31m
sync-app-56754c787b-tmmv8   1/1     Running            0               31m

So this ruled out IAM policy or permissions issue. This also confirms that NAT/IGN have been setup. I ran a test nicolaka/netshoot image and checked DNS configurations:

test-pod:~# cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local ec2.internal
nameserver 172.20.0.10
options ndots:5

# this is IP of core dns 
test-pod:~# ping -c 5 172.20.0.10 
PING 172.20.0.10 (172.20.0.10) 56(84) bytes of data.

--- 172.20.0.10 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4081ms

#  IP of one of the core dns pod
test-pod:~# ping -c 5 10.0.125.73
PING 10.0.125.73 (10.0.125.73) 56(84) bytes of data.

--- 10.0.125.73 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4088ms

#  IP of another of the core dns pod
test-pod:~# ping -c 5 10.0.107.114
PING 10.0.107.114 (10.0.107.114) 56(84) bytes of data.

--- 10.0.107.114 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4092ms

I restarted core dns pods, this time I was able to ping to one of the pods IP (but service IP still failed). Upon restarting core dns, one of the sync-app pods were also able to connect to external API and became healthy.

Example

My code is in Go. Since I am new to Pulumi, I assumed I'd be doing some mistake. So I tested with https://github.com/pulumi/pulumi-eks/tree/204401a/examples/nodegroup example and noticed the same issue. Here is how you can reproduce:

  1. Copy the nodegroup example
  2. I updated index.ts to create a VPC and use that. The code is from https://github.com/pulumi/pulumi-eks/blob/204401a2c2473d41dbf96ec795c74e3f48acff65/examples/subnet-tags/index.ts 
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as eks from "@pulumi/eks";
import * as iam from "./iam";
import * as awsx from "@pulumi/awsx";
import { SubnetType } from "@pulumi/awsx/ec2";

const projectName = pulumi.getProject();
const tags = { "project": "subnets", "account": "pulumi"};

const vpc = new awsx.ec2.Vpc("vpc",
    {
        tags: {"Name": `${projectName}`, ...tags},
        // Tag subnets for specific load-balancer usage.
        // Any non-null tag value is valid.
        // See:
        //  - https://docs.aws.amazon.com/eks/latest/userguide/network_reqs.html
        subnetSpecs: [
            {type: SubnetType.Public, tags: {"kubernetes.io/role/elb": "1", ...tags}},
            {type: SubnetType.Private, tags: {"kubernetes.io/role/internal-elb": "1", ...tags}},
        ],
        subnetStrategy: "Auto", // Upstream awsx default is being switched to "Auto", explicitly set to "Auto" to disable warning messages.
    },
    {
        // Inform pulumi to ignore tag changes to the VPCs or subnets, so that
        // tags auto-added by AWS EKS do not get removed during future
        // refreshes and updates, as they are added outside of pulumi's management
        // and would be removed otherwise.
        // See: https://github.com/pulumi/pulumi-eks/issues/271#issuecomment-548452554
        transformations: [(args: any) => {
            if (args.type === "aws:ec2/vpc:Vpc" || args.type === "aws:ec2/subnet:Subnet") {
                return {
                    props: args.props,
                    opts: pulumi.mergeOptions(args.opts, { ignoreChanges: ["tags"] }),
                };
            }
            return undefined;
        }],
    },
);

/**
 * Identical IAM for all NodeGroups: all NodeGroups share the same `instanceRole`.
 */

// Create example IAM roles and profiles to show to use them with NodeGroups.
// Note, all roles for the instance profiles are required to at least have
// the following EKS Managed Policies attached to successfully auth and join the
// cluster:
//   - "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
//   - "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
//   - "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
const role0 = iam.createRole("example-role0");
const instanceProfile0 = new aws.iam.InstanceProfile("example-instanceProfile0", {role: role0});

// Create an EKS cluster with a shared IAM instance role to register with the
// cluster auth.
const cluster1 = new eks.Cluster("example-nodegroup-iam-simple", {
    skipDefaultNodeGroup: true,
    deployDashboard: false,
    // nodeAmiId: "ami-0384725f0d30527c7",
    instanceRole: role0,
    vpcId: vpc.vpcId,
    publicSubnetIds: vpc.publicSubnetIds,
    privateSubnetIds: vpc.privateSubnetIds,
});

// There are two approaches that can be used to add additional NodeGroups.
// 1. A `createNodeGroup` API on `eks.Cluster`
// 2. A `NodeGroup` resource which accepts an `eks.Cluster` as input

// Create the node group using an `instanceProfile` tied to the shared, cluster
// instance role registered with the cluster auth through `instanceRole`.
cluster1.createNodeGroup("example-ng-simple-ondemand", {
    instanceType: "t3.medium",
    desiredCapacity: 1,
    minSize: 1,
    maxSize: 2,
    // amiId: "ami-0384725f0d30527c7",
    labels: {"ondemand": "true"},
    instanceProfile: instanceProfile0,
});

const ng = new eks.NodeGroupV2("example-ng2-simple-ondemand", {
    cluster: cluster1,
    instanceType: "t3.medium",
    desiredCapacity: 1,
    minSize: 1,
    maxSize: 2,
    // amiId: "ami-0384725f0d30527c7",
    labels: {"ondemand": "true"},
    instanceProfile: instanceProfile0,
});

// Export the cluster's kubeconfig.
export const kubeconfig1 = cluster1.kubeconfig;
  1. Once cluster is setup, try the test image:
$ kubectl run test-pod --image=nicolaka/netshoot -- sleep infinity

$ kubectl exec -it test-pod -- /bin/bash

$ test-pod:~# nslookup google.com
;; communications error to 172.20.0.10#53: timed out
;; communications error to 172.20.0.10#53: timed out
;; communications error to 172.20.0.10#53: timed out
;; no servers could be reached

$

if above worked, then most likely it was able to connect to a healthy core dns pod. Creating 1-2 more pods, you can run into this issue. I will share my go version of the code in some time.

Output of pulumi about

CLI
Version      3.132.0
Go Version   go1.23.1
Go Compiler  gc

Plugins
KIND      NAME        VERSION
resource  aws         6.51.1
resource  awsx        2.14.0
resource  docker      4.5.5
resource  docker      3.6.1
resource  eks         2.7.9
resource  kubernetes  4.18.0
language  nodejs      3.132.0-dev.0

Host
OS       darwin
Version  14.5
Arch     x86_64

This project is written in nodejs: executable='/Users/avi/.local/state/fnm_multishells/49302_1726506123444/bin/node' version='v18.20.3'

Current Stack: organization/example-nodegroup/d

TYPE                                                 URN
pulumi:pulumi:Stack                                  urn:pulumi:d::example-nodegroup::pulumi:pulumi:Stack::example-nodegroup-d
eks:index:Cluster                                    urn:pulumi:d::example-nodegroup::eks:index:Cluster::example-nodegroup-iam-simple
eks:index:NodeGroupV2                                urn:pulumi:d::example-nodegroup::eks:index:NodeGroupV2::example-ng2-simple-ondemand
pulumi:providers:aws                                 urn:pulumi:d::example-nodegroup::pulumi:providers:aws::default_6_51_1
pulumi:providers:awsx                                urn:pulumi:d::example-nodegroup::pulumi:providers:awsx::default_2_14_0
eks:index:ServiceRole                                urn:pulumi:d::example-nodegroup::eks:index:Cluster$eks:index:ServiceRole::example-nodegroup-iam-simple-eksRole
awsx:ec2:Vpc                                         urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc::vpc
aws:iam/role:Role                                    urn:pulumi:d::example-nodegroup::aws:iam/role:Role::example-role0
aws:iam/role:Role                                    urn:pulumi:d::example-nodegroup::eks:index:Cluster$eks:index:ServiceRole$aws:iam/role:Role::example-nodegroup-iam-simple-eksRole-role
pulumi:providers:aws                                 urn:pulumi:d::example-nodegroup::pulumi:providers:aws::default_6_47_0
aws:iam/rolePolicyAttachment:RolePolicyAttachment    urn:pulumi:d::example-nodegroup::aws:iam/rolePolicyAttachment:RolePolicyAttachment::example-role0-policy-2
aws:iam/rolePolicyAttachment:RolePolicyAttachment    urn:pulumi:d::example-nodegroup::aws:iam/rolePolicyAttachment:RolePolicyAttachment::example-role0-policy-1
aws:iam/rolePolicyAttachment:RolePolicyAttachment    urn:pulumi:d::example-nodegroup::aws:iam/rolePolicyAttachment:RolePolicyAttachment::example-role0-policy-0
aws:iam/rolePolicyAttachment:RolePolicyAttachment    urn:pulumi:d::example-nodegroup::eks:index:Cluster$eks:index:ServiceRole$aws:iam/rolePolicyAttachment:RolePolicyAttachment::example-nodegroup-iam-simple-eksRole-4b490823
aws:iam/instanceProfile:InstanceProfile              urn:pulumi:d::example-nodegroup::aws:iam/instanceProfile:InstanceProfile::example-instanceProfile0
aws:ec2/vpc:Vpc                                      urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc::vpc
aws:ec2/internetGateway:InternetGateway              urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/internetGateway:InternetGateway::vpc
aws:ec2/subnet:Subnet                                urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::vpc-private-3
aws:ec2/subnet:Subnet                                urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::vpc-private-1
aws:ec2/subnet:Subnet                                urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::vpc-private-2
aws:ec2/routeTable:RouteTable                        urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::vpc-private-3
aws:ec2/routeTable:RouteTable                        urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::vpc-private-1
aws:ec2/routeTable:RouteTable                        urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::vpc-private-2
aws:ec2/routeTableAssociation:RouteTableAssociation  urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::vpc-private-3
aws:ec2/routeTableAssociation:RouteTableAssociation  urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::vpc-private-2
aws:ec2/routeTableAssociation:RouteTableAssociation  urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::vpc-private-1
aws:ec2/subnet:Subnet                                urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::vpc-public-2
aws:ec2/subnet:Subnet                                urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::vpc-public-3
aws:ec2/subnet:Subnet                                urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::vpc-public-1
aws:ec2/routeTable:RouteTable                        urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::vpc-public-2
aws:ec2/routeTable:RouteTable                        urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::vpc-public-3
aws:ec2/routeTable:RouteTable                        urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::vpc-public-1
aws:ec2/eip:Eip                                      urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/eip:Eip::vpc-1
aws:ec2/eip:Eip                                      urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/eip:Eip::vpc-3
aws:ec2/eip:Eip                                      urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/eip:Eip::vpc-2
aws:ec2/routeTableAssociation:RouteTableAssociation  urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::vpc-public-3
aws:ec2/routeTableAssociation:RouteTableAssociation  urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::vpc-public-2
aws:ec2/routeTableAssociation:RouteTableAssociation  urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::vpc-public-1
aws:ec2/route:Route                                  urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::vpc-public-2
aws:ec2/route:Route                                  urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::vpc-public-3
aws:ec2/route:Route                                  urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::vpc-public-1
aws:ec2/natGateway:NatGateway                        urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/natGateway:NatGateway::vpc-2
aws:ec2/natGateway:NatGateway                        urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/natGateway:NatGateway::vpc-3
aws:ec2/route:Route                                  urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::vpc-private-3
aws:ec2/route:Route                                  urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::vpc-private-2
aws:ec2/natGateway:NatGateway                        urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/natGateway:NatGateway::vpc-1
aws:ec2/route:Route                                  urn:pulumi:d::example-nodegroup::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::vpc-private-1
pulumi:providers:pulumi                              urn:pulumi:d::example-nodegroup::pulumi:providers:pulumi::default
aws:ec2/securityGroup:SecurityGroup                  urn:pulumi:d::example-nodegroup::eks:index:Cluster$aws:ec2/securityGroup:SecurityGroup::example-nodegroup-iam-simple-eksClusterSecurityGroup
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::example-nodegroup::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::example-nodegroup-iam-simple-eksClusterInternetEgressRule
aws:eks/cluster:Cluster                              urn:pulumi:d::example-nodegroup::eks:index:Cluster$aws:eks/cluster:Cluster::example-nodegroup-iam-simple-eksCluster
pulumi:providers:kubernetes                          urn:pulumi:d::example-nodegroup::eks:index:Cluster$pulumi:providers:kubernetes::example-nodegroup-iam-simple-eks-k8s
pulumi:providers:kubernetes                          urn:pulumi:d::example-nodegroup::eks:index:Cluster$pulumi:providers:kubernetes::example-nodegroup-iam-simple-provider
eks:index:NodeGroup                                  urn:pulumi:d::example-nodegroup::eks:index:Cluster$eks:index:NodeGroup::example-ng-simple-ondemand
aws:ec2/securityGroup:SecurityGroup                  urn:pulumi:d::example-nodegroup::eks:index:Cluster$aws:ec2/securityGroup:SecurityGroup::example-nodegroup-iam-simple-nodeSecurityGroup
aws:ec2/securityGroup:SecurityGroup                  urn:pulumi:d::example-nodegroup::eks:index:NodeGroupV2$aws:ec2/securityGroup:SecurityGroup::example-ng2-simple-ondemand-nodeSecurityGroup
kubernetes:core/v1:ConfigMap                         urn:pulumi:d::example-nodegroup::eks:index:Cluster$kubernetes:core/v1:ConfigMap::example-nodegroup-iam-simple-nodeAccess
pulumi:providers:eks                                 urn:pulumi:d::example-nodegroup::pulumi:providers:eks::default_2_7_9
eks:index:RandomSuffix                               urn:pulumi:d::example-nodegroup::eks:index:Cluster$eks:index:NodeGroup$eks:index:RandomSuffix::example-ng-simple-ondemand-cfnStackName
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::example-nodegroup::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::example-nodegroup-iam-simple-eksNodeClusterIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::example-nodegroup::eks:index:NodeGroupV2$aws:ec2/securityGroupRule:SecurityGroupRule::example-ng2-simple-ondemand-eksNodeIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::example-nodegroup::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::example-nodegroup-iam-simple-eksClusterIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::example-nodegroup::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::example-nodegroup-iam-simple-eksExtApiServerClusterIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::example-nodegroup::eks:index:NodeGroupV2$aws:ec2/securityGroupRule:SecurityGroupRule::example-ng2-simple-ondemand-eksNodeInternetEgressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::example-nodegroup::eks:index:NodeGroupV2$aws:ec2/securityGroupRule:SecurityGroupRule::example-ng2-simple-ondemand-eksClusterIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::example-nodegroup::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::example-nodegroup-iam-simple-eksNodeIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::example-nodegroup::eks:index:NodeGroupV2$aws:ec2/securityGroupRule:SecurityGroupRule::example-ng2-simple-ondemand-eksNodeClusterIngressRule
aws:ec2/launchTemplate:LaunchTemplate                urn:pulumi:d::example-nodegroup::eks:index:NodeGroupV2$aws:ec2/launchTemplate:LaunchTemplate::example-ng2-simple-ondemand-launchTemplate
aws:ec2/launchConfiguration:LaunchConfiguration      urn:pulumi:d::example-nodegroup::eks:index:Cluster$eks:index:NodeGroup$aws:ec2/launchConfiguration:LaunchConfiguration::example-ng-simple-ondemand-nodeLaunchConfiguration
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::example-nodegroup::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::example-nodegroup-iam-simple-eksNodeInternetEgressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::example-nodegroup::eks:index:NodeGroupV2$aws:ec2/securityGroupRule:SecurityGroupRule::example-ng2-simple-ondemand-eksExtApiServerClusterIngressRule
eks:index:VpcCni                                     urn:pulumi:d::example-nodegroup::eks:index:Cluster$eks:index:VpcCni::example-nodegroup-iam-simple-vpc-cni
aws:autoscaling/group:Group                          urn:pulumi:d::example-nodegroup::eks:index:NodeGroupV2$aws:autoscaling/group:Group::example-ng2-simple-ondemand
aws:cloudformation/stack:Stack                       urn:pulumi:d::example-nodegroup::eks:index:Cluster$eks:index:NodeGroup$aws:cloudformation/stack:Stack::example-ng-simple-ondemand-nodes

Found no pending operations associated with d

Backend
Name           v-MacBook-Pro.local
URL            file://~
User           avi
Organizations
Token type     personal

Dependencies:
NAME            VERSION
@types/node     22.5.4
typescript      4.9.5
@pulumi/aws     6.51.1
@pulumi/awsx    2.14.0
@pulumi/eks     2.7.9
@pulumi/pulumi  3.132.0

Pulumi locates its logs in /var/folders/3l/n25ms2z97wg1d6k4xthfx7j40000gn/T/ by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

avinassh commented 1 month ago

Here is my Go code: 

package main

import (
    "fmt"

    "github.com/pulumi/pulumi-aws/sdk/v6/go/aws"
    "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/iam"
    "github.com/pulumi/pulumi-awsx/sdk/v2/go/awsx/ec2"
    "github.com/pulumi/pulumi-eks/sdk/v2/go/eks"
    "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
    corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
    metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
    storagev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/storage/v1"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func ptrTo[T any](v T) *T {
    return &v
}

func newVPC(ctx *pulumi.Context, clusterName string) (*ec2.Vpc, error) {
    vpc, err := ec2.NewVpc(ctx, fmt.Sprintf("%s-vpc", clusterName), &ec2.VpcArgs{
        AvailabilityZoneNames: []string{"eu-west-1a", "eu-west-1b", "eu-west-1c"},
        Tags: pulumi.StringMap{
            "Name":                                 pulumi.String(fmt.Sprintf("%s-vpc", clusterName)),
            "kubernetes.io/cluster/" + clusterName: pulumi.String("shared"),
        },
        EnableDnsHostnames: pulumi.BoolPtr(true),
        EnableDnsSupport:   pulumi.BoolPtr(true),
        SubnetSpecs: []ec2.SubnetSpecArgs{
            {
                Type: ec2.SubnetTypePublic,
                Name: ptrTo("public"),
                Tags: pulumi.StringMap{
                    "Name":                                 pulumi.String(fmt.Sprintf("%s-public-subnet", clusterName)),
                    "kubernetes.io/role/elb":               pulumi.String("1"),
                    "kubernetes.io/cluster/" + clusterName: pulumi.String("shared"),
                },
            },
            {
                Type: ec2.SubnetTypePrivate,
                Name: ptrTo("private"),
                Tags: pulumi.StringMap{
                    "Name":                                 pulumi.String(fmt.Sprintf("%s-private-subnet", clusterName)),
                    "kubernetes.io/role/internal-elb":      pulumi.String("1"),
                    "kubernetes.io/cluster/" + clusterName: pulumi.String("shared"),
                },
            },
        },
        SubnetStrategy: ptrTo(ec2.SubnetAllocationStrategyAuto),
    })
    return vpc, err
}

func createClusterRole(ctx *pulumi.Context, name string) (*iam.Role, *iam.InstanceProfile, error) {
    role, err := iam.NewRole(ctx, fmt.Sprintf("%s-instance-role", name), &iam.RoleArgs{
        AssumeRolePolicy: pulumi.String(`{
                    "Version": "2012-10-17",
                    "Statement": [{
                        "Action": "sts:AssumeRole",
                        "Effect": "Allow",
                        "Principal": {
                            "Service": "ec2.amazonaws.com"
                        }
                    }]
                }`),
        ManagedPolicyArns: pulumi.ToStringArray([]string{
            "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
            "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
            "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
            "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",
        }),
    })
    if err != nil {
        return nil, nil, err
    }
    instanceProfile, err := iam.NewInstanceProfile(ctx, fmt.Sprintf("%s-instance-profile", name), &iam.InstanceProfileArgs{
        Role: role,
    })
    if err != nil {
        return nil, nil, err
    }
    return role, instanceProfile, nil
}

func createCluster(ctx *pulumi.Context, clusterName, imageAmiId string, vpc *ec2.Vpc, instanceRole iam.RoleInput, instanceProfile *iam.InstanceProfile) (*eks.Cluster, error) {
    cluster, err := eks.NewCluster(ctx, clusterName, &eks.ClusterArgs{
        Name:                         pulumi.String(clusterName),
        VpcId:                        vpc.VpcId,
        PublicSubnetIds:              vpc.PublicSubnetIds,
        PrivateSubnetIds:             vpc.PrivateSubnetIds,
        NodeAssociatePublicIpAddress: pulumi.BoolRef(false),
        SkipDefaultNodeGroup:         pulumi.BoolRef(true),
        CreateOidcProvider:           pulumi.Bool(true),
        InstanceRole:                 instanceRole,
        NodeAmiId:                    pulumi.String(imageAmiId),
    })
    if err != nil {
        return nil, err
    }

    cluster.EksCluster.Name().ApplyT(func(name string) error {
        _, err = eks.NewNodeGroupV2(ctx, "routers", &eks.NodeGroupV2Args{
            Cluster:         cluster,
            InstanceType:    pulumi.String("m5.large"),
            DesiredCapacity: pulumi.Int(1),
            MinSize:         pulumi.Int(1),
            MaxSize:         pulumi.Int(10),
            NodeSubnetIds: pulumi.StringArray{vpc.PrivateSubnetIds.ApplyT(func(ids []string) string {
                return ids[0]
            }).(pulumi.StringOutput)},
            InstanceProfile: instanceProfile,
            Labels: map[string]string{
                "cluster-name":   name,
                "nodegroup-name": "routers",
            },
        }, pulumi.DependsOn([]pulumi.Resource{cluster}))
        if err != nil {
            return err
        }

        _, err = eks.NewNodeGroupV2(ctx, "daemon", &eks.NodeGroupV2Args{
            Cluster:         cluster,
            InstanceType:    pulumi.String("c5d.large"),
            DesiredCapacity: pulumi.Int(1),
            MinSize:         pulumi.Int(1),
            MaxSize:         pulumi.Int(10),
            NodeSubnetIds: pulumi.StringArray{vpc.PrivateSubnetIds.ApplyT(func(ids []string) string {
                return ids[1]
            }).(pulumi.StringOutput)},
            InstanceProfile: instanceProfile,
            Labels: map[string]string{
                "cluster-name":   name,
                "nodegroup-name": "daemon",
            },
        }, pulumi.DependsOn([]pulumi.Resource{cluster}))
        if err != nil {
            return err
        }

        _, err = eks.NewNodeGroupV2(ctx, "internal", &eks.NodeGroupV2Args{
            Cluster:         cluster,
            InstanceType:    pulumi.String("m5.large"),
            DesiredCapacity: pulumi.Int(1),
            MinSize:         pulumi.Int(1),
            MaxSize:         pulumi.Int(10),
            NodeSubnetIds: pulumi.StringArray{vpc.PrivateSubnetIds.ApplyT(func(ids []string) string {
                return ids[2]
            }).(pulumi.StringOutput)},
            InstanceProfile: instanceProfile,
            Labels: map[string]string{
                "cluster-name":   name,
                "nodegroup-name": "internal",
            },
        }, pulumi.DependsOn([]pulumi.Resource{cluster}))
        if err != nil {
            return err
        }
        return nil
    })
    return cluster, nil
}

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        region := "eu-west-1"
        clusterName := "my-cluster-aws-" + region
        imageAmiId := "ami-0b429fd2c7bbec0d5"

        reg, err := aws.GetRegion(ctx, nil, nil)
        if err != nil {
            return err
        }
        if region != reg.Id {
            return fmt.Errorf("invalid region configured. expected %s, got %s", region, reg.Id)
        }

        vpc, err := newVPC(ctx, clusterName)
        if err != nil {
            return err
        }
        instanceRole, instanceProfile, err := createClusterRole(ctx, clusterName)
        if err != nil {
            return err
        }
        cluster, err := createCluster(ctx, clusterName, imageAmiId, vpc, instanceRole, instanceProfile)
        if err != nil {
            return err
        }

        ctx.Export("clusterName", cluster.Core.Cluster().Name())
        ctx.Export("kubeconfig", cluster.Kubeconfig)

        k, _ := cluster.GetKubeconfig(ctx, nil)
        k8sProvider, err := kubernetes.NewProvider(ctx, "k8s-provider", &kubernetes.ProviderArgs{
            Kubeconfig: k,
        })
        if err != nil {
            return err
        }
        _, err = corev1.NewNamespace(ctx, "daemon", &corev1.NamespaceArgs{
            Metadata: &metav1.ObjectMetaArgs{
                Name: pulumi.String("daemon"),
                Labels: pulumi.StringMap{
                    "name": pulumi.String("daemon"),
                },
            },
        }, pulumi.Provider(k8sProvider))
        if err != nil {
            return err
        }

        _, err = corev1.NewNamespace(ctx, "internal", &corev1.NamespaceArgs{
            Metadata: &metav1.ObjectMetaArgs{
                Name: pulumi.String("internal"),
                Labels: pulumi.StringMap{
                    "name": pulumi.String("internal"),
                },
            },
        }, pulumi.Provider(k8sProvider))
        if err != nil {
            return err
        }

        _, err = storagev1.NewStorageClass(ctx, "local-storage", &storagev1.StorageClassArgs{
            Metadata: &metav1.ObjectMetaArgs{
                Name: pulumi.String("local-storage"),
            },
            Provisioner:       pulumi.String("kubernetes.io/no-provisioner"),
            VolumeBindingMode: pulumi.String("WaitForFirstConsumer"),
        }, pulumi.Provider(k8sProvider))
        return err
    })
}

output of pulumi about

CLI
Version      3.132.0
Go Version   go1.23.1
Go Compiler  gc

Plugins
KIND      NAME        VERSION
resource  aws         6.51.1
resource  awsx        2.14.0
resource  docker      4.4.3
resource  eks         2.7.9
language  go          3.132.0
resource  kubernetes  4.18.1

Host
OS       darwin
Version  14.5
Arch     x86_64

This project is written in go: executable='/usr/local/bin/go' version='go version go1.22.3 darwin/amd64'

Current Stack: organization/pulu/d

TYPE                                                 URN
pulumi:providers:aws                                 urn:pulumi:d::pulu::pulumi:providers:aws::default_6_51_1
pulumi:pulumi:Stack                                  urn:pulumi:d::pulu::pulumi:pulumi:Stack::pulu-d
pulumi:providers:awsx                                urn:pulumi:d::pulu::pulumi:providers:awsx::default_2_14_0
awsx:ec2:Vpc                                         urn:pulumi:d::pulu::awsx:ec2:Vpc::my-cluster-aws-eu-west-1-vpc
aws:iam/role:Role                                    urn:pulumi:d::pulu::aws:iam/role:Role::my-cluster-aws-eu-west-1-instance-role
pulumi:providers:aws                                 urn:pulumi:d::pulu::pulumi:providers:aws::default_6_47_0
aws:iam/instanceProfile:InstanceProfile              urn:pulumi:d::pulu::aws:iam/instanceProfile:InstanceProfile::my-cluster-aws-eu-west-1-instance-profile
aws:ec2/vpc:Vpc                                      urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc::my-cluster-aws-eu-west-1-vpc
aws:ec2/internetGateway:InternetGateway              urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/internetGateway:InternetGateway::my-cluster-aws-eu-west-1-vpc
aws:ec2/subnet:Subnet                                urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::my-cluster-aws-eu-west-1-vpc-private-2
aws:ec2/subnet:Subnet                                urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::my-cluster-aws-eu-west-1-vpc-private-3
aws:ec2/subnet:Subnet                                urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::my-cluster-aws-eu-west-1-vpc-private-1
aws:ec2/routeTable:RouteTable                        urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::my-cluster-aws-eu-west-1-vpc-private-2
aws:ec2/routeTable:RouteTable                        urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::my-cluster-aws-eu-west-1-vpc-private-3
aws:ec2/routeTable:RouteTable                        urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::my-cluster-aws-eu-west-1-vpc-private-1
aws:ec2/routeTableAssociation:RouteTableAssociation  urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::my-cluster-aws-eu-west-1-vpc-private-2
aws:ec2/routeTableAssociation:RouteTableAssociation  urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::my-cluster-aws-eu-west-1-vpc-private-3
aws:ec2/routeTableAssociation:RouteTableAssociation  urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::my-cluster-aws-eu-west-1-vpc-private-1
aws:ec2/subnet:Subnet                                urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::my-cluster-aws-eu-west-1-vpc-public-1
aws:ec2/subnet:Subnet                                urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::my-cluster-aws-eu-west-1-vpc-public-2
aws:ec2/subnet:Subnet                                urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::my-cluster-aws-eu-west-1-vpc-public-3
aws:ec2/routeTable:RouteTable                        urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::my-cluster-aws-eu-west-1-vpc-public-1
aws:ec2/eip:Eip                                      urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/eip:Eip::my-cluster-aws-eu-west-1-vpc-1
aws:ec2/eip:Eip                                      urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/eip:Eip::my-cluster-aws-eu-west-1-vpc-2
aws:ec2/routeTable:RouteTable                        urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::my-cluster-aws-eu-west-1-vpc-public-3
aws:ec2/eip:Eip                                      urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/eip:Eip::my-cluster-aws-eu-west-1-vpc-3
aws:ec2/routeTable:RouteTable                        urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::my-cluster-aws-eu-west-1-vpc-public-2
aws:ec2/routeTableAssociation:RouteTableAssociation  urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::my-cluster-aws-eu-west-1-vpc-public-1
aws:ec2/routeTableAssociation:RouteTableAssociation  urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::my-cluster-aws-eu-west-1-vpc-public-2
aws:ec2/routeTableAssociation:RouteTableAssociation  urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::my-cluster-aws-eu-west-1-vpc-public-3
aws:ec2/route:Route                                  urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::my-cluster-aws-eu-west-1-vpc-public-1
aws:ec2/route:Route                                  urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::my-cluster-aws-eu-west-1-vpc-public-3
aws:ec2/route:Route                                  urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::my-cluster-aws-eu-west-1-vpc-public-2
aws:ec2/natGateway:NatGateway                        urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/natGateway:NatGateway::my-cluster-aws-eu-west-1-vpc-1
aws:ec2/route:Route                                  urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::my-cluster-aws-eu-west-1-vpc-private-1
aws:ec2/natGateway:NatGateway                        urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/natGateway:NatGateway::my-cluster-aws-eu-west-1-vpc-3
aws:ec2/route:Route                                  urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::my-cluster-aws-eu-west-1-vpc-private-3
aws:ec2/natGateway:NatGateway                        urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/natGateway:NatGateway::my-cluster-aws-eu-west-1-vpc-2
aws:ec2/route:Route                                  urn:pulumi:d::pulu::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::my-cluster-aws-eu-west-1-vpc-private-2
pulumi:providers:pulumi                              urn:pulumi:d::pulu::pulumi:providers:pulumi::default
pulumi:providers:eks                                 urn:pulumi:d::pulu::pulumi:providers:eks::default_2_7_9
eks:index:Cluster                                    urn:pulumi:d::pulu::eks:index:Cluster::my-cluster-aws-eu-west-1
eks:index:ServiceRole                                urn:pulumi:d::pulu::eks:index:Cluster$eks:index:ServiceRole::my-cluster-aws-eu-west-1-eksRole
pulumi:providers:aws                                 urn:pulumi:d::pulu::pulumi:providers:aws::default_6_45_0
aws:iam/role:Role                                    urn:pulumi:d::pulu::eks:index:Cluster$eks:index:ServiceRole$aws:iam/role:Role::my-cluster-aws-eu-west-1-eksRole-role
aws:iam/rolePolicyAttachment:RolePolicyAttachment    urn:pulumi:d::pulu::eks:index:Cluster$eks:index:ServiceRole$aws:iam/rolePolicyAttachment:RolePolicyAttachment::my-cluster-aws-eu-west-1-eksRole-4b490823
aws:ec2/securityGroup:SecurityGroup                  urn:pulumi:d::pulu::eks:index:Cluster$aws:ec2/securityGroup:SecurityGroup::my-cluster-aws-eu-west-1-eksClusterSecurityGroup
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::pulu::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::my-cluster-aws-eu-west-1-eksClusterInternetEgressRule
aws:eks/cluster:Cluster                              urn:pulumi:d::pulu::eks:index:Cluster$aws:eks/cluster:Cluster::my-cluster-aws-eu-west-1-eksCluster
pulumi:providers:kubernetes                          urn:pulumi:d::pulu::eks:index:Cluster$pulumi:providers:kubernetes::my-cluster-aws-eu-west-1-provider
pulumi:providers:kubernetes                          urn:pulumi:d::pulu::eks:index:Cluster$pulumi:providers:kubernetes::my-cluster-aws-eu-west-1-eks-k8s
aws:iam/openIdConnectProvider:OpenIdConnectProvider  urn:pulumi:d::pulu::eks:index:Cluster$aws:iam/openIdConnectProvider:OpenIdConnectProvider::my-cluster-aws-eu-west-1-oidcProvider
aws:ec2/securityGroup:SecurityGroup                  urn:pulumi:d::pulu::eks:index:Cluster$aws:ec2/securityGroup:SecurityGroup::my-cluster-aws-eu-west-1-nodeSecurityGroup
kubernetes:core/v1:ConfigMap                         urn:pulumi:d::pulu::eks:index:Cluster$kubernetes:core/v1:ConfigMap::my-cluster-aws-eu-west-1-nodeAccess
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::pulu::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::my-cluster-aws-eu-west-1-eksExtApiServerClusterIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::pulu::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::my-cluster-aws-eu-west-1-eksClusterIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::pulu::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::my-cluster-aws-eu-west-1-eksNodeIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::pulu::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::my-cluster-aws-eu-west-1-eksNodeClusterIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::pulu::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::my-cluster-aws-eu-west-1-eksNodeInternetEgressRule
eks:index:VpcCni                                     urn:pulumi:d::pulu::eks:index:Cluster$eks:index:VpcCni::my-cluster-aws-eu-west-1-vpc-cni
eks:index:NodeGroupV2                                urn:pulumi:d::pulu::eks:index:NodeGroupV2::internal
eks:index:NodeGroupV2                                urn:pulumi:d::pulu::eks:index:NodeGroupV2::daemon
eks:index:NodeGroupV2                                urn:pulumi:d::pulu::eks:index:NodeGroupV2::routers
pulumi:providers:kubernetes                          urn:pulumi:d::pulu::pulumi:providers:kubernetes::k8s-provider
aws:ec2/securityGroup:SecurityGroup                  urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:ec2/securityGroup:SecurityGroup::internal-nodeSecurityGroup
aws:ec2/securityGroup:SecurityGroup                  urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:ec2/securityGroup:SecurityGroup::routers-nodeSecurityGroup
aws:ec2/securityGroup:SecurityGroup                  urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:ec2/securityGroup:SecurityGroup::daemon-nodeSecurityGroup
kubernetes:storage.k8s.io/v1:StorageClass            urn:pulumi:d::pulu::kubernetes:storage.k8s.io/v1:StorageClass::local-storage
kubernetes:core/v1:Namespace                         urn:pulumi:d::pulu::kubernetes:core/v1:Namespace::internal
kubernetes:core/v1:Namespace                         urn:pulumi:d::pulu::kubernetes:core/v1:Namespace::daemon
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:ec2/securityGroupRule:SecurityGroupRule::internal-eksExtApiServerClusterIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:ec2/securityGroupRule:SecurityGroupRule::internal-eksClusterIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:ec2/securityGroupRule:SecurityGroupRule::routers-eksNodeIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:ec2/securityGroupRule:SecurityGroupRule::daemon-eksNodeIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:ec2/securityGroupRule:SecurityGroupRule::internal-eksNodeInternetEgressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:ec2/securityGroupRule:SecurityGroupRule::routers-eksClusterIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:ec2/securityGroupRule:SecurityGroupRule::routers-eksNodeClusterIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:ec2/securityGroupRule:SecurityGroupRule::daemon-eksNodeInternetEgressRule
aws:ec2/launchTemplate:LaunchTemplate                urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:ec2/launchTemplate:LaunchTemplate::internal-launchTemplate
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:ec2/securityGroupRule:SecurityGroupRule::internal-eksNodeIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:ec2/securityGroupRule:SecurityGroupRule::daemon-eksClusterIngressRule
aws:ec2/launchTemplate:LaunchTemplate                urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:ec2/launchTemplate:LaunchTemplate::routers-launchTemplate
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:ec2/securityGroupRule:SecurityGroupRule::routers-eksExtApiServerClusterIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:ec2/securityGroupRule:SecurityGroupRule::daemon-eksExtApiServerClusterIngressRule
aws:ec2/launchTemplate:LaunchTemplate                urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:ec2/launchTemplate:LaunchTemplate::daemon-launchTemplate
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:ec2/securityGroupRule:SecurityGroupRule::internal-eksNodeClusterIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:ec2/securityGroupRule:SecurityGroupRule::routers-eksNodeInternetEgressRule
aws:ec2/securityGroupRule:SecurityGroupRule          urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:ec2/securityGroupRule:SecurityGroupRule::daemon-eksNodeClusterIngressRule
aws:autoscaling/group:Group                          urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:autoscaling/group:Group::internal
aws:autoscaling/group:Group                          urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:autoscaling/group:Group::daemon
aws:autoscaling/group:Group                          urn:pulumi:d::pulu::eks:index:NodeGroupV2$aws:autoscaling/group:Group::routers

Found no pending operations associated with d

Backend
Name           v-MacBook-Pro.local
URL            file://~
User           avi
Organizations
Token type     personal

Dependencies:
NAME                                        VERSION
github.com/pulumi/pulumi-aws/sdk/v6         v6.51.1
github.com/pulumi/pulumi-awsx/sdk/v2        v2.14.0
github.com/pulumi/pulumi-eks/sdk/v2         v2.7.9
github.com/pulumi/pulumi-kubernetes/sdk/v4  v4.18.1
github.com/pulumi/pulumi/sdk/v3             v3.132.0
flostadler commented 1 month ago

Hey @avinassh, the example you're referencing shows how to set up self-managed node groups in two different ways. What's important to note here is that each self managed node group will have their own security group. Those security groups are created with the minimal required rules. This means that by default nodes in one group aren't allowed to communicate with nodes in the other group.

This has the effect that pods on one self managed node group cannot communicate with pods on another self-managed node group by default. Can you please try adding the required ingress rules between the different node groups? Or simpler, just create a single node group.

Generally I'd recommend to use EKS managed node groups instead (unless self-managed is a requirement). The managed node groups are generally easier to operate. They also share the same security group like the API server (see https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html) so you do not have to set up additional ingress rules for cross node group traffic.

This is how you can create a managed node group: https://github.com/pulumi/pulumi-eks/blob/8bb0b9ee3f8b13b3be6989d7cfe1f8b80e5fba8a/examples/managed-nodegroups/index.ts#L42-L50

avinassh commented 1 month ago

Hey @flostadler, it worked! Thank you so much for offering the solution.

Regarding managed vs non managed nodes, I don't have any strong opinions (or requirements), so I will definitely explore managed nodes.