search

pulumi / pulumi-eks

A Pulumi component for easily creating and managing an Amazon EKS Cluster
https://www.pulumi.com/registry/packages/eks/
Apache License 2.0
168 stars 76 forks source link

TypeError: Cannot read properties of undefined (reading 'map') - eks cluster with 2 managed node groups #1202

Closed jamesongithub closed 4 days ago

jamesongithub commented 1 week ago

What happened?

trying to create eks cluster with 2 managed node groups

Example

eks_vpc = awsx.ec2.Vpc("eks-vpc",
    enable_dns_support=True,
    enable_dns_hostnames=True,
    cidr_block="192.168.0.0/16",
    subnet_strategy=awsx.ec2.SubnetAllocationStrategy.AUTO,
)

eks_cluster = eks.Cluster("eks-cluster",
    # Put the cluster in the new VPC created earlier
    vpc_id=eks_vpc.vpc_id,
    # Public subnets will be used for load balancers
    public_subnet_ids=eks_vpc.public_subnet_ids,
    # Private subnets will be used for cluster nodes
    private_subnet_ids=eks_vpc.private_subnet_ids,
    # Change configuration values to change any of the following settings

    # Do not give worker nodes a public IP address
    node_associate_public_ip_address=False,
    # Change these values for a private cluster (VPN access required)
    endpoint_private_access=False,
    endpoint_public_access=True,
    skip_default_node_group=True,
)

node_role = aws.iam.Role("node-role", 
    assume_role_policy="""{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": { "Service": "ec2.amazonaws.com" },
                "Action": "sts:AssumeRole"
            }
        ]
    }"""
)

node_role_policy_attachment_eks_worker = aws.iam.RolePolicyAttachment("node-role-policy-eks-worker",
    role=node_role.name,
    policy_arn="arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
)

node_role_policy_attachment_ecr = aws.iam.RolePolicyAttachment("node-role-policy-ecr",
    role=node_role.name,
    policy_arn="arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
)

node_role_policy_attachment_eks_cni = aws.iam.RolePolicyAttachment("node-role-policy-eks-cni",
    role=node_role.name,
    policy_arn="arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
)

# Create a managed node group with labels
managed_node_group_infra = eks.ManagedNodeGroup("infra",
    cluster=eks_cluster.core,
    node_group_name="infra-nodes",
    subnet_ids=eks_vpc.private_subnet_ids,
    instance_types=["c6a.2xlarge"],
    node_role_arn=node_role.arn,
    scaling_config=aws.eks.NodeGroupScalingConfigArgs(
        desired_size=1,
        min_size=1,
        max_size=1,
    ),
    labels={"app.myapp.com/pool": "infra"}
)

managed_node_group_compute = eks.ManagedNodeGroup("compute",
    cluster=eks_cluster.core,
    node_group_name="compute-nodes",
    subnet_ids=eks_vpc.private_subnet_ids,
    instance_types=["c6a.2xlarge"],
    node_role_arn=node_role.arn,
    scaling_config=aws.eks.NodeGroupScalingConfigArgs(
        desired_size=1,
        min_size=1,
        max_size=1,
    ),
    labels={"app.myapp.com/pool": "compute"}
)

Output of pulumi about

CLI          
Version      3.120.0
Go Version   go1.22.4
Go Compiler  gc

Plugins
KIND      NAME        VERSION
resource  aws         6.40.0
resource  awsx        2.12.1
resource  docker      4.5.4
resource  eks         2.7.1
resource  kubernetes  4.13.1
language  python      unknown

Host     
OS       darwin
Version  14.2.1
Arch     arm64

Backend        
Name           pulumi.com
URL            
User          
Organizations 
Token type     personal

Dependencies:
NAME         VERSION
pip          24.0
pulumi_awsx  2.12.1
pulumi_eks   2.7.1
setuptools   70.0.0
wheel        0.43.0

Additional context

stack trace

    error: Program failed with an unhandled exception:
    Traceback (most recent call last):
      File "/venv/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1010, in do_rpc_call
        return monitor.RegisterResource(req)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/venv/lib/python3.12/site-packages/grpc/_channel.py", line 1160, in __call__
        return _end_unary_response_blocking(state, call, False, None)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/venv/lib/python3.12/site-packages/grpc/_channel.py", line 1003, in _end_unary_response_blocking
        raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Cannot read properties of undefined (reading 'map')"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Cannot read properties of undefined (reading \'map\')", grpc_status:2, created_time:"2024-06-17T14:41:05.141244-07:00"}"
    >

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/opt/homebrew/bin/pulumi-language-python-exec", line 191, in <module>
        loop.run_until_complete(coro)
      File "/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
        return future.result()
               ^^^^^^^^^^^^^^^
      File "/venv/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 138, in run_in_stack
        await run_pulumi_func(run)
      File "/venv/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 52, in run_pulumi_func
        await wait_for_rpcs()
      File "/venv/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 114, in wait_for_rpcs
        await task
      File "/venv/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1015, in do_register
        resp = await asyncio.get_event_loop().run_in_executor(None, do_rpc_call)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/thread.py", line 58, in run
        result = self.fn(*self.args, **self.kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/venv/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1012, in do_rpc_call
        handle_grpc_error(exn)
      File "/venv/lib/python3.12/site-packages/pulumi/runtime/settings.py", line 307, in handle_grpc_error
        raise grpc_error_to_exception(exn)
    Exception: Cannot read properties of undefined (reading 'map')
    error: TypeError: Cannot read properties of undefined (reading 'map')
        at /snapshot/eks/bin/nodegroup.js:905:32
        at /snapshot/eks/node_modules/@pulumi/pulumi/output.js:260:35
        at Generator.next (<anonymous>)
        at /snapshot/eks/node_modules/@pulumi/pulumi/output.js:21:71
        at new Promise (<anonymous>)
        at __awaiter (/snapshot/eks/node_modules/@pulumi/pulumi/output.js:17:12)
        at applyHelperAsync (/snapshot/eks/node_modules/@pulumi/pulumi/output.js:239:12)
        at /snapshot/eks/node_modules/@pulumi/pulumi/output.js:193:63
        at processTicksAndRejections (node:internal/process/task_queues:95:5)
    error: TypeError: Cannot read properties of undefined (reading 'map')
        at /snapshot/eks/bin/nodegroup.js:905:32
        at /snapshot/eks/node_modules/@pulumi/pulumi/output.js:260:35
        at Generator.next (<anonymous>)
        at /snapshot/eks/node_modules/@pulumi/pulumi/output.js:21:71
        at new Promise (<anonymous>)
        at __awaiter (/snapshot/eks/node_modules/@pulumi/pulumi/output.js:17:12)
        at applyHelperAsync (/snapshot/eks/node_modules/@pulumi/pulumi/output.js:239:12)
        at /snapshot/eks/node_modules/@pulumi/pulumi/output.js:193:63
        at processTicksAndRejections (node:internal/process/task_queues:95:5)

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

rquitales commented 1 week ago

@jamesongithub Apologies that you're facing this. I was able to reproduce on my end. It seems like the error is coming from https://github.com/pulumi/pulumi-eks/blame/c95cf4402c8d3c1f71440c566a0b3d8caa2090b2/nodejs/eks/nodegroup.ts#L1662.

Note, I did not encounter this error when running this in a typescript pulumi program.

jamesongithub commented 1 week ago

Hi @rquitales ,

Thanks for reproducing.

I can't tell from that stack trace.

A node role arn was passed to both node groups. Seems like that condition should pass?

thanks

rquitales commented 1 week ago

@jamesongithub After looking at the repro code in more detail, it looks like you are missing declaring the node role within your cluster's instance roles.

Here is a python example which demonstrates this: https://github.com/pulumi/pulumi-eks/blob/c95cf4402c8d3c1f71440c566a0b3d8caa2090b2/examples/managed-nodegroups-py/__main__.py#L15

When updating the code you provided, I can successfully create the managed node groups:

eks_cluster = eks.Cluster("eks-cluster",
    # Put the cluster in the new VPC created earlier
    vpc_id=eks_vpc.vpc_id,
    # Public subnets will be used for load balancers
    public_subnet_ids=eks_vpc.public_subnet_ids,
    # Private subnets will be used for cluster nodes
    private_subnet_ids=eks_vpc.private_subnet_ids,
    # Change configuration values to change any of the following settings

    # Do not give worker nodes a public IP address
    node_associate_public_ip_address=False,
    # Change these values for a private cluster (VPN access required)
    endpoint_private_access=False,
    endpoint_public_access=True,
    skip_default_node_group=True,
    instance_roles=[node_role],
)

# ...  rest of code

This is necessary because the nodegroup role needs to be set on the cluster to ensure that the aws-auth ConfigMap contains the required entries. Alternatively, you could use the newer Access Entry auth mode instead. Please see this documentation on enabling Access Entry for EKS clusters using this provider.

I'll keep this issue open however, since we can better handle the Cannot read properties of undefined (reading 'map') error we both observed so that a more meaningful error is presented.

jamesongithub commented 1 week ago

Ok, thanks, Ill test it out.

It actually doesn't mention it as a prerequisite here: https://docs.aws.amazon.com/eks/latest/userguide/create-managed-node-group.html

Might be helpful to have a check on eks_cluster like there is on the managed_node_groups like you had referenced if it's a required attribute. (I'm unclear if it's needed only if you use managed node groups or not)

jamesongithub commented 5 days ago

@rquitales I added just the node role only and it did not work.

i had to change this reference from eks_cluster.core to eks_cluster

managed_node_group_infra = eks.ManagedNodeGroup("infra",
...
    cluster=eks_cluster.core,
    node_group_name="infra-nodes",
...
managed_node_group_infra = eks.ManagedNodeGroup("infra",
...
    cluster=eks_cluster,
    node_group_name="infra-nodes",
...

both changes, the node role in eks_cluster and this eks_cluster reference was required to deploy successfully

rquitales commented 5 days ago

Thanks for the updates and further investigation - I'll look into why cluster.core isn't working. Glad you've managed to get unblocked though.