pulumi / pulumi-eks

A Pulumi component for easily creating and managing an Amazon EKS Cluster
https://www.pulumi.com/registry/packages/eks/
Apache License 2.0
171 stars 80 forks source link

Set `additionalSecrets` on eksCluster #832

Closed aureq closed 3 weeks ago

aureq commented 1 year ago

What happened?

When using this eks provider, I noticed that cluster.kubeconfig is not set as a secret. This is a problem, because the value is stored in plain text in the stack state file. Unfortunately, the value cannot be protected from the outside of the component using AdditionalSecretOutputs since it's a component resource.

Additionally, the provider should secret clusterCertificateAuthority.data, certificateAuthorities[].data as well.

I think this is the block of code that would need the change. https://github.com/pulumi/pulumi-eks/blob/master/nodejs/eks/cluster.ts#L522-L525

Steps to reproduce

  1. Create an EKS cluster
  2. run pulumi stack export

Expected Behavior

Actual Behavior

Output of pulumi about

CLI          
Version      3.46.1
Go Version   go1.19.2
Go Compiler  gc

Plugins
NAME        VERSION
aws         5.22.0
aws         5.10.0
eks         1.0.0
kubernetes  3.22.2
kubernetes  3.20.2
nodejs      unknown

Host     
OS       debian
Version  11.5
Arch     x86_64

This project is written in nodejs: executable='/usr/local/bin/node' version='v18.12.1'

Current Stack: dev

TYPE                                                        URN
pulumi:pulumi:Stack                                         urn:pulumi:dev::aws-ts-eks-grafana::pulumi:pulumi:Stack::aws-ts-eks-grafana-dev
pulumi:providers:aws                                        urn:pulumi:dev::aws-ts-eks-grafana::pulumi:providers:aws::default_5_22_0
pulumi:providers:aws                                        urn:pulumi:dev::aws-ts-eks-grafana::pulumi:providers:aws::default_5_10_0
custom:resource:VPC                                         urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC::eks-grafana
custom:resource:EKS                                         urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$custom:resource:EKS::eks-grafana
aws:ec2/vpc:Vpc                                             urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$aws:ec2/vpc:Vpc::eks-grafana-vpc
aws:iam/role:Role                                           urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$custom:resource:EKS$aws:iam/role:Role::eks-grafana-role
aws:ec2/internetGateway:InternetGateway                     urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$aws:ec2/vpc:Vpc$aws:ec2/internetGateway:InternetGateway::eks-grafana-igw
aws:ec2/subnet:Subnet                                       urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::eks-grafana-private-subnet-ap-southeast-2b
aws:ec2/securityGroup:SecurityGroup                         urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$aws:ec2/vpc:Vpc$aws:ec2/securityGroup:SecurityGroup::eks-grafana-public-sg-ssh
aws:ec2/subnet:Subnet                                       urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::eks-grafana-private-subnet-ap-southeast-2a
aws:ec2/subnet:Subnet                                       urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::eks-grafana-private-subnet-ap-southeast-2c
aws:ec2/subnet:Subnet                                       urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::eks-grafana-public-subnet-ap-southeast-2a
aws:ec2/subnet:Subnet                                       urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::eks-grafana-public-subnet-ap-southeast-2c
aws:ec2/subnet:Subnet                                       urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::eks-grafana-public-subnet-ap-southeast-2b
eks:index:Cluster                                           urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster::eks-grafana-cluster
aws:ec2/routeTable:RouteTable                               urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$aws:ec2/vpc:Vpc$aws:ec2/routeTable:RouteTable::eks-grafana-rt
aws:iam/rolePolicyAttachment:RolePolicyAttachment           urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$custom:resource:EKS$aws:iam/role:Role$aws:iam/rolePolicyAttachment:RolePolicyAttachment::eks-grafana-role-policy-2
aws:iam/rolePolicyAttachment:RolePolicyAttachment           urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$custom:resource:EKS$aws:iam/role:Role$aws:iam/rolePolicyAttachment:RolePolicyAttachment::eks-grafana-role-policy-0
aws:iam/rolePolicyAttachment:RolePolicyAttachment           urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$custom:resource:EKS$aws:iam/role:Role$aws:iam/rolePolicyAttachment:RolePolicyAttachment::eks-grafana-role-policy-1
eks:index:ServiceRole                                       urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$eks:index:ServiceRole::eks-grafana-cluster-eksRole
aws:ec2/securityGroup:SecurityGroup                         urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$aws:ec2/securityGroup:SecurityGroup::eks-grafana-cluster-eksClusterSecurityGroup
aws:ec2/routeTableAssociation:RouteTableAssociation         urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$aws:ec2/vpc:Vpc$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::eks-grafana-public-rt-assoc-ap-southeast-2c
aws:ec2/routeTableAssociation:RouteTableAssociation         urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$aws:ec2/vpc:Vpc$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::eks-grafana-public-rt-assoc-ap-southeast-2a
aws:ec2/routeTableAssociation:RouteTableAssociation         urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$aws:ec2/vpc:Vpc$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::eks-grafana-public-rt-assoc-ap-southeast-2b
aws:iam/role:Role                                           urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$eks:index:ServiceRole$aws:iam/role:Role::eks-grafana-cluster-eksRole-role
aws:ec2/securityGroupRule:SecurityGroupRule                 urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::eks-grafana-cluster-eksClusterInternetEgressRule
aws:iam/rolePolicyAttachment:RolePolicyAttachment           urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$eks:index:ServiceRole$aws:iam/rolePolicyAttachment:RolePolicyAttachment::eks-grafana-cluster-eksRole-4b490823
aws:eks/cluster:Cluster                                     urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$aws:eks/cluster:Cluster::eks-grafana-cluster-eksCluster
aws:ec2/securityGroup:SecurityGroup                         urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$aws:ec2/securityGroup:SecurityGroup::eks-grafana-cluster-nodeSecurityGroup
aws:iam/openIdConnectProvider:OpenIdConnectProvider         urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$aws:iam/openIdConnectProvider:OpenIdConnectProvider::eks-grafana-cluster-oidcProvider
aws:ec2/securityGroupRule:SecurityGroupRule                 urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::eks-grafana-cluster-eksNodeClusterIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule                 urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::eks-grafana-cluster-eksNodeIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule                 urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::eks-grafana-cluster-eksExtApiServerClusterIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule                 urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::eks-grafana-cluster-eksNodeInternetEgressRule
aws:ec2/securityGroupRule:SecurityGroupRule                 urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::eks-grafana-cluster-eksClusterIngressRule
aws:iam/role:Role                                           urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$custom:resource:EKS$aws:iam/role:Role::eks-grafana-ca-role
pulumi:providers:kubernetes                                 urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$pulumi:providers:kubernetes::eks-grafana-cluster-eks-k8s
pulumi:providers:eks                                        urn:pulumi:dev::aws-ts-eks-grafana::pulumi:providers:eks::default
pulumi:providers:kubernetes                                 urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$pulumi:providers:kubernetes::eks-grafana-cluster-provider
kubernetes:storage.k8s.io/v1:StorageClass                   urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$kubernetes:storage.k8s.io/v1:StorageClass::eks-grafana-cluster-gp2
kubernetes:core/v1:ConfigMap                                urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$kubernetes:core/v1:ConfigMap::eks-grafana-cluster-nodeAccess
eks:index:VpcCni                                            urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$eks:index:VpcCni::eks-grafana-cluster-vpc-cni
kubernetes:core/v1:Namespace                                urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$kubernetes:core/v1:Namespace::eks-grafana
kubernetes:core/v1:ServiceAccount                           urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$kubernetes:core/v1:ServiceAccount::cluster-autoscaler
kubernetes:helm.sh/v3:Chart                                 urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$kubernetes:core/v1:Namespace$kubernetes:helm.sh/v3:Chart::eks-grafana-chart-ingress-nginx
aws:eks/nodeGroup:NodeGroup                                 urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$aws:eks/nodeGroup:NodeGroup::eks-grafana-cluster-ng
kubernetes:helm.sh/v3:Chart                                 urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$aws:eks/nodeGroup:NodeGroup$kubernetes:helm.sh/v3:Chart::eks-grafana-chart-cluster-autoscaler
kubernetes:core/v1:ConfigMap                                urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$kubernetes:core/v1:Namespace$kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:ConfigMap::eks-grafana/eks-grafana-chart-ingress-nginx-controller
kubernetes:rbac.authorization.k8s.io/v1:RoleBinding         urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$kubernetes:core/v1:Namespace$kubernetes:helm.sh/v3:Chart$kubernetes:rbac.authorization.k8s.io/v1:RoleBinding::eks-grafana/eks-grafana-chart-ingress-nginx
kubernetes:core/v1:ServiceAccount                           urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$kubernetes:core/v1:Namespace$kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:ServiceAccount::eks-grafana/eks-grafana-chart-ingress-nginx
kubernetes:networking.k8s.io/v1:IngressClass                urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$kubernetes:core/v1:Namespace$kubernetes:helm.sh/v3:Chart$kubernetes:networking.k8s.io/v1:IngressClass::nginx
kubernetes:core/v1:Service                                  urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$kubernetes:core/v1:Namespace$kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Service::eks-grafana/eks-grafana-chart-ingress-nginx-controller
kubernetes:rbac.authorization.k8s.io/v1:Role                urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$kubernetes:core/v1:Namespace$kubernetes:helm.sh/v3:Chart$kubernetes:rbac.authorization.k8s.io/v1:Role::eks-grafana/eks-grafana-chart-ingress-nginx
kubernetes:rbac.authorization.k8s.io/v1:ClusterRole         urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$kubernetes:core/v1:Namespace$kubernetes:helm.sh/v3:Chart$kubernetes:rbac.authorization.k8s.io/v1:ClusterRole::eks-grafana-chart-ingress-nginx
kubernetes:apps/v1:Deployment                               urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$kubernetes:core/v1:Namespace$kubernetes:helm.sh/v3:Chart$kubernetes:apps/v1:Deployment::eks-grafana/eks-grafana-chart-ingress-nginx-controller
kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding  urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$kubernetes:core/v1:Namespace$kubernetes:helm.sh/v3:Chart$kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding::eks-grafana-chart-ingress-nginx
kubernetes:core/v1:Service                                  urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$aws:eks/nodeGroup:NodeGroup$kubernetes:helm.sh/v3:Chart$kubernetes:core/v1:Service::kube-system/eks-grafana-chart-cluster-autoscaler-aws-cluster-autoscaler
kubernetes:rbac.authorization.k8s.io/v1:Role                urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$aws:eks/nodeGroup:NodeGroup$kubernetes:helm.sh/v3:Chart$kubernetes:rbac.authorization.k8s.io/v1:Role::kube-system/eks-grafana-chart-cluster-autoscaler-aws-cluster-autoscaler
kubernetes:rbac.authorization.k8s.io/v1:RoleBinding         urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$aws:eks/nodeGroup:NodeGroup$kubernetes:helm.sh/v3:Chart$kubernetes:rbac.authorization.k8s.io/v1:RoleBinding::kube-system/eks-grafana-chart-cluster-autoscaler-aws-cluster-autoscaler
kubernetes:policy/v1beta1:PodDisruptionBudget               urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$aws:eks/nodeGroup:NodeGroup$kubernetes:helm.sh/v3:Chart$kubernetes:policy/v1beta1:PodDisruptionBudget::kube-system/eks-grafana-chart-cluster-autoscaler-aws-cluster-autoscaler
kubernetes:apps/v1:Deployment                               urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$aws:eks/nodeGroup:NodeGroup$kubernetes:helm.sh/v3:Chart$kubernetes:apps/v1:Deployment::kube-system/eks-grafana-chart-cluster-autoscaler-aws-cluster-autoscaler
kubernetes:rbac.authorization.k8s.io/v1:ClusterRole         urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$aws:eks/nodeGroup:NodeGroup$kubernetes:helm.sh/v3:Chart$kubernetes:rbac.authorization.k8s.io/v1:ClusterRole::eks-grafana-chart-cluster-autoscaler-aws-cluster-autoscaler
kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding  urn:pulumi:dev::aws-ts-eks-grafana::custom:resource:VPC$eks:index:Cluster$aws:eks/nodeGroup:NodeGroup$kubernetes:helm.sh/v3:Chart$kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding::eks-grafana-chart-cluster-autoscaler-aws-cluster-autoscaler

Found no pending operations associated with dev

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/aureq
User           aureq
Organizations  aureq, team-ce, menfin, menfin-team, demo, pulumi

Dependencies:
NAME                VERSION
@pulumi/aws         5.22.0
@pulumi/eks         1.0.0
@pulumi/kubernetes  3.22.2
@pulumi/pulumi      3.48.0
@types/netmask      1.0.30
netmask             2.0.2
@types/node         18.11.10

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

roothorp commented 1 year ago

I've raised https://github.com/pulumi/pulumi/issues/11545 with regards to making additionalSecretOutputs work with components, which I think is the best way forward with this. Would this be an acceptable resolution for you @aureq?

justinvp commented 1 year ago

pulumi/pulumi#11545 with regards to making additionalSecretOutputs work with components

We likely can address this inside EKS w/out waiting on pulumi/pulumi#11545. I'd have to try it out, but we should be able to wrap the kubeconfig as a secret when registering the output as well as when returning it as state from Construct.

aureq commented 1 year ago

@justinvp @roothorp Yes, I think addressing this inside pulumi-eks is probably the better way. For such sensitive information like kubeconfig, clusterCertificateAuthority.data and certificateAuthorities[].data, our users shouldn't have to second guess (like looking into the stack state) what is a secret and what is not.

I think https://github.com/pulumi/pulumi/issues/11545 would partly answer the problem because if a resource is private to the component resource, then additionalSecretOutputs may not be able to reach that property (though may not be the case for pulumi-eks).

pawelprazak commented 8 months ago

I've stumbled upon this issue as well, for the record, this affects multiple places in the schema:

pawelprazak commented 8 months ago

After looking into this a bit further, the CA data looks non-sensitive/public: https://stackoverflow.com/questions/71444145/should-the-k8s-cluster-certificate-authority-be-kept-secret

Also, the AWS provider does not mark the CA as secret in the schema (and if it were it should propagate through outputs):

"certificateAuthority": {
          "$ref": "#/types/aws:eks%2FClusterCertificateAuthority:ClusterCertificateAuthority",
          "description": "Attribute block containing `certificate-authority-data` for your cluster. Detailed below.\n"
        },

IIUC, the secret / token is fetched just-in-time at runtime, using this command:

{
              - name: "aws"
              - user: {
                  - exec: {
                      - apiVersion: "client.authentication.k8s.io/v1beta1"
                      - args      : [
                      -     [0]: "eks"
                      -     [1]: "get-token"
                      -     [2]: "--cluster-name"
                      -     [3]: "my-cluster-eksCluster-1528258"
                        ]
                      - command   : "aws"
                      - env       : [
                      -     [0]: {
                              - name : "KUBERNETES_EXEC_INFO"
                              - value: (json) {
                                  - apiVersion: "client.authentication.k8s.io/v1beta1"
                                }

                            }
                        ]
                    }
                }

If the above is true, then the kubeconfig generated by the current implementation should not contain any secrets.

Looking at the kubeconfig documentation, what looks like secrets are:

I hope this gives some important context to this conversation.

flostadler commented 3 weeks ago

@pawelprazak is right here, neither kubeconfig nor the CA data is sensitive so it doesn't need to be stored safely.