Cannot disable default pulumi providers for EKS cluster component

[flostadler]

What happened?

The EKS cluster component uses a Custom Resource to manage the VPC CNI. The cluster component does not support setting a provider for this Custom Resource and this trips up disable-default-providers.

In the long term we should fix this by replacing our custom VPC CNI by the EKS addon https://github.com/pulumi/pulumi-eks/issues/1261.

As a workaround users can opt out of using this custom resource by setting useDefaultVpcCni and instead manage the CNI with the EKS addon like so:

import * as pulumi from "@pulumi/pulumi";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as aws from "@pulumi/aws";

const awsProvider = new aws.Provider("prov", {
    region: "us-west-2",
});

const awsxProvider = new awsx.Provider("prov");
const projectName = pulumi.getProject();

// Create a VPC with public subnets only
const vpc = new awsx.ec2.Vpc(`${projectName}-vpc`, {
    tags: {"Name": `${projectName}-2`},
    subnetSpecs: [
        { type: "Public" }
    ],
    natGateways: {
        strategy: "None",
    }
}, { provider: awsxProvider, providers: {aws: awsProvider} });

const cluster = new eks.Cluster(`${projectName}-cluster`, {
    vpcId: vpc.vpcId,
    publicSubnetIds: vpc.publicSubnetIds,
    skipDefaultNodeGroup: true,
    useDefaultVpcCni: true,
}, { providers: { aws: awsProvider } });

const addon = new aws.eks.Addon("vpc-cni", {
        addonName: "vpc-cni",
        clusterName: cluster.core.cluster.name,
        configurationValues: JSON.stringify({
            // Add your desired config here
        }),
    },
    { provider: awsProvider },
);

Example

import * as pulumi from "@pulumi/pulumi";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as aws from "@pulumi/aws";

const awsProvider = new aws.Provider("prov", {
    region: "us-west-2",
});

const awsxProvider = new awsx.Provider("prov");
const projectName = pulumi.getProject();

// Create a VPC with public subnets only
const vpc = new awsx.ec2.Vpc(`${projectName}-vpc`, {
    tags: {"Name": `${projectName}-2`},
    subnetSpecs: [
        { type: "Public" }
    ],
    natGateways: {
        strategy: "None",
    }
}, { provider: awsxProvider, providers: {aws: awsProvider} });

const cluster = new eks.Cluster(`${projectName}-cluster`, {
    vpcId: vpc.vpcId,
    publicSubnetIds: vpc.publicSubnetIds,
    skipDefaultNodeGroup: true,
}, { providers: { aws: awsProvider } });

Output of pulumi about

n/a

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[t0yv0]

Ref: https://www.pulumi.com/docs/concepts/config/#pulumidisable-default-providers

[t0yv0]

@flostadler this is interesting, I had another look. VpcCni is implemented inline in the pulumi-eks provider itself: https://github.com/pulumi/pulumi-eks/blob/master/nodejs/eks/cmd/provider/cni.ts#L276

Here is the invocation call: https://github.com/pulumi/pulumi-eks/blob/master/nodejs/eks/cluster.ts#L905

It sounds like it indirects though... since it's emitting a RegisterResource call it may end up calling a different version of the eks provider from the current one. This is not great. I wonder if we could inject a ResoruceOption version into this call to tie somehow and force the use of the current version of the eks provider.

Also it is possible that behavior is different here between Node and other languages since Node omits the regular gRPC boundary.

[flostadler]

@t0yv0 we are setting the the ResourceOption version for the CNI resource: https://github.com/pulumi/pulumi-eks/blob/3d710939789205cffea51b97d55e6f3721d302c7/nodejs/eks/cni.ts#L235 But you're right, before that this caused the latest version to be fetched..

This is still a very complex (and brittle) situation. Doing https://github.com/pulumi/pulumi-eks/issues/1261 will remove the need for all of this