(eks): 'dependency cannot cross stage boundaries' error occured when create HelmChart in Stage

kkparkclouflake commented 2 years ago

What is the problem?

I tried to deploy EKS, EKS NodeGroup, and HelmCharts in single Stage. But it fails when synth time. After remove 'HelmChart' statements, Synth works well. I think it's a problem of Kubectl Nested stack.... so Maybe it can occured when deploy K8s manifest too... but I don't know. I didn't tested.

Reproduction Steps

    ....
    pipeline.addStage(new ClusterStackStage(this, props.config.eks.cluster.name + "-Stage", props));
  }
}

export class ClusterStackStage extends cdk.Stage {
    constructor(scope: cdk.Construct, id: string, props: GlobalProps) {
      super(scope, id, props);

      // All stacks share one props for reuse variable like EKS Cluster and configurations.

      // It deploys EKS Cluster. cluster stored to props.
      const clusterStack = new ClusterStack(this, 'ClusterStack', props);

      // It deploys EKS NodeGroup.
      const ngStack = new ClusterNodeGroupStack(this, 'ClusterNodeGroupStack', props);

      // It deploys EBS CSI driver, EFS CSI driver, AWS LoadBalancer Controller, External-DNS..... and more.
      // So this creates IAM Role and Policy, EKS Service Account, and HelmChart.
      new DriverStack(this, 'DriverStack', props);
    }
}

What did you expect to happen?

Just works.

What actually happened?

D:\CDK\eks-sample\node_modules\@aws-cdk\core\lib\deps.ts:39
    throw new Error(`You cannot add a dependency from '${source.node.path}' (in ${describeStage(sourceStage)}) to '${target.node.path}' (in ${describeStage(targetStage)}): dependency cannot cross stage boundaries`);       
          ^
Error: You cannot add a dependency from 'CodePipelineStack/cluster-Stage/DriverStack' (in Stage 'CodePipelineStack/cluster-Stage') to 'ClusterStack' (in the App): dependency cannot cross stage boundaries
    at Object.addDependency (D:\CDK\eks-sample\node_modules\@aws-cdk\core\lib\deps.ts:39:11)
    at DriverStack.addDependency (D:\CDK\eks-sample\node_modules\@aws-cdk\core\lib\stack.ts:271:5)
    at resolveValue (D:\CDK\eks-sample\node_modules\@aws-cdk\core\lib\private\refs.ts:100:12)
    at Object.resolveReferences (D:\CDK\eks-sample\node_modules\@aws-cdk\core\lib\private\refs.ts:30:24)
    at Object.prepareApp (D:\CDK\eks-sample\node_modules\@aws-cdk\core\lib\private\prepare-app.ts:31:3)
    at Object.synthesize (D:\CDK\eks-sample\node_modules\@aws-cdk\core\lib\private\synthesis.ts:24:3)
    at ClusterStackStage.synth (D:\CDK\eks-sample\node_modules\@aws-cdk\core\lib\stage.ts:94:23)
    at Object.pipelineSynth (D:\CDK\eks-sample\node_modules\@aws-cdk\pipelines\lib\private\construct-internals.ts:27:16)
    at Function.fromStage (D:\CDK\eks-sample\node_modules\@aws-cdk\pipelines\lib\blueprint\stage-deployment.ts:27:22)
    at Wave.addStage (D:\CDK\eks-sample\node_modules\@aws-cdk\pipelines\lib\blueprint\wave.ts:34:33)
Subprocess exited with error 1

CDK CLI Version

1.134.0 (build dd5e12d)

Framework Version

"aws-cdk": "^1.134.0"

Node.js Version

v14.17.4

OS

Windows 10

Language

Typescript

Language Version

"typescript": "~3.9.7"

Other information

No response

rix0rrr commented 2 years ago

This is not a pipelines issue but an EKS issue. The root cause is the same as https://github.com/aws/aws-cdk/issues/17643 and the resolution is the same as https://github.com/aws/aws-cdk/pull/17730

otaviomacedo commented 2 years ago

Hi, @kkparkclouflake

I'm a bit confused, because the error message doesn't match your code snippet. The message says that ClusterStack is in App. But your code shows ClusterStack being created in the scope of ClusterStackStage., which should be fine.

Can you show me how you're creating the Cluster and the HelmChart?

kkparkclouflake commented 2 years ago

Hi, @kkparkclouflake

I'm a bit confused, because the error message doesn't match your code snippet. The message says that ClusterStack is in App. But your code shows ClusterStack being created in the scope of ClusterStackStage., which should be fine.

Can you show me how you're creating the Cluster and the HelmChart?

Hi, @otaviomacedo ! That's it. I create EKS cluster, Node Group, and Helm Chart inside Stage. and that App error occured.

Create EKS Cluster inside ClusterStack.

const cluster = /* some EKS cluster creation */;

Insert cluster into shared props
```
props.cluster = cluster;
```

Create HelmChart inside DriverStack using props

new eks.HelmChart(this, config.releaseName, {
    cluster: props.cluster!,
    release: config.releaseName,
    repository: 'https://charts.bitnami.com/bitnami',
    chart: 'metrics-server',
    namespace: config.namespace,
    values: parseYaml(valuesFile)!,
})

anyway, I have make some investigations, and (maybe) found specific reason of this error. It occured when create HelmChart with imported cluster (using eks.Cluster.fromClusterAttributes).

const cluster = eks.Cluster.fromClusterAttributes(this, 'LookupEks', {
      clusterName,
      clusterSecurityGroupId: config.securityGroupId,
      vpc: props.vpc,
      openIdConnectProvider: oidcProvider,
      kubectlRoleArn: config.kubectlRoleArn
    });

In my CDK source, the Cluster can be an Imported cluster (eks.ICluster) by configurations. It's OK with created EKS cluster, but always error with imported EKS cluster.

I've tested it like this;

 const cluster = new eks.Cluster(this, clusterName, {
    version: eks.KubernetesVersion.of('1.19'),
    clusterName,
    defaultCapacity: 0,
    vpc: props.vpc,
    vpcSubnets: [
      { subnetType: ec2.SubnetType.PRIVATE_WITH_NAT }
    ],
  });

...and synth completed without error.

otaviomacedo commented 2 years ago

Did you, by any chance, import the cluster with:

Cluster.fromClusterAttributes(app, 'cluster', {...})
//                            ☝️

kkparkclouflake commented 2 years ago

Did you, by any chance, import the cluster with:
Cluster.fromClusterAttributes(app, 'cluster', {...})
//                            ☝️

No. I import with this, and that statement are inside my custom Construct. I copied and pasted my source as-is.

// entrypoint
const app = new cdk.App();
...
// 2021-12-09 Match to Stage source
new ClusterStack(this, 'ClusterStack-' + clusterName, props);
// ⬇️ to `ClusterStack`
const eks = new LookupEks(this, 'ExistEksCluster', props);
// ⬇️ to `LookupEks`
const cluster = eks.Cluster.fromClusterAttributes(this, 'LookupEks', {
      clusterName,
      clusterSecurityGroupId: config.securityGroupId,
      vpc: props.vpc,
      openIdConnectProvider: oidcProvider,
      kubectlRoleArn: config.kubectlRoleArn
    });

otaviomacedo commented 2 years ago

I don't think the problem has to do with Cluster.fromClusterAttributes. In your last snippet, you have:

const clusterStack = new ClusterStack(app, 'ClusterStack-' + config.eks.cluster.name , globalProps);

Here clusterStack is in the scope of the App, as the error message indicates.

kkparkclouflake commented 2 years ago

I don't think the problem has to do with Cluster.fromClusterAttributes. In your last snippet, you have:
const clusterStack = new ClusterStack(app, 'ClusterStack-' + config.eks.cluster.name , globalProps);
Here clusterStack is in the scope of the App, as the error message indicates.

Oh, It's my copy mistake. I copied that line from CDK Entrypoint (in bin/ directory). In my Stage code, That line are have this scope like;

import * as cdk from '@aws-cdk/core';
import { GlobalProps } from '../props/global-props';
import { ClusterStack } from '../cluster-stack';
import { ClusterNodeGroupStack } from '../cluster-node-group-stack';
import { DriverStack } from '../driver-stack';

export class ClusterStackStage extends cdk.Stage {
    constructor(scope: cdk.Construct, id: string, props: GlobalProps) {
      super(scope, id, props);

      const clusterName = props.config.eks.cluster.name;

      // Here
      new ClusterStack(this, 'ClusterStack-' + clusterName, props);
      new ClusterNodeGroupStack(this, 'ClusterNodeGroupStack-' + clusterName, props);
      new DriverStack(this, 'DriverStack-' + clusterName, props);
    }
}

I've recheck this problem, but still error occured if it's imported cluster

kkparkclouflake commented 2 years ago

I resolved this!

I add some debug logs into @aws-cdk\core\lib\deps.ts like this;

...
    if (sourceStage !== targetStage) {
        // eslint-disable-next-line max-len
        console.log("sourceStack -> ", sourceStack);
        console.log(" ----------------------------- ");
        console.log("sourceStage -> ", sourceStage);
        console.log(" ----------------------------- ");
        console.log("targetStack -> ", targetStack);
        console.log(" ----------------------------- ");
        console.log("targetStage -> ", targetStage);
        throw new Error(`You cannot add a dependency from '${source.node.path}' (in ${describeStage(sourceStage)}) to '${target.node.path}' (in ${describeStage(targetStage)}): dependency cannot cross stage boundaries`);
    }
...

and... It print weird result.

SourceStack: DriverStack in ClusterStackStage scope
SourceStage: ClusterStackStage in CodePipelineStack scope
TargetStack: ClusterStack in App scope
TargetStage: App in undefined scope

It's very weird...

ClusterStack are defined just two times whole project. entrypoint, and stage. so... I check my entrypoint. On entrypoint, Initialize other stacks first. and CodePipelineStack initialized finally.

let config = // some config read statements

// Build EKS props
const globalProps: GlobalProps = {
  env: {account: account, region: config.eks.region || 'us-west-2'},
  config: config
}

const clusterStack = new ClusterStack(app, `ClusterStack-${config.eks.cluster.name}`, globalProps);
const ngStack = new ClusterNodeGroupStack(app, `ClusterNodeGroupStack-${config.eks.cluster.name}`, globalProps);
new DriverStack(app, `DriverStack-${config.eks.cluster.name}`, globalProps);
ngStack.addDependency(clusterStack);

// ----------- Pipeline Stack -----------

const cpStack = new CodePipelineStack(app, `CodePipelineStack-${config.eks.cluster.name}`, globalProps);

and... I realize my mistake. I share the globalProps object between normal Stacks and CodePipelineStack. That object contains some CDK object like vpc, EKS cluster for sharing that.

I think CodePipelineStack and ClusterStackStage needs fresh GlobalProps object, so seperately define Props. And synth.... failed.

Error: ENOENT: no such file or directory, open 'cdk.out\assembly-CodePipelineStack-<MASKED>-Pipeline-Stage\CodePipelineStack<MASKED>PipelineStageDriverStack<MASKED>CodePipelineStack<MASKED>PipelineStageClusterStack<MASKED>ExistEksClusterLookupEks57BFF4C9KubectlProviderEC48CD0C.nested.template.json'
    at Object.openSync (fs.js:498:3)
    at Object.writeFileSync (fs.js:1524:35)
    at KubectlProvider._synthesizeTemplate (D:\CDK\<MASKED>\node_modules\@aws-cdk\core\lib\stack.ts:447:8)
    at NestedStackSynthesizer.synthesizeStackTemplate (D:\CDK\<MASKED>\node_modules\@aws-cdk\core\lib\stack-synthesizers\stack-synthesizer.ts:23:11)
    at NestedStackSynthesizer.synthesize (D:\CDK\<MASKED>\node_modules\@aws-cdk\core\lib\stack-synthesizers\nested.ts:39:10)
    at D:\CDK\<MASKED>\node_modules\@aws-cdk\core\lib\private\synthesis.ts:184:29
    at visit (D:\CDK\<MASKED>\node_modules\@aws-cdk\core\lib\private\synthesis.ts:231:5)
    at visit (D:\CDK\<MASKED>\node_modules\@aws-cdk\core\lib\private\synthesis.ts:227:5)
    at visit (D:\CDK\<MASKED>\node_modules\@aws-cdk\core\lib\private\synthesis.ts:227:5)
    at synthesizeTree (D:\CDK\<MASKED>\node_modules\@aws-cdk\core\lib\private\synthesis.ts:176:3)

But it's different error. I don't know why, but resolved by changing Stage scope.

    // Add stage
    pipeline.addStage(new ClusterStackStage(scope, `${pipelineName}-Stage`, props));
                                         // ↑↑↑↑↑ 'this' to 'scope'

After all, Synth works now.

I think it's not an CDK issue. I close the issue.

github-actions[bot] commented 2 years ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

aws / aws-cdk