search

data-dot-all / dataall

A modern data marketplace that makes collaboration among diverse users (like business, analysts and engineers) easier, increasing efficiency and agility in data projects on AWS.
https://data-dot-all.github.io/dataall/
Apache License 2.0
232 stars 82 forks source link

cdk synth is failing on latest commit, 'apihandlerSGoss' in NestedStack #1120

Closed rbernotas closed 5 months ago

rbernotas commented 7 months ago

Describe the bug

We pulled in the latest commit 78014277561d87bd2520538ecd6ee5f42b61ecf7 on top of the 2.3 release, and found that cdk synth is failing with the following errors during the Build stage of CodePipeline:

[Container] 2024/03/21 20:06:20.064703 Running command cdk synth [2024-03-21 20:06:29,387][app.py : 50] Trying to get cdkjson parameter from SSM [2024-03-21 20:06:29,485][app.py : 60] SSM parameter not found - Proceeding with cdk.json and cdk.context.json in code [2024-03-21 20:06:29,553][app.py : 65] Loaded context from cdk.json file in repository [WARNING] aws-cdk-lib.aws_ec2.VpcProps#cidr is deprecated. Use ipAddresses instead This API will be removed in the next major release. [WARNING] aws-cdk-lib.aws_ec2.SubnetType#PRIVATE_WITH_NAT is deprecated. usePRIVATE_WITH_EGRESS This API will be removed in the next major release. [WARNING] aws-cdk-lib.CfnResource#addDependsOn is deprecated. use addDependency This API will be removed in the next major release. [WARNING] aws-cdk-lib.CfnResource#addDependsOn is deprecated. use addDependency This API will be removed in the next major release. [WARNING] aws-cdk-lib.aws_ec2.VpcProps#cidr is deprecated. Use ipAddresses instead This API will be removed in the next major release. [WARNING] aws-cdk-lib.aws_ec2.SubnetType#PRIVATE_WITH_NAT is deprecated. usePRIVATE_WITH_EGRESS This API will be removed in the next major release. [WARNING] aws-cdk-lib.aws_lambda.EcrImageCodeProps#tag is deprecated. usetagOrDigest This API will be removed in the next major release. [WARNING] aws-cdk-lib.aws_lambda.EcrImageCodeProps#tag is deprecated. usetagOrDigest` This API will be removed in the next major release. jsii.errors.JavaScriptError: Error: There is already a Construct with name 'apihandlerSGoss' in NestedStack [Lambdas] at Node.addChild (/tmp/jsii-kernel-hQVUhe/node_modules/constructs/lib/construct.js:375:19) at new Node (/tmp/jsii-kernel-hQVUhe/node_modules/constructs/lib/construct.js:38:21) at new Construct (/tmp/jsii-kernel-hQVUhe/node_modules/constructs/lib/construct.js:426:21) at new Resource (/tmp/jsii-kernel-hQVUhe/node_modules/aws-cdk-lib/core/lib/resource.js:1:1309) at new SecurityGroupBase (/tmp/jsii-kernel-hQVUhe/node_modules/aws-cdk-lib/aws-ec2/lib/security-group.js:1:1051) at new SecurityGroup (/tmp/jsii-kernel-hQVUhe/node_modules/aws-cdk-lib/aws-ec2/lib/security-group.js:1:5332) at Kernel._Kernel_create (/tmp/tmp836cvn7g/lib/program.js:10108:25) at Kernel.create (/tmp/tmp836cvn7g/lib/program.js:9779:93) at KernelHost.processRequest (/tmp/tmp836cvn7g/lib/program.js:11696:36) at KernelHost.run (/tmp/tmp836cvn7g/lib/program.js:11656:22)

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/codebuild/output/src361467506/src/./deploy/app.py", line 81, in pipeline = PipelineStack( ^^^^^^^^^^^^^^ File "/root/.pyenv/versions/3.11.6/lib/python3.11/site-packages/jsii/_runtime.py", line 118, in call inst = super(JSIIMeta, cast(JSIIMeta, cls)).call(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/codebuild/output/src361467506/src/deploy/stacks/pipeline.py", line 175, in init backend_stage = self.set_backend_stage(target_env, repository_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/codebuild/output/src361467506/src/deploy/stacks/pipeline.py", line 601, in set_backend_stage BackendStage( File "/root/.pyenv/versions/3.11.6/lib/python3.11/site-packages/jsii/_runtime.py", line 118, in call inst = super(JSIIMeta, cast(JSIIMeta, cls)).call(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/codebuild/output/src361467506/src/deploy/stacks/backend_stage.py", line 42, in init backend_stack = BackendStack( ^^^^^^^^^^^^^ File "/root/.pyenv/versions/3.11.6/lib/python3.11/site-packages/jsii/_runtime.py", line 118, in call inst = super(JSIIMeta, cast(JSIIMeta, cls)).call(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/codebuild/output/src361467506/src/deploy/stacks/backend_stack.py", line 172, in init self.lambda_api_stack = LambdaApiStack( ^^^^^^^^^^^^^^^ File "/root/.pyenv/versions/3.11.6/lib/python3.11/site-packages/jsii/_runtime.py", line 118, in call inst = super(JSIIMeta, cast(JSIIMeta, cls)).call(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/codebuild/output/src361467506/src/deploy/stacks/lambda_api.py", line 94, in init api_handler_sg = self.create_lambda_sgs(envname, 'apihandler', resource_prefix, vpc) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/codebuild/output/src361467506/src/deploy/stacks/lambda_api.py", line 339, in create_lambda_sgs lambda_sg = ec2.SecurityGroup( ^^^^^^^^^^^^^^^^^^ File "/root/.pyenv/versions/3.11.6/lib/python3.11/site-packages/jsii/_runtime.py", line 118, in call inst = super(JSIIMeta, cast(JSIIMeta, cls)).call(args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.pyenv/versions/3.11.6/lib/python3.11/site-packages/aws_cdk/aws_ec2/init.py", line 78445, in init jsii.create(self.class, self, [scope, id, props]) File "/root/.pyenv/versions/3.11.6/lib/python3.11/site-packages/jsii/_kernel/init.py", line 334, in create response = self.provider.create( ^^^^^^^^^^^^^^^^^^^^^ File "/root/.pyenv/versions/3.11.6/lib/python3.11/site-packages/jsii/_kernel/providers/process.py", line 365, in create return self._process.send(request, CreateResponse) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.pyenv/versions/3.11.6/lib/python3.11/site-packages/jsii/_kernel/providers/process.py", line 342, in send raise RuntimeError(resp.error) from JavaScriptError(resp.stack) RuntimeError: There is already a Construct with name 'apihandlerSGoss' in NestedStack [Lambdas]

Subprocess exited with error 1

[Container] 2024/03/21 20:06:33.389637 Command did not exit successfully cdk synth exit status 1 [Container] 2024/03/21 20:06:33.396297 Phase complete: BUILD State: FAILED `

How to Reproduce

Merge to 78014277561d87bd2520538ecd6ee5f42b61ecf7 and deploy.

Expected behavior

cdk synth should complete successfully.

Your project

No response

Screenshots

No response

OS

n/a

Python version

n/a

AWS data.all version

2.3 plus latest commits up to 78014277561d87bd2520538ecd6ee5f42b61ecf7

Additional context

No response

noah-paige commented 7 months ago

Hi @rbernotas - the error above appears due to the same logicalId name apihandlerSGoss (more generally of the form apihandlerSG{envname}) being passed to 2 constructs in the LambdaApiStack

Looking through the code and the previous commits since v2.3 - we have not edited the logic in creating the lambda security groups for the API Handler - have you added any custom logic around here?

Nonetheless, I have tried to replicate the error by pulling the latest commits from open source (once by starting from v2.3 release and another from starting from v2.2). Both did not run into any issues in the CodePipeline Build Step and appear to be deploying normally (will update when/if deploy completes successfully)

rbernotas commented 7 months ago

We did not add any custom logic for the lambda security groups. We did note that it was complaining about the security groups and tracked it down thinking we could just delete it, but there were 4 associated security groups and a couple of network interfaces attached to it. We ended up reverting the commits and didn't have this issue with the next deploy. We'll let you know if we see this again.

noah-paige commented 7 months ago

what are the commits that you reverted? Is it only the following 2?

rbernotas commented 7 months ago

It would have been all commits Mar 8 through Mar 15 and inclusive of those dates. We also had an additional unrelated internal commit having to do with authentication that didn't have anything to do with the security groups.

noah-paige commented 5 months ago

Hi @rbernotas - I plan to close this PR since the issue is no longer being faced, but please re-open if there is any additional concerns thanks!