search

aws / aws-sam-cli

CLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
https://aws.amazon.com/serverless/sam/
Apache License 2.0
6.51k stars 1.17k forks source link

sam-beta-cdk raises an error without any additional context #2968

Closed jabalsad closed 2 years ago

jabalsad commented 3 years ago

Description:

sam-beta-cdk raises an error when trying to start-api or invoke locally. I'm hoping I don't have to post the entire CDK stack template here because it is large.

Steps to reproduce:

  1. download and install: brew install aws-sam-cli-beta-cdk (osx)
  2. run sam-beta-cdk local start-api

Observed result:

› sam-beta-cdk local start-api --debug
2021-06-17 16:35:40,390 | Determining project type...
2021-06-17 16:35:40,390 | The project is a CDK project.
2021-06-17 16:35:40,434 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-06-17 16:35:40,439 | CDK Toolkit found at ./node_modules/aws-cdk/bin/cdk
2021-06-17 16:35:40,439 | Synthesizing CDK App
2021-06-17 16:35:40,439 | command: ['./node_modules/aws-cdk/bin/cdk', 'synth', '--no-staging', '-o', '.aws-sam/.cdk-out']
2021-06-17 16:35:51,424 | Cloud assembly synthed at .aws-sam/.cdk-out
2021-06-17 16:35:51,424 | _validate_cloud_assembly: .aws-sam/.cdk-out
2021-06-17 16:35:51,431 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'c87edbda-20b4-4869-9a03-c866c6fc96f7', 'installationId': '4d5c1751-ca4e-4c25-8c05-f33973bc5430', 'sessionId': '37acf4f4-8fd2-4b40-a68e-67100f4fa95a', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.10', 'samcliVersion': '1.22.0.dev202104291816', 'metricSpecificAttributes': {'projectType': 'CDK'}, 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam-beta-cdk local start-api', 'duration': 10994, 'exitReason': 'KeyError', 'exitCode': 255}}]}
2021-06-17 16:35:51,670 | Telemetry response: 200
Traceback (most recent call last):
  File "/usr/local/bin/sam-beta-cdk", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202104291816/libexec/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202104291816/libexec/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202104291816/libexec/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202104291816/libexec/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202104291816/libexec/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202104291816/libexec/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202104291816/libexec/lib/python3.8/site-packages/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202104291816/libexec/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202104291816/libexec/lib/python3.8/site-packages/samcli/lib/telemetry/metric.py", line 174, in wrapped
    raise exception  # pylint: disable=raising-bad-type
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202104291816/libexec/lib/python3.8/site-packages/samcli/lib/telemetry/metric.py", line 141, in wrapped
    return_value = func(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202104291816/libexec/lib/python3.8/site-packages/samcli/lib/iac/utils/helpers.py", line 30, in wrapper
    project = iac_plugin.get_project(lookup_paths)
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202104291816/libexec/lib/python3.8/site-packages/samcli/lib/iac/cdk/plugin.py", line 116, in get_project
    project = self._get_project_from_cloud_assembly(cloud_assembly_dir)
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202104291816/libexec/lib/python3.8/site-packages/samcli/lib/iac/cdk/plugin.py", line 207, in _get_project_from_cloud_assembly
    stacks: List[Stack] = [self._build_stack(cloud_assembly, ca_stack) for ca_stack in cloud_assembly.stacks]
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202104291816/libexec/lib/python3.8/site-packages/samcli/lib/iac/cdk/plugin.py", line 207, in <listcomp>
    stacks: List[Stack] = [self._build_stack(cloud_assembly, ca_stack) for ca_stack in cloud_assembly.stacks]
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202104291816/libexec/lib/python3.8/site-packages/samcli/lib/iac/cdk/plugin.py", line 222, in _build_stack
    self._build_resources_section(assets, ca_stack, cloud_assembly, section, section_dict)
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202104291816/libexec/lib/python3.8/site-packages/samcli/lib/iac/cdk/plugin.py", line 280, in _build_resources_section
    asset = assets[asset_path]
KeyError: '/Users/jleroux/w/stedi/go/src/github.com/stedi/LedgerService/.aws-sam/.cdk-out/asset.92b31737b3dd47bb510f50634eaec000886b4a4c4988aa68e481619a4c928163'

Expected result:

It starts the API

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: OSX
  2. sam --version: `SAM CLI, version 1.22.0.dev202104291816``
  3. AWS region: us-east-1

Add --debug flag to command you are running

wchengru commented 3 years ago

hey @jabalsad , thanks for reporting this issue! To help us figuring out the root cause, can you please check what asset is under path {your project path}/.aws-sam/.cdk-out/asset.92b31737b3dd47bb510f50634eaec000886b4a4c4988aa68e481619a4c928163' And also, can you please run cat {your project path}/.aws-sam/.cdk-out/*.template.json | grep aws:asset:path so that we can see if the metadata in template file is correct?

Update: Can you please also share cdk.json file under the root path of your project?

jabalsad commented 3 years ago

There is a index.js file under that asset path (it's 12,000 lines so didn't want to paste here).

› cat .aws-sam/.cdk-out/*.template.json | grep aws:asset:path
        "aws:asset:path": "/Users/jleroux/w/stedi/go/src/github.com/stedi/LedgerService/.aws-sam/.cdk-out/asset.69aeff140f3d9ca35d661fd9af358cbaa3776d1b98579cafff700bd664e01176",
        "aws:asset:path": "/Users/jleroux/w/stedi/go/src/github.com/stedi/LedgerService/.aws-sam/.cdk-out/asset.69aeff140f3d9ca35d661fd9af358cbaa3776d1b98579cafff700bd664e01176",
        "aws:asset:path": "/Users/jleroux/w/stedi/go/src/github.com/stedi/LedgerService/.aws-sam/.cdk-out/asset.69aeff140f3d9ca35d661fd9af358cbaa3776d1b98579cafff700bd664e01176",
        "aws:asset:path": "/Users/jleroux/w/stedi/go/src/github.com/stedi/LedgerService/.aws-sam/.cdk-out/asset.69aeff140f3d9ca35d661fd9af358cbaa3776d1b98579cafff700bd664e01176",
        "aws:asset:path": "/Users/jleroux/w/stedi/go/src/github.com/stedi/LedgerService/.aws-sam/.cdk-out/asset.69aeff140f3d9ca35d661fd9af358cbaa3776d1b98579cafff700bd664e01176",
        "aws:asset:path": "/Users/jleroux/w/stedi/go/src/github.com/stedi/LedgerService/.aws-sam/.cdk-out/asset.92b31737b3dd47bb510f50634eaec000886b4a4c4988aa68e481619a4c928163",
        "aws:asset:path": "/Users/jleroux/w/stedi/go/src/github.com/stedi/LedgerService/node_modules/@aws-cdk/custom-resources/lib/provider-framework/runtime",
wchengru commented 3 years ago

Thanks @jabalsad ! How large is your cdk.json file? Do you mind sharing the content if it isn't too large or not containing any private content.

jabalsad commented 3 years ago

The cdk.json file is small:

{
  "app": "npx ts-node ./bin/app.ts",
  "context": {
    "@aws-cdk/core:enableStackNameDuplicates": true,
    "@aws-cdk/core:newStyleStackSynthesis": true
  }
}
jabalsad commented 3 years ago

I'm guessing you're more interested in the actual template?

straygar commented 3 years ago

Not sure why, but this workaround at least allows you to continue using sam-beta-cdk:

"@aws-cdk/core:newStyleStackSynthesis": false
grifhammeramazon commented 3 years ago

I have run into this same issue when trying to run sam-beta-cdk build --cdk-app "npx ts-node bin/test.ts".

pseudo-su commented 3 years ago

I've also run into this issue when trying to use sam-beta-cdk to run my golang serverless application locally

The branch where I'm trying to convert to using the CDK on the my project is here.

https://github.com/pseudo-su/golang-serverless-cdk-template/pull/1

Using newStyleStackSynthesis as suggested to bypass the error has had no affect for me.

Versions

$ sam-beta-cdk --version
> SAM CLI, version 1.22.0.dev202107140310
$ cdk --version
> 1.115.0 (build f0ca40f)

cdk.json

{
  "app": "npx ts-node --project cdk/tsconfig.json --prefer-ts-exts cdk/app.ts",
  "context": {
    "@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": true,
    "@aws-cdk/core:enableStackNameDuplicates": "true",
    "aws-cdk:enableDiffNoFail": "true",
    "@aws-cdk/core:stackRelativeExports": "true",
    "@aws-cdk/aws-ecr-assets:dockerIgnoreSupport": true,
    "@aws-cdk/aws-secretsmanager:parseOwnedSecretName": true,
    "@aws-cdk/aws-kms:defaultKeyPolicies": true,
    "@aws-cdk/aws-s3:grantWriteWithoutAcl": true,
    "@aws-cdk/aws-ecs-patterns:removeDefaultDesiredCount": true,
    "@aws-cdk/aws-rds:lowercaseDbIdentifier": true,
    "@aws-cdk/aws-efs:defaultEncryptionAtRest": true,
    "@aws-cdk/aws-lambda:recognizeVersionProps": true
  }
}

Error log

Synthesizing CDK App
Traceback (most recent call last):
  File "/usr/local/bin/sam-beta-cdk", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202107140310/libexec/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202107140310/libexec/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202107140310/libexec/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202107140310/libexec/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202107140310/libexec/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202107140310/libexec/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202107140310/libexec/lib/python3.8/site-packages/samcli/lib/iac/utils/helpers.py", line 50, in wrapper
    iac_plugin, project = get_iac_plugin(project_type, kwargs, with_build)
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202107140310/libexec/lib/python3.8/site-packages/samcli/lib/iac/utils/helpers.py", line 40, in get_iac_plugin
    project = iac_plugin.get_project(lookup_paths)
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202107140310/libexec/lib/python3.8/site-packages/samcli/lib/iac/cdk/plugin.py", line 114, in get_project
    project = self._get_project_from_cloud_assembly(cloud_assembly_dir)
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202107140310/libexec/lib/python3.8/site-packages/samcli/lib/iac/cdk/plugin.py", line 211, in _get_project_from_cloud_assembly
    stacks: List[Stack] = [self._build_stack(cloud_assembly, ca_stack) for ca_stack in cloud_assembly.stacks]
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202107140310/libexec/lib/python3.8/site-packages/samcli/lib/iac/cdk/plugin.py", line 211, in <listcomp>
    stacks: List[Stack] = [self._build_stack(cloud_assembly, ca_stack) for ca_stack in cloud_assembly.stacks]
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202107140310/libexec/lib/python3.8/site-packages/samcli/lib/iac/cdk/plugin.py", line 232, in _build_stack
    self._build_resources_section(assets, ca_stack, cloud_assembly, section, section_dict)
  File "/usr/local/Cellar/aws-sam-cli-beta-cdk/202107140310/libexec/lib/python3.8/site-packages/samcli/lib/iac/cdk/plugin.py", line 292, in _build_resources_section
    asset = assets[asset_path]
KeyError: '/Development/golang-serverless-cdk-template/.aws-sam/.cdk-out/asset.bdcfdea2d98c16355538f9f634bcd57355fd4b78dc36661da93c6546fa77bcd6'
make: *** [dev] Error 1

Asset files

output of cat .aws-sam/.cdk-out/*.template.json | grep aws:asset:path

"aws:asset:path": "/Users/stableford/Development/Projects/Personal/golang-serverless-cdk-template/.aws-sam/.cdk-out/asset.bdcfdea2d98c16355538f9f634bcd57355fd4b78dc36661da93c6546fa77bcd6",
"aws:asset:path": "/Users/stableford/Development/Projects/Personal/golang-serverless-cdk-template/.aws-sam/.cdk-out/asset.11b1fbd97647ec3b4f3f2f3fd5c7fa53f0386bfd5cc831be00cb829546b5e262",
"aws:asset:path": "/Users/stableford/Development/Projects/Personal/golang-serverless-cdk-template/.aws-sam/.cdk-out/asset.39d9372624a85803a5161b014625dc5a138ff0817ffa2c1213cbeb27fb547a26",
"aws:asset:path": "/Users/stableford/Development/Projects/Personal/golang-serverless-cdk-template/.aws-sam/.cdk-out/asset.39d9372624a85803a5161b014625dc5a138ff0817ffa2c1213cbeb27fb547a26",
"aws:asset:path": "/Users/stableford/Development/Projects/Personal/golang-serverless-cdk-template/.aws-sam/.cdk-out/asset.25d701c2234f6c2a11e2c0592dc76db6d7621fb883c1e9fbe742072fd542e46a",
"aws:asset:path": "/Users/stableford/Development/Projects/Personal/golang-serverless-cdk-template/.aws-sam/.cdk-out/asset.699b4676864820810a21ba9e48842d0cade14020fda9a0e0339612d56da7ab30",
"aws:asset:path": "/Users/stableford/Development/Projects/Personal/golang-serverless-cdk-template/.aws-sam/.cdk-out/asset.eb606aa355165ed613af9b46f6961758add994cf2d06f6557d6d4fc583a0924f",
"aws:asset:path": "/Users/stableford/Development/Projects/Personal/golang-serverless-cdk-template/.aws-sam/.cdk-out/asset.11b1fbd97647ec3b4f3f2f3fd5c7fa53f0386bfd5cc831be00cb829546b5e262",
Screen Shot 2021-07-25 at 11 26 15 am
moelasmar commented 2 years ago

SAM CLI's CDK support went into GA yesterday with some changes. Now sam-cli won't run cdk but user will have to run and provide the path. This should solve this problem. Please let us know if you have more questions. Please find more details here