search

aws / aws-sam-cli

CLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
https://aws.amazon.com/serverless/sam/
Apache License 2.0
6.47k stars 1.16k forks source link

Bug: Using YAML anchors for Resource Metadata makes resource disappear due to template mutation #5901

Closed davidjb closed 9 months ago

davidjb commented 10 months ago

Description:

I have a large number of Go Lambda functions defined in my template.yaml and I was looking to simply my SAM template and make it more DRY by using anchors on their Metadata property like so:

Resources:
  FirstFunction:
    Type: AWS::Serverless::Function
    Metadata: &GoFunctionMetadata
      BuildMethod: go1.x
      BuildProperties:
        TrimGoPath: true
    [...]
  SecondFunction:
    Metadata: *GoFunctionMetadata
  [...]

However, when doing this, the given function becomes no longer available to SAM (see below).

The reasoning because SAM adds the SamResourceId key within a resource's metadata (https://github.com/aws/aws-sam-cli/blob/develop/samcli/lib/samlib/resource_metadata_normalizer.py#L95-L99) and this gets used to create a list of available resources to build (https://github.com/aws/aws-sam-cli/blob/develop/samcli/commands/build/build_context.py#L565-L568). The problem occurs because the underlying OrderedDict structure is shared between functions, then the shared object gets mutated by the the normalizer when setting its properties.

This same issue can/will occur anywhere else the template structure where the template is being mutated so prior making changes, the original data structure should be copied.

Steps to reproduce:

  1. Create template.yaml like so:
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31

Resources:
  FirstFunction:
    Type: AWS::Serverless::Function
    Metadata: &GoFunctionMetadata
      BuildMethod: go1.x
      BuildProperties:
        TrimGoPath: true
    Properties:
      CodeUri: dummy
      Handler: bootstrap
      Runtime: provided.al2
  SecondFunction:
    Type: AWS::Serverless::Function
    Metadata: *GoFunctionMetadata
    Properties:
      CodeUri: dummy
      Handler: bootstrap
      Runtime: provided.al2
  ThirdFunction:
    Type: AWS::Serverless::Function
    Metadata: *GoFunctionMetadata
    Properties:
      CodeUri: dummy
      Handler: bootstrap
      Runtime: provided.al2
  1. Run sam build SecondFunction --debug

Observed result:

$ sam build SecondFunction --debug
2023-09-07 17:39:26,871 | No config file found in this directory.
2023-09-07 17:39:26,872 | OSError occurred while reading TOML file: [Errno 2] No such file or directory: '/private/tmp/foo/samconfig.toml'
2023-09-07 17:39:26,873 | Config file location: /private/tmp/foo/samconfig.toml
2023-09-07 17:39:26,873 | Config file '/private/tmp/foo/samconfig.toml' does not exist
2023-09-07 17:39:26,875 | Using SAM Template at /private/tmp/foo/template.yaml
2023-09-07 17:39:26,891 | Using config file: samconfig.toml, config environment: default
2023-09-07 17:39:26,891 | Expand command line arguments to:
2023-09-07 17:39:26,892 | --template_file=/private/tmp/foo/template.yaml --resource_logical_id=SecondFunction --mount_with=READ --build_dir=.aws-sam/build
--cache_dir=.aws-sam/cache
2023-09-07 17:39:26,924 | 'build' command is called
2023-09-07 17:39:26,926 | No Parameters detected in the template
2023-09-07 17:39:26,934 | There is no customer defined id or cdk path defined for resource FirstFunction, so we will use the resource logical id as the resource id
2023-09-07 17:39:26,934 | Sam customer defined id is more priority than other IDs. Customer defined id for resource SecondFunction is FirstFunction
2023-09-07 17:39:26,934 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ThirdFunction is FirstFunction
2023-09-07 17:39:26,935 | 0 stacks found in the template
2023-09-07 17:39:26,935 | No Parameters detected in the template
2023-09-07 17:39:26,941 | There is no customer defined id or cdk path defined for resource FirstFunction, so we will use the resource logical id as the resource id
2023-09-07 17:39:26,941 | Sam customer defined id is more priority than other IDs. Customer defined id for resource SecondFunction is FirstFunction
2023-09-07 17:39:26,941 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ThirdFunction is FirstFunction
2023-09-07 17:39:26,942 | 3 resources found in the stack
2023-09-07 17:39:26,942 | Found Serverless function with name='FirstFunction' and CodeUri='dummy'
2023-09-07 17:39:26,942 | --base-dir is not presented, adjusting uri dummy relative to /private/tmp/foo/template.yaml
2023-09-07 17:39:26,943 | Found Serverless function with name='SecondFunction' and CodeUri='dummy'
2023-09-07 17:39:26,943 | --base-dir is not presented, adjusting uri dummy relative to /private/tmp/foo/template.yaml
2023-09-07 17:39:26,943 | Found Serverless function with name='ThirdFunction' and CodeUri='dummy'
2023-09-07 17:39:26,943 | --base-dir is not presented, adjusting uri dummy relative to /private/tmp/foo/template.yaml
2023-09-07 17:39:26,945 | 3 resources found in the stack
2023-09-07 17:39:26,945 | Found Serverless function with name='FirstFunction' and CodeUri='dummy'
2023-09-07 17:39:26,946 | Found Serverless function with name='SecondFunction' and CodeUri='dummy'
2023-09-07 17:39:26,946 | Found Serverless function with name='ThirdFunction' and CodeUri='dummy'
2023-09-07 17:39:26,946 | SecondFunction not found. Possible options in your template: ['ThirdFunction']

Build Failed
2023-09-07 17:39:26,947 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-09-07 17:39:26,947 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-09-07 17:39:26,948 | Unable to find Click Context for getting session_id.
Error: Unable to find a function or layer with name 'SecondFunction'

Expected result:

All functions to be available and buildable.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

{
  "version": "1.96.0",
  "system": {
    "python": "3.11.5",
    "os": "macOS-13.5.1-arm64-arm-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "24.0.5",
    "aws_cdk": "Not available",
    "terraform": "1.1.0"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]

Add --debug flag to command you are running

mndeveci commented 10 months ago

Thanks for reporting this issue.

As you mentioned, it is happening inside the resource metadata normalizer. But it is also related the way that you reference some values in the template. The python representation of your template also references those values, so changing one of the metadata actually applying that change to all the others. For that reason, all resources' metadata got updated if one of them changed.

I will be labeling this as bug and talking with the team for next steps.

github-actions[bot] commented 9 months ago

Patch is released in v1.98.0. Closing