search

aws / aws-sam-cli

CLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
https://aws.amazon.com/serverless/sam/
Apache License 2.0
6.52k stars 1.17k forks source link

Bug: Parameters are permitted that don't satisfy constraints (e.g. type, `AllowedValues`, `AllowedPattern`, length, value, etc) #6511

Open davidjb opened 10 months ago

davidjb commented 10 months ago

Description:

As described in https://github.com/aws/aws-sam-cli/issues/6110, SAM doesn't attempt to validate values for Parameters according to their AllowedValues.

CloudFormation will validate values, but only during the deploy process (e.g. once a changeset is running). This means a lot can occur before this point with invalid values: with an invalid value, you could still end up with broken, successful builds, attempts to download invalid containers or files, or failures anywhere else in the process, since a Parameter can be used anywhere.

Steps to reproduce:

  1. Create template.yml:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

# These are nonsense but prove that validation isn't occurring within SAM
Parameters:
  MyParam:
    Default: C
    AllowedValues: [A, B, C, D]
    MinLength: 1
    MaxLength: 2
    AllowedPattern: C

  NumParam:
    Type: Number
    MinValue: 1
    MaxValue: 10

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: !Sub hello_world/${MyParam}-${NumParam}
      Handler: app.lambda_handler
      Runtime: python3.11

  1. Run sam build --parameter-overrides MyParam="fake" NumParam=11 and observe results, as per next section below. SAM attempts to find hello_world/fake-11, despite the value of "fake" not being in the AllowedValues list and outside of the length bounds, and NumParm being outside of the allowed range.

  2. Likewise, running with NumParam="fake" is permitted, despite this value being a different type (a string and not a number). Validation also doesn't occur for any other parameter Type, such as specifying a,b,c for the value of a List<Number>.

Observed result:

sam build --parameter-overrides MyParam="fake" errors, trying to find the CodeUri of helloworld/fake, as it has templated in the unverified MyParam value:

2024-01-04 17:01:06,667 | No config file found in this directory.
2024-01-04 17:01:06,667 | OSError occurred while reading TOML file: [Errno 2] No such file or directory: '/tmp/sam-app/sam-app/samconfig.toml'
2024-01-04 17:01:06,667 | Config file location: /tmp/sam-app/sam-app/samconfig.toml
2024-01-04 17:01:06,668 | Config file '/tmp/sam-app/sam-app/samconfig.toml' does not exist
2024-01-04 17:01:06,669 | Using SAM Template at /tmp/sam-app/sam-app/template.yaml
2024-01-04 17:01:06,683 | OSError occurred while reading TOML file: [Errno 2] No such file or directory: '/tmp/sam-app/sam-app/samconfig.toml'
2024-01-04 17:01:06,683 | Using config file: samconfig.toml, config environment: default
2024-01-04 17:01:06,683 | Expand command line arguments to:
2024-01-04 17:01:06,683 | --template_file=/tmp/sam-app/sam-app/template.yaml --parameter_overrides={'MyParam': 'fake', 'NumParam': 'fake'} --mount_with=READ --build_dir=.aws-sam/build --cache_dir=.aws-sam/cache 
2024-01-04 17:01:06,700 | 'build' command is called
2024-01-04 17:01:06,701 | Collected default values for parameters: {'MyParam': 'C'}
2024-01-04 17:01:06,710 | There is no customer defined id or cdk path defined for resource HelloWorldFunction, so we will use the resource logical id as the resource id
2024-01-04 17:01:06,710 | 0 stacks found in the template
2024-01-04 17:01:06,710 | Collected default values for parameters: {'MyParam': 'C'}
2024-01-04 17:01:06,717 | There is no customer defined id or cdk path defined for resource HelloWorldFunction, so we will use the resource logical id as the resource id
2024-01-04 17:01:06,717 | 1 resources found in the stack 
2024-01-04 17:01:06,717 | Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/fake-fake'
2024-01-04 17:01:06,717 | --base-dir is not presented, adjusting uri hello_world/fake-fake relative to /tmp/sam-app/sam-app/template.yaml
2024-01-04 17:01:06,718 | 1 resources found in the stack 
2024-01-04 17:01:06,718 | Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/fake-fake'
2024-01-04 17:01:06,718 | Instantiating build definitions
2024-01-04 17:01:06,719 | Same function build definition found, adding function (Previous: BuildDefinition(python3.11, /tmp/sam-app/sam-app/hello_world/fake-fake, Zip, , 129f5541-532f-4daa-993f-280f24eadddd, {}, {}, x86_64, []), Current: BuildDefinition(python3.11, /tmp/sam-app/sam-app/hello_world/fake-fake, Zip, , 1a2c0420-a72a-4f6b-8b2e-5ceba1dd02f4, {}, {}, x86_64, []), Function: Function(function_id='HelloWorldFunction', name='HelloWorldFunction', functionname='HelloWorldFunction', runtime='python3.11', memory=None, timeout=None, handler='app.lambda_handler', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/tmp/sam-app/sam-app/hello_world/fake-fake', environment=None, rolearn=None, layers=[], events=None, metadata={'SamResourceId': 'HelloWorldFunction'}, inlinecode=None, codesign_config_arn=None, architectures=None, function_url_config=None, function_build_info=<FunctionBuildInfo.BuildableZip: ('BuildableZip', 'Regular ZIP function which can be build with SAM CLI')>, stack_path='', runtime_management_config=None, logging_config=None))
2024-01-04 17:01:06,719 | Building codeuri: /tmp/sam-app/sam-app/hello_world/fake-fake runtime: python3.11 metadata: {} architecture: x86_64 functions: HelloWorldFunction
2024-01-04 17:01:06,720 | Building to following folder /tmp/sam-app/sam-app/.aws-sam/build/HelloWorldFunction
2024-01-04 17:01:06,720 | Loading workflow module 'aws_lambda_builders.workflows'
2024-01-04 17:01:06,721 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2024-01-04 17:01:06,721 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2024-01-04 17:01:06,722 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2024-01-04 17:01:06,723 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2024-01-04 17:01:06,724 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2024-01-04 17:01:06,724 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2024-01-04 17:01:06,725 | Registering workflow 'NodejsNpmEsbuildBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm-esbuild', application_framework=None)'
2024-01-04 17:01:06,726 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2024-01-04 17:01:06,726 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2024-01-04 17:01:06,727 | Registering workflow 'RustCargoLambdaBuilder' with capability 'Capability(language='rust', dependency_manager='cargo', application_framework=None)'
2024-01-04 17:01:06,727 | Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2024-01-04 17:01:06,727 | requirements.txt file not found. Continuing the build without dependencies.
2024-01-04 17:01:06,747 | Running workflow 'PythonPipBuilder'
2024-01-04 17:01:06,747 |  Running PythonPipBuilder:CopySource
2024-01-04 17:01:06,747 | Skipping copy operation since source /tmp/sam-app/sam-app/hello_world/fake-fake does not exist
2024-01-04 17:01:06,747 | PythonPipBuilder:CopySource succeeded
2024-01-04 17:01:06,747 | There is no customer defined id or cdk path defined for resource HelloWorldFunction, so we will use the resource logical id as the resource id
2024-01-04 17:01:06,747 | 1 resources found in the stack 
2024-01-04 17:01:06,747 | Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/fake-fake'

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided
2024-01-04 17:01:06,749 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-01-04 17:01:06,749 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-01-04 17:01:06,749 | Unable to find Click Context for getting session_id.

Expected result:

Validation errors would be shown when the SAM config is loaded for parameters that are outside the conditions, before the values are used for building. Builds would not succeed when invalid values are used and deploys would not begin.

As per https://github.com/aws/aws-sam-cli/issues/3244, these values will be validated but only during deployment by CloudFormation, assuming you're able to create a configuration that is able to be built successfully (e.g. suitable parameter values chosen depending on their use in template.yaml).

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

{
  "version": "1.105.0",
  "system": {
    "python": "3.11.6",
    "os": "Linux-6.4.16-linuxkit-aarch64-with-glibc2.35"
  },
  "additional_dependencies": {
    "docker_engine": "24.0.6",
    "aws_cdk": "Not available",
    "terraform": "Not available"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}
mildaniel commented 10 months ago

Thanks again @davidjb. My personal view on this is that it's not feasible to do every validation that CFN does, so we should do the validations that correspond to our local emulation. From your examples, it seems there are some valid use cases for early validation. I will discuss it with the team for prioritization.

davidjb commented 10 months ago

Thanks @mildaniel for the consideration, I really appreciate it.

From my experience, using parameters without validating means unexpected results and a range of impacts. The impact may be minor/negligible if a parameter was used in, say, a description or comment, but when used for operational values (architectures, paths, env variables etc - as is in my production config), mistakes or missed parameters can subtly creep in and be hard to detect.

As Parameters are (can be) used by SAM in builds or locally (and passed as a CLI flag or in config), I was expecting they’d be validated as SAM’s other options/flags are.