search

aws / serverless-application-model

The AWS Serverless Application Model (AWS SAM) transform is a AWS CloudFormation macro that transforms SAM templates into CloudFormation templates.
https://aws.amazon.com/serverless/sam
Apache License 2.0
9.36k stars 2.38k forks source link

Parameters not supported for SemanticVersion in AWS::Serverless::Application resource #1094

Closed mixja closed 5 years ago

mixja commented 5 years ago

Description

When you attempt to reference a parameter for the SemanticVersion property of the Location object, SAM processing fails with an internal transform failure.

Note as per https://github.com/awslabs/serverless-application-model/issues/897, support for this has been recently added to SAM so I expect SAM CLI just needs to be updated to support this change.

Steps to reproduce

Attempt to run sam build for following template:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Some Template
Parameters:
  SecretsProvisionerApplicationVersion:
    Description: Secrets Provisioner Version
    Type: String
    Default: 0.1.0
Resources:
  SecretsProvisionerApplication:
    Type: AWS::Serverless::Application
    Properties:
      Location:
        ApplicationId: arn:aws:serverlessrepo:ap-southeast-2:012345678901:applications/secrets-provisioner
        SemanticVersion: !Ref SecretsProvisionerApplicationVersion

Observed result

sam build --debug fails with the following:

$ sam build --debug
2019-04-27 23:17:52 Using SAM Template at /Users/jmenga/Source/yellownz/cloudformation-resources/template.yaml
2019-04-27 23:17:52 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-04-27 23:17:52 Changing event name from before-call.apigateway to before-call.api-gateway
2019-04-27 23:17:52 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-04-27 23:17:52 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-04-27 23:17:52 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-04-27 23:17:52 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-04-27 23:17:52 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-04-27 23:17:52 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-04-27 23:17:52 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-04-27 23:17:52 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-04-27 23:17:52 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-04-27 23:17:52 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-04-27 23:17:52 Changing event name from before-call.apigateway to before-call.api-gateway
2019-04-27 23:17:52 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-04-27 23:17:52 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-04-27 23:17:52 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-04-27 23:17:52 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-04-27 23:17:52 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-04-27 23:17:52 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-04-27 23:17:52 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-04-27 23:17:52 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-04-27 23:17:52 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-04-27 23:17:52 'build' command is called
2019-04-27 23:17:52 Plugin 'ServerlessAppPlugin' raised an exception: unhashable type: 'dict'
Traceback (most recent call last):
  File "/Users/jmenga/Library/Python/3.7/lib/python/site-packages/samtranslator/plugins/__init__.py", line 130, in act
    getattr(plugin, method_name)(*args, **kwargs)
  File "/Users/jmenga/Library/Python/3.7/lib/python/site-packages/samtranslator/plugins/application/serverless_app_plugin.py", line 85, in on_before_transform_template
    if key not in self._applications:
TypeError: unhashable type: 'dict'
Traceback (most recent call last):
  File "/Users/jmenga/Library/Python/3.7/bin/sam", line 10, in <module>
    sys.exit(cli())
  File "/Users/jmenga/Library/Python/3.7/lib/python/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/jmenga/Library/Python/3.7/lib/python/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/jmenga/Library/Python/3.7/lib/python/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/jmenga/Library/Python/3.7/lib/python/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/jmenga/Library/Python/3.7/lib/python/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/jmenga/Library/Python/3.7/lib/python/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/Users/jmenga/Library/Python/3.7/lib/python/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/jmenga/Library/Python/3.7/lib/python/site-packages/samcli/commands/build/command.py", line 100, in cli
    skip_pull_image, parameter_overrides, mode)  # pragma: no cover
  File "/Users/jmenga/Library/Python/3.7/lib/python/site-packages/samcli/commands/build/command.py", line 131, in do_cli
    mode=mode) as ctx:
  File "/Users/jmenga/Library/Python/3.7/lib/python/site-packages/samcli/commands/build/build_context.py", line 59, in __enter__
    self._function_provider = SamFunctionProvider(self._template_dict, self._parameter_overrides)
  File "/Users/jmenga/Library/Python/3.7/lib/python/site-packages/samcli/commands/local/lib/sam_function_provider.py", line 45, in __init__
    self.template_dict = SamBaseProvider.get_template(template_dict, parameter_overrides)
  File "/Users/jmenga/Library/Python/3.7/lib/python/site-packages/samcli/commands/local/lib/sam_base_provider.py", line 62, in get_template
    template_dict = SamTranslatorWrapper(template_dict).run_plugins()
  File "/Users/jmenga/Library/Python/3.7/lib/python/site-packages/samcli/lib/samlib/wrapper.py", line 67, in run_plugins
    parser.parse(template_copy, all_plugins)  # parse() will run all configured plugins
  File "/Users/jmenga/Library/Python/3.7/lib/python/site-packages/samcli/lib/samlib/wrapper.py", line 120, in parse
    sam_plugins.act(LifeCycleEvents.before_transform_template, sam_template)
  File "/Users/jmenga/Library/Python/3.7/lib/python/site-packages/samtranslator/plugins/__init__.py", line 136, in act
    raise ex
  File "/Users/jmenga/Library/Python/3.7/lib/python/site-packages/samtranslator/plugins/__init__.py", line 130, in act
    getattr(plugin, method_name)(*args, **kwargs)
  File "/Users/jmenga/Library/Python/3.7/lib/python/site-packages/samtranslator/plugins/application/serverless_app_plugin.py", line 85, in on_before_transform_template
    if key not in self._applications:
TypeError: unhashable type: 'dict'

Expected result

Template should be processed successfully.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: Mac
  2. sam --version: SAM CLI, version 0.15.0
jfuss commented 5 years ago

@mixja Thanks for the report. Looks like https://github.com/awslabs/serverless-application-model/pull/856 broke the integration. Specifically, the prepare_plugins added an optional value of parameters and defaults to an empty dictionary. We have a wrapper that ends up calling this method: https://github.com/awslabs/aws-sam-cli/blob/develop/samcli/commands/local/lib/sam_base_provider.py#L62 which calls https://github.com/awslabs/aws-sam-cli/blob/develop/samcli/lib/samlib/wrapper.py#L64.

We will need to change how we run plugins now.

As a side note: Why are you running build in this case? The template you provided does not have any functions in it?

mixja commented 5 years ago

Thanks - it would affect SAM templates that have both functions and other Serverless applications - I just left out other resources to focus on the actual issue.

gojko commented 5 years ago

Same error seems to be happening when using parameter substitution in ApplicationId, so the following block causes it to explode (even when the parameter is properly set up)

  ImageMagick:
    Type: AWS::Serverless::Application
    Properties:
      Location:
        ApplicationId: !Sub "arn:aws:serverlessrepo:us-east-1:${ImageMagickLayerAppId}"
        SemanticVersion: 1.0.0

this works OK:

  ImageMagick:
    Type: AWS::Serverless::Application
    Properties:
      Location:
        ApplicationId: arn:aws:serverlessrepo:us-east-1:145266761615:applications/image-magick-lambda-layer
        SemanticVersion: 1.0.0

here's some system info if it's helpful

$ sam --version
SAM CLI, version 0.16.1
$ aws --version
aws-cli/1.16.162 Python/3.7.1 Darwin/18.5.0 botocore/1.12.152

actual stack trace

2019-05-21 00:22:20 Plugin 'ServerlessAppPlugin' raised an exception: unhashable type: 'dict'
Traceback (most recent call last):
  File "/project/pythonenv/lib/python3.7/site-packages/samtranslator/plugins/__init__.py", line 130, in act
    getattr(plugin, method_name)(*args, **kwargs)
  File "/project/pythonenv/lib/python3.7/site-packages/samtranslator/plugins/application/serverless_app_plugin.py", line 85, in on_before_transform_template
    if key not in self._applications:
TypeError: unhashable type: 'dict'
Traceback (most recent call last):
  File "/project/pythonenv/bin/sam", line 10, in <module>
    sys.exit(cli())
  File "/project/pythonenv/lib/python3.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/project/pythonenv/lib/python3.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/project/pythonenv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/project/pythonenv/lib/python3.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/project/pythonenv/lib/python3.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/project/pythonenv/lib/python3.7/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/project/pythonenv/lib/python3.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/project/pythonenv/lib/python3.7/site-packages/samcli/commands/build/command.py", line 103, in cli
    skip_pull_image, parameter_overrides, mode)  # pragma: no cover
  File "/project/pythonenv/lib/python3.7/site-packages/samcli/commands/build/command.py", line 136, in do_cli
    mode=mode) as ctx:
  File "/project/pythonenv/lib/python3.7/site-packages/samcli/commands/build/build_context.py", line 65, in __enter__
    self._function_provider = SamFunctionProvider(self._template_dict, self._parameter_overrides)
  File "/project/pythonenv/lib/python3.7/site-packages/samcli/commands/local/lib/sam_function_provider.py", line 46, in __init__
    self.template_dict = SamBaseProvider.get_template(template_dict, parameter_overrides)
  File "/project/pythonenv/lib/python3.7/site-packages/samcli/commands/local/lib/sam_base_provider.py", line 62, in get_template
    template_dict = SamTranslatorWrapper(template_dict).run_plugins()
  File "/project/pythonenv/lib/python3.7/site-packages/samcli/lib/samlib/wrapper.py", line 67, in run_plugins
    parser.parse(template_copy, all_plugins)  # parse() will run all configured plugins
  File "/project/pythonenv/lib/python3.7/site-packages/samcli/lib/samlib/wrapper.py", line 120, in parse
    sam_plugins.act(LifeCycleEvents.before_transform_template, sam_template)
  File "/project/pythonenv/lib/python3.7/site-packages/samtranslator/plugins/__init__.py", line 136, in act
    raise ex
  File "/project/pythonenv/lib/python3.7/site-packages/samtranslator/plugins/__init__.py", line 130, in act
    getattr(plugin, method_name)(*args, **kwargs)
  File "/project/pythonenv/lib/python3.7/site-packages/samtranslator/plugins/application/serverless_app_plugin.py", line 85, in on_before_transform_template
    if key not in self._applications:
TypeError: unhashable type: 'dict'
make: *** [deploy] Error 1
praneetap commented 5 years ago

This should be fixed as per https://github.com/awslabs/serverless-application-model/issues/897#issuecomment-487137339. The next sam-cli release v0.20.0 upgrades to the latest sam version and should address this issue.