search

aws / aws-sam-cli

CLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
https://aws.amazon.com/serverless/sam/
Apache License 2.0
6.53k stars 1.17k forks source link

Feature request: Support LanguageExtensions feature Fn::ForEach #5647

Open twasink opened 1 year ago

twasink commented 1 year ago

It would be useful if the LanguageExtensions transform feature Fn::ForEach was supported with SAM CLI.

The documentation for the LanguageExtensions transforms states that you can use it with the Serverless transform:

If you're using multiple transforms, use a list format. If you're using custom macros, place AWS-provided transforms after your custom macros. If you're using both the AWS::LanguageExtensions and AWS::Serverless transforms, the AWS::LanguageExtensions transform must come before the AWS::Serverless transform in the list.

However, when you try to use the example for the Fn::ForEach function (with or without include the Serverless Transform), the sam-cli fails to understand the syntax.

Example template.yaml:

AWSTemplateFormatVersion: 2010-09-09
Transform: 
  - - AWS::LanguageExtensions
    - AWS::Serverless-2016-10-31
Resources:
  'Fn::ForEach::Topics':
    - TopicName
    - - Success
      - Failure
      - Timeout
      - Unknown
    - 'SnsTopic${TopicName}':
        Type: 'AWS::SNS::Topic'
        Properties:
          TopicName: !Ref TopicName
          FifoTopic: true

Example output from sam build:

$ sam build
2023-07-30 17:52:37 Plugin 'ServerlessAppPlugin' raised an exception: 'list' object has no attribute 'get'
Traceback (most recent call last):
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samtranslator/plugins/sam_plugins.py", line 130, in act
    getattr(plugin, method_name)(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samtranslator/metrics/method_decorator.py", line 116, in wrapper_cw_timer
    exec_result = func(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samtranslator/plugins/application/serverless_app_plugin.py", line 124, in on_before_transform_template
    for logical_id, app in template.iterate({SamResourceType.Application.value}):
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samtranslator/sdk/template.py", line 34, in iterate
    resource = SamResource(resource_dict)
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samtranslator/sdk/resource.py", line 26, in __init__
    self.type = resource_dict.get("Type")
AttributeError: 'list' object has no attribute 'get'

Error: 'list' object has no attribute 'get'
Traceback:
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samcli/lib/telemetry/metric.py", line 184, in wrapped
    raise exception  # pylint: disable=raising-bad-type
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samcli/lib/telemetry/metric.py", line 149, in wrapped
    return_value = func(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samcli/lib/utils/version_checker.py", line 42, in wrapped
    actual_result = func(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samcli/cli/main.py", line 95, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samcli/commands/build/command.py", line 166, in cli
    do_cli(
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samcli/commands/build/command.py", line 238, in do_cli
    with BuildContext(
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samcli/commands/build/build_context.py", line 184, in __enter__
    self.set_up()
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samcli/commands/build/build_context.py", line 190, in set_up
    self._stacks, remote_stack_full_paths = SamLocalStackProvider.get_stacks(
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samcli/lib/providers/sam_stack_provider.py", line 269, in get_stacks
    current = SamLocalStackProvider(
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samcli/lib/providers/sam_stack_provider.py", line 61, in __init__
    self._template_dict = self.get_template(
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samcli/lib/providers/sam_base_provider.py", line 193, in get_template
    template_dict = SamTranslatorWrapper(template_dict, parameter_values=parameters_values).run_plugins()
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samcli/lib/samlib/wrapper.py", line 73, in run_plugins
    parser.parse(template_copy, all_plugins)  # parse() will run all configured plugins
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samcli/lib/samlib/wrapper.py", line 130, in parse
    sam_plugins.act(LifeCycleEvents.before_transform_template, sam_template)
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samtranslator/plugins/sam_plugins.py", line 136, in act
    raise ex
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samtranslator/plugins/sam_plugins.py", line 130, in act
    getattr(plugin, method_name)(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samtranslator/metrics/method_decorator.py", line 116, in wrapper_cw_timer
    exec_result = func(*args, **kwargs)
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samtranslator/plugins/application/serverless_app_plugin.py", line 124, in on_before_transform_template
    for logical_id, app in template.iterate({SamResourceType.Application.value}):
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samtranslator/sdk/template.py", line 34, in iterate
    resource = SamResource(resource_dict)
  File "/usr/local/Cellar/aws-sam-cli/1.94.0/libexec/lib/python3.8/site-packages/samtranslator/sdk/resource.py", line 26, in __init__
    self.type = resource_dict.get("Type")

An unexpected error was encountered while executing "sam build".
Search for an existing issue:
https://github.com/aws/aws-sam-cli/issues?q=is%3Aissue+is%3Aopen+Bug%3A%20sam%20build%20-%20AttributeError
Or create a bug report:
https://github.com/aws/aws-sam-cli/issues/new?template=Bug_report.md&title=Bug%3A%20sam%20build%20-%20AttributeError

(This was with SAM CLI version 1.94)

moelasmar commented 1 year ago

Thanks @twasink for raising this issue. I will discuss this issue with the team to see how can we prioritize it.

cortexcompiler commented 1 year ago

I am also running into this. Would be nice to use Fn::ForEach to do some things similar to SAM serverless resources. I was hoping to add a group of AWS::Serverless::Function resources along with essentially templated LogGroup SubscriptionFilter resources for each of the functions. However, I immediately ran into this error trying to run sam build

cortexcompiler commented 1 year ago

Also, is this really a feature request? I would expect SAM to support native CloudFormation functions. I feel like this is actually a bug.

AEkman commented 1 year ago

I can confirm that I have seen the same issue. When SAM is validating I get the following error: Error: 'list' object has no attribute 'get'

AEkman commented 1 year ago

It would also be good to mention that it's currently not supported in documentation!

ggiallo28 commented 1 year ago

I am running the same issue with "aws cloudformation package"

maciejGolebio commented 1 year ago

image

ryanniuu commented 1 year ago

same issue when running "sam validate" i would say this is a bug not feature request

nelsonmora48 commented 1 year ago

Ya vamos por la 1.98 y aun no se corrige este problema

francisreyes-tfs commented 1 year ago

Pull request 8096 from aws-cli looks promising but needs review

rhbecker commented 10 months ago

PR 8096 (aws-cli) was merged in mid-December. A couple of comments on related issues here and here indicate that recent aws-cli versions no longer have this problem.

Is the aws-cli "upstream" relative to sam-cli, meaning that we'll inevitably see this same fix in an imminent sam-cli releaes? Or does the sam-cli team need to independently address this issue?

Regardless, I'm hoping this might be near the finish line, so that I don't need to train our staff on 2 different stack management tools. 🤞

mlmland commented 10 months ago

I'm having this issue as well. Any updates?

michal-sa commented 10 months ago

PR 8096 (aws-cli) was merged in mid-December. A couple of comments on related issues here and here indicate that recent aws-cli versions no longer have this problem.

Is the aws-cli "upstream" relative to sam-cli, meaning that we'll inevitably see this same fix in an imminent sam-cli releaes? Or does the sam-cli team need to independently address this issue?

Regardless, I'm hoping this might be near the finish line, so that I don't need to train our staff on 2 different stack management tools. 🤞

I'm still seeing this issue with:

Feels like, even with the fixed aws-cli, it needs to be fixed for sam-cli as well.

alexandrosgkesos commented 8 months ago

Still doesn't work with $ sam --version SAM CLI, version 1.111.0

rhbecker commented 8 months ago

@moelasmar Please consider changing this issue's label from type/feature to type/bug.

The language used on SAM's main marketing page, under the Why SAM? header, reinforces what several participants have asserted - i.e. that this issue is a bug ...

You can also define resources using CloudFormation in your SAM template and use the full suite of resources, intrinsic functions, and other template features that are available in AWS CloudFormation.

In reviewing this repo's issue history, I observe that type/feature issues are rarely resolved, while type/bug issues receive a good amount of attention.

rapala61 commented 6 months ago

Hi everyone, I'm just here to bump this bug to keep it in the radar. I have a ~1000 line template that could be reduced to ~100 if this issue was resolved.

scolebrook commented 6 months ago

Bump!

https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-specification-intrinsic-functions.html

The SAM docs state that adding the AWS::LanguageExtensions transform resolves it's lack of support for some intrinsic functions.

Intrinsic functions are built-in functions that enable you to assign values to properties that are only available at runtime. AWS SAM has limited support for certain intrinsic function properties, so it is unable to resolve some intrinsic functions. Consequently, we recommend adding the AWS::LanguageExtensions transform to resolve this:

This issue is either a bug in the documentation (it should clearly state what intrinsic functions aren't supported even when using AWS::LanguageExtensions) or it's a bug in SAM not functioning as documented.

CloudFormation itself can correctly resolve the transforms with both Language Extensions and Serverless listed in that order.

prower-turnitin commented 6 months ago

Please work on this. ForEach seems pretty basic function to me. If it can't be done with AWS::LanguageExtensions can SAM have these features built into itself?

wasabideveloper commented 5 months ago

+1

Hi, I am running into the same issue here. Please change the issue type to type/bug and give this issue more attention. This is definitely a bug, because the documentation clearly indicates that SAM supports all CloudFormation features. Thanks!

Details regarding my case if it helps: Template:

AWSTemplateFormatVersion: "2010-09-09"
Transform:
  - AWS::LanguageExtensions
  - AWS::Serverless-2016-10-31 # Removing Serverless transformation does not change anything

# ...

Resources:
  ReadOnlyGroup:
    Type: AWS::IdentityStore::Group
    Properties:
      Description: Read-only (all accounts)
      DisplayName: read-only
      IdentityStoreId: !Ref identityStoreId

  "Fn::ForEach::ReadOnlyGroupAssignment":
    - AccountId
    - !Split [",", !Ref allAccountIds]
    - "ReadOnlyGroupAssignment${AccountId}":
        Type: AWS::SSO::Assignment
        Properties:
          InstanceArn: !Ref identityCenterInstanceArn
          PermissionSetArn: !GetAtt ReadOnlyPermissionSet.PermissionSetArn
          TargetId: !Ref AccountId
          TargetType: "AWS_ACCOUNT"
          PrincipalType: "GROUP"
          PrincipalId: !GetAtt ReadOnlyGroup.GroupId

Console output of sam build:

Starting Build use cache                                                                                                                   
2024-07-05 08:13:52 Plugin 'ServerlessAppPlugin' raised an exception: 'list' object has no attribute 'get'
Traceback (most recent call last):
  File "samtranslator/plugins/sam_plugins.py", line 130, in act
  File "samtranslator/metrics/method_decorator.py", line 116, in wrapper_cw_timer
  File "samtranslator/plugins/application/serverless_app_plugin.py", line 124, in on_before_transform_template
  File "samtranslator/sdk/template.py", line 34, in iterate
  File "samtranslator/sdk/resource.py", line 26, in __init__
AttributeError: 'list' object has no attribute 'get'

Error: 'list' object has no attribute 'get'
Traceback:
  File "click/core.py", line 1055, in main
  File "click/core.py", line 1657, in invoke
  File "click/core.py", line 1404, in invoke
  File "click/core.py", line 760, in invoke
  File "click/decorators.py", line 84, in new_func
  File "click/core.py", line 760, in invoke
  File "samcli/lib/telemetry/metric.py", line 184, in wrapped
  File "samcli/lib/telemetry/metric.py", line 149, in wrapped
  File "samcli/lib/utils/version_checker.py", line 42, in wrapped
  File "samcli/cli/main.py", line 95, in wrapper
  File "samcli/commands/build/command.py", line 166, in cli
  File "samcli/commands/build/command.py", line 238, in do_cli
  File "samcli/commands/build/build_context.py", line 184, in __enter__
  File "samcli/commands/build/build_context.py", line 190, in set_up
  File "samcli/lib/providers/sam_stack_provider.py", line 280, in get_stacks
  File "samcli/lib/providers/sam_stack_provider.py", line 269, in get_stacks
  File "samcli/lib/providers/sam_stack_provider.py", line 61, in __init__
  File "samcli/lib/providers/sam_base_provider.py", line 193, in get_template
  File "samcli/lib/samlib/wrapper.py", line 73, in run_plugins
  File "samcli/lib/samlib/wrapper.py", line 130, in parse
  File "samtranslator/plugins/sam_plugins.py", line 136, in act
  File "samtranslator/plugins/sam_plugins.py", line 130, in act
  File "samtranslator/metrics/method_decorator.py", line 116, in wrapper_cw_timer
  File "samtranslator/plugins/application/serverless_app_plugin.py", line 124, in on_before_transform_template
  File "samtranslator/sdk/template.py", line 34, in iterate
  File "samtranslator/sdk/resource.py", line 26, in __init__

An unexpected error was encountered while executing "sam build".
Search for an existing issue:
https://github.com/aws/aws-sam-cli/issues?q=is%3Aissue+is%3Aopen+Bug%3A%20sam%20build%20-%20AttributeError
Or create a bug report:
https://github.com/aws/aws-sam-cli/issues/new?template=Bug_report.md&title=Bug%3A%20sam%20build%20-%20AttributeError
raymalt commented 2 months ago

This would be a very useful feature for us, since it would allow us to generate our resources without bloating the template file.

wasabideveloper commented 2 months ago

This would be a very useful feature for us, since it would allow us to generate our resources without bloating the template file.

As a temporary workaround you can deploy the Count macro from the AWS CloudFormation repository in your account ( https://github.com/aws-cloudformation/aws-cloudformation-templates/tree/main/CloudFormation/MacrosExamples/Count) and use it to "loop" over parameter lists:

AWSTemplateFormatVersion: "2010-09-09"
Transform:
  - Count
  - AWS::Serverless-2016-10-31

# ...
Resources:
  CountMacro:
    Type: AWS::CloudFormation::Macro
    Properties:
      Name: Count
      FunctionName: !Ref countMacroFunctionArn

  # ...
  ReadOnlyPermissionSet:
    Type: AWS::SSO::PermissionSet
    Properties:
      Name: ReadOnly
      Description: Read only access based on AWS ReadOnlyAccess policy
      InstanceArn: !Ref identityCenterInstanceArn
      ManagedPolicies:
        - !Sub "arn:${AWS::Partition}:iam::aws:policy/ReadOnlyAccess"

  ReadOnlyGroupAssignments:
    Type: AWS::SSO::Assignment
    Count: !Ref allAccountIds
    Properties:
      InstanceArn: !Ref identityCenterInstanceArn
      PermissionSetArn: !GetAtt ReadOnlyPermissionSet.PermissionSetArn
      TargetId: "%s"
      TargetType: AWS_ACCOUNT
      PrincipalType: GROUP
      PrincipalId: !GetAtt ReadOnlyGroup.GroupId

Of course it would be better SAM supported Fn::ForEach natively, but at least you can avoid to hardcode resources that need to be generated based on list parameters.

raymalt commented 2 months ago

This would be a very useful feature for us, since it would allow us to generate our resources without bloating the template file.

As a temporary workaround you can deploy the Count macro from the AWS CloudFormation repository in your account ( https://github.com/aws-cloudformation/aws-cloudformation-templates/tree/main/CloudFormation/MacrosExamples/Count) and use it to "loop" over parameter lists:

AWSTemplateFormatVersion: "2010-09-09"
Transform:
  - Count
  - AWS::Serverless-2016-10-31

# ...
Resources:
  CountMacro:
    Type: AWS::CloudFormation::Macro
    Properties:
      Name: Count
      FunctionName: !Ref countMacroFunctionArn

  # ...
  ReadOnlyPermissionSet:
    Type: AWS::SSO::PermissionSet
    Properties:
      Name: ReadOnly
      Description: Read only access based on AWS ReadOnlyAccess policy
      InstanceArn: !Ref identityCenterInstanceArn
      ManagedPolicies:
        - !Sub "arn:${AWS::Partition}:iam::aws:policy/ReadOnlyAccess"

  ReadOnlyGroupAssignments:
    Type: AWS::SSO::Assignment
    Count: !Ref allAccountIds
    Properties:
      InstanceArn: !Ref identityCenterInstanceArn
      PermissionSetArn: !GetAtt ReadOnlyPermissionSet.PermissionSetArn
      TargetId: "%s"
      TargetType: AWS_ACCOUNT
      PrincipalType: GROUP
      PrincipalId: !GetAtt ReadOnlyGroup.GroupId

Of course it would be better SAM supported Fn::ForEach natively, but at least you can avoid to hardcode resources that need to be generated based on list parameters.

Thanks, this looks promising. However, we need to generate a few different resources for each tenant (Lambdas, Roles, Kinesis streams etc), some of which take parameters that we would like to loop through, e.g. CodeUri for the Lambdas. We were thinking of using Mappings and ForEach over a list of keys in the mapping, thereby extracting the parameters (with FindInMap) that are exclusive for each tenant.

Do you have a workaround suggestion to achieve this? I suppose we could write our own macro for this purpose, but we'd rather not... :)

wasabideveloper commented 2 months ago

This would be a very useful feature for us, since it would allow us to generate our resources without bloating the template file.

As a temporary workaround you can deploy the Count macro from the AWS CloudFormation repository in your account ( https://github.com/aws-cloudformation/aws-cloudformation-templates/tree/main/CloudFormation/MacrosExamples/Count) and use it to "loop" over parameter lists:

AWSTemplateFormatVersion: "2010-09-09"
Transform:
  - Count
  - AWS::Serverless-2016-10-31

# ...
Resources:
  CountMacro:
    Type: AWS::CloudFormation::Macro
    Properties:
      Name: Count
      FunctionName: !Ref countMacroFunctionArn

  # ...
  ReadOnlyPermissionSet:
    Type: AWS::SSO::PermissionSet
    Properties:
      Name: ReadOnly
      Description: Read only access based on AWS ReadOnlyAccess policy
      InstanceArn: !Ref identityCenterInstanceArn
      ManagedPolicies:
        - !Sub "arn:${AWS::Partition}:iam::aws:policy/ReadOnlyAccess"

  ReadOnlyGroupAssignments:
    Type: AWS::SSO::Assignment
    Count: !Ref allAccountIds
    Properties:
      InstanceArn: !Ref identityCenterInstanceArn
      PermissionSetArn: !GetAtt ReadOnlyPermissionSet.PermissionSetArn
      TargetId: "%s"
      TargetType: AWS_ACCOUNT
      PrincipalType: GROUP
      PrincipalId: !GetAtt ReadOnlyGroup.GroupId

Of course it would be better SAM supported Fn::ForEach natively, but at least you can avoid to hardcode resources that need to be generated based on list parameters.

Thanks, this looks promising. However, we need to generate a few different resources for each tenant (Lambdas, Roles, Kinesis streams etc), some of which take parameters that we would like to loop through, e.g. CodeUri for the Lambdas. We were thinking of using Mappings and ForEach over a list of keys in the mapping, thereby extracting the parameters (with FindInMap) that are exclusive for each tenant.

Do you have a workaround suggestion to achieve this? I suppose we could write our own macro for this purpose, but we'd rather not... :)

The Count macro is quite simple. It allows to iterate over a variable of type CommaDelimitedList or iterate x times when the variable is a number. It does not support complex inputs like mappings etc. I didn't review the whole code of the Count Lambda but I assume the same limitation applies for the %s placeholder and it is not possible to use expressions like !FindInMap. Of course one could extend the Lambda code... Another solution could be packing these resources into a substack but keep the mapping in the main stack. This way a simple variable would be passed to Count. This is a "trick" mentioned in this Medium article: https://medium.com/aws-tip/implementing-variables-in-aws-cloudformation-templates-0202e696d606

So I think you could make your use case work with Count but not as straight forward and simple like it would be using Fn::ForEach.

metadaddy commented 2 months ago

@wasabideveloper Thanks for pointing to Count - it does look useful, but it seems to be restricted to resources. I have a situation where I need a variable number of statements in a policy. For example, something like::

#...
Parameters:
  BucketNames:
    Description: List of S3 bucket names
    Type: CommaDelimitedList

Rules:
  mustSpecifyAtLeastOneBucketName:
    Assertions:
      - Assert:
          !Not [!Equals [!Length [!Ref SourceBucketNames], 0]]
        AssertDescription: 'You must specify at least one source bucket name'

Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      #...
      Policies:
        # Allow GetObject on each of the listed buckets
        'Fn::ForEach::Statements':
          - SourceBucketName
          - Ref: SourceBucketNames
          - Statement:
              - Effect: Allow
                Action: s3:GetObject
                Resource: !Sub 'arn:aws:s3:::${SourceBucketName}/*'

This seems like such a basic requirement, but I haven't been able to figure out how to implement it without writing a custom macro.

wasabideveloper commented 2 months ago

@wasabideveloper Thanks for pointing to Count - it does look useful, but it seems to be restricted to resources. I have a situation where I need a variable number of statements in a policy. For example, something like::

#...
Parameters:
  BucketNames:
    Description: List of S3 bucket names
    Type: CommaDelimitedList

Rules:
  mustSpecifyAtLeastOneBucketName:
    Assertions:
      - Assert:
          !Not [!Equals [!Length [!Ref SourceBucketNames], 0]]
        AssertDescription: 'You must specify at least one source bucket name'

Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      #...
      Policies:
        # Allow GetObject on each of the listed buckets
        'Fn::ForEach::Statements':
          - SourceBucketName
          - Ref: SourceBucketNames
          - Statement:
              - Effect: Allow
                Action: s3:GetObject
                Resource: !Sub 'arn:aws:s3:::${SourceBucketName}/*'

This seems like such a basic requirement, but I haven't been able to figure out how to implement it without writing a custom macro.

Yes, I think it's restricted to resources only. To implement your use case, a custom resource might be the only option.

raymalt commented 2 months ago

The Count macro is quite simple. It allows to iterate over a variable of type CommaDelimitedList or iterate x times when the variable is a number. It does not support complex inputs like mappings etc. I didn't review the whole code of the Count Lambda but I assume the same limitation applies for the %s placeholder and it is not possible to use expressions like !FindInMap. Of course one could extend the Lambda code... Another solution could be packing these resources into a substack but keep the mapping in the main stack. This way a simple variable would be passed to Count. This is a "trick" mentioned in this Medium article: https://medium.com/aws-tip/implementing-variables-in-aws-cloudformation-templates-0202e696d606

So I think you could make your use case work with Count but not as straight forward and simple like it would be using Fn::ForEach.

Thanks for suggesting the use of Substack, it seems to fit our use case perfectly. It's not the loop enumeration itself that is important to us, it's rather the ability to reuse templates and pass parameters. The only drawback I've found (so far) is that the observability of Substack resources is limited when deploying - it seems you can only track that changes will be applied to the entire stack rather than individual services - but we'll have to live with that.

wasabideveloper commented 2 months ago

The only drawback I've found (so far) is that the observability of Substack resources is limited when deploying - it seems you can only track that changes will be applied to the entire stack rather than individual services - but we'll have to live with that.

@raymalt This is quite an annoying limitation of Substacks. Until now I did not find a workaround to overcome this issue.

btalbot commented 1 week ago

Still broken and bugged for SAM CLI, version 1.129.0 1.5 YEARS later ...