Unsupported intrinsic functions

[hoffa]

The problem

Some properties have limited support for intrinsic functions (e.g. Ref, Fn::GetAtt, Fn::If, etc.).

Using unsupported intrinsic functions in such properties can cause issues with deployment.

Why it happens

AWS SAM is a AWS CloudFormation macro; it receives a SAM template as input (as-is, along with the intrinsic functions), and returns a CloudFormation template which is then deployed by CloudFormation.

This means that SAM is unable to resolve some intrinsic functions. For example, an !If condition with a Ref to a stack parameter is resolvable (SAM has access to the stack parameters), but a Ref to a stack resource is not (as the transform happens before deployment). Furthermore, SAM supports limited intrinsic function resolution for only some properties.

It's not an issue for properties that are passed as-is to properties of the underlying CloudFormation resources, but it becomes an issue when SAM must know the value for its transform logic.

Workarounds

Add the AWS::LanguageExtensions transform

The AWS::LanguageExtensions transform will resolve intrinsic functions if the value is known when Transforms are run.

Replace:

Transform: AWS::Serverless-2016-10-31

With:

Transform:
  - AWS::LanguageExtensions
  - AWS::Serverless-2016-10-31

The AWS SAM CLI currently doesn't process AWS::LanguageExtensions locally, so it won't work where local transforms are needed.

[!NOTE] There is a known issue where AWS::Serverless-2016-10-31 and AWS::LanguageExtensions can conflict; see https://github.com/aws-cloudformation/cfn-language-discussion/issues/109.

Use a pass-through property, if available

Pass-through properties are passed directly to the underlying CloudFormation resources, hence intrinsic functions work. Check the "AWS CloudFormation compatibility" notice under properties to see whether the property is passed as-is to an underlying CloudFormation resource.

For example for the Schedule event type, use the State property instead of Enabled.

Use raw CloudFormation

If nothing else works, you can always switch to using the underlying CloudFormation resources directly. Since they are not processed by SAM, CloudFormation will be able to resolve the intrinsic functions.

You can get the transformed CloudFormation template of a stack <my-stack> using:

aws cloudformation get-template --query TemplateBody --change-set-name "$(aws cloudformation describe-stacks --query 'Stacks[0].ChangeSetId' --output text --stack-name <my-stack>)"

Which you can then use to replace the affected resources.

See also https://github.com/aws/serverless-application-model/issues/3007#issuecomment-1464194429 for other ways of transforming a SAM template into a CloudFormation template.

[tom-mcclintock]

Chiming in to say I came here via aws/serverless-application-model#2377 and am very interested to see this get supported.

In my use case, I use !Ref to set VPC endpoint IDs:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Parameters:
  VPCE:
    Type: String

Resources:
  ApiGateway:
    Type: AWS::Serverless::Api
    Properties:
      StageName: somestagename
      EndpointConfiguration:
        Type: PRIVATE
        VPCEndpointIds:
          - !Ref VPCE

Running sam validate yields an error, but also ends with the green message ... is a valid SAM Template:

Template schema validation reported the following errors: [Resources.ApiGateway.Properties.EndpointConfiguration.VPCEndpointIds.0] {'Ref': 'VPCE'} is not of type 'string'
template.yaml is a valid SAM Template

So, it is not a blocker, but it would be nice to eliminate this error :). Thanks for working on this.

[berzi]

Is anyone working on this? I'm seemingly unable to define API endpoints in nested stacks because importing the API (AWS::Serverless::Api) from the parent stack doesn't work (resulting in "RestApiId must be a valid reference to an 'AWS::Serv erless::Api' resource in same template" and I suspect this is the reason. It seems like a rather vital feature to have.

[hoffa]

Is anyone working on this? I'm seemingly unable to define API endpoints in nested stacks because importing the API (AWS::Serverless::Api) from the parent stack doesn't work (resulting in "RestApiId must be a valid reference to an 'AWS::Serv erless::Api' resource in same template" and I suspect this is the reason. It seems like a rather vital feature to have.

At a glance unsure whether it's related; if you create a new issue with reproducible steps we can assist you there. As for intrinsic function resolution as described in this issue, there are no current plans to address this in the SAM transform, as AWS::LanguageExtensions already does it.

[tariromukute]

I am coming from issue aws/serverless-application-model#1159. I am failing to get the deployment to work for LayerVersion. Please see a snap of the template.yaml below.

utilitiesLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: utilities-layer
      Description: Utility layer
      ContentUri: !Ref UtilitiesPath
      CompatibleRuntimes:
        - nodejs12.x
      LicenseInfo: "MIT"
      RetentionPolicy: Retain

The error I am getting is as below due to ContentUri: !Ref UtilitiesPath

"FAILED" Status: FAILED. Reason: Transform AWS::Serverless-2016-10-31 failed with: Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [utilitiesLayerfc27b2746a] is invalid. 'ContentUri' is not a valid S3 Uri of the form 's3://bucket/key' with optional versionId query parameter.

Any ideas on how I can resolve this, or any work arounds?

[mgorski-mg]

I am coming from issue aws/serverless-application-model#1159. I am failing to get the deployment to work for LayerVersion. Please see a snap of the template.yaml below.

utilitiesLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: utilities-layer
      Description: Utility layer
      ContentUri: !Ref UtilitiesPath
      CompatibleRuntimes:
        - nodejs12.x
      LicenseInfo: "MIT"
      RetentionPolicy: Retain

The error I am getting is as below due to ContentUri: !Ref UtilitiesPath

"FAILED" Status: FAILED. Reason: Transform AWS::Serverless-2016-10-31 failed with: Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [utilitiesLayerfc27b2746a] is invalid. 'ContentUri' is not a valid S3 Uri of the form 's3://bucket/key' with optional versionId query parameter.

Any ideas on how I can resolve this, or any work arounds?

If UtilitiesPath is S3Bucket then !Ref is resolved to it's ID(name). More info here. The ContentUri field requires Uri: s3://. What you probably need is

!Sub s3://${UtilitiesPath}

[tariromukute]

Thanks @mgorski-mg UtilitiesPath is a path to the folder with the layers. My Parameters sections looks like below.

AWSEndpoint:
    Type: String
    Description: The endpoint to use for AWS client. Useful when we want request to go to localstack
    Default: None
  UtilitiesPath:
    Type: String
    Description: The base path to utilities. Needed for running inside docker with docker compose. This should reference the base path inside docker container
    Default: "utilities-layer/"

If I put ContentUri: utilities-layer/ it works fine. However I need to parametrise it because when I try to run this with docker compose the path needs to reference the working directory inside the docker container. You can see the discussion on this here https://github.com/aws/aws-sam-cli/issues/2837#issuecomment-1430149431

[mgorski-mg]

@tariromukute oh I see 🙈 that's strange them. Then it's probably connected to this issue, your right. You can go with this AWS:: LanguageExtensions. I checked it some time ago and it is doing the job. I see your comment from the previous issue where you mentioned that it is still not working with this transformation. Then I have no other ideas unfortunately 😞

For me it started to work even without this transformation. https://github.com/mgorski-mg/aws-orphaned-log-groups-collector/blob/master/template.yaml#L46

[tariromukute]

Hey @mgorski-mg for responding quickly. I tried adding AWS::LanguageExtensions the error above is with AWS::LanguageExtensions and like below.

AWSTemplateFormatVersion: "2010-09-09"
Transform:
  - AWS::LanguageExtensions
  - AWS::Serverless-2016-10-31

When I remove AWS::LanguageExtensions I get a different but similar error, see the error below

Error: Failed to create changeset for the stack: rvw-stitch-tr-app, ex: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state: For expression "Status" we matched expected path: "FAILED" Status: FAILED. Reason: Transform AWS::Serverless-2016-10-31 failed with: Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [utilitiesLayer3b82bbbef7] is invalid. 'ContentUri' requires Bucket and Key properties to be specified.

[mgorski-mg]

@tariromukute i know i know now. I edited my previous comment.

[tariromukute]

@tariromukute i know i know now. I edited my previous comment.

Seeing the edit now, thanks @mgorski-mg. Seems like it resolves the issue for some properties.

[hoffa]

@tariromukute The issue seems to be that the intrinsic function must be resolved before deployment (local paths are unsupported by CloudFormation; under the hood SAM CLI uploads the assets and creates an intermediary template with the final S3 URIs), and (1) the SAM transform does not attempt to resolve the intrinsic function locally, and (2) the SAM CLI does not support running AWS::LanguageExtensions locally. The limitation is briefly mentioned in https://github.com/aws/serverless-application-model/issues/2533#issue-1405166577.

[tariromukute]

Thank you @hoffa for the explanation. I went through the comment, however I wasn't able to understand what the workaround for this issue is. How can I use aws cloudformation directly to deploy a project that I initialised with sam init?

[hoffa]

@tariromukute You have to use SAM CLI to deploy any templates with local paths (such as most sam init templates), as SAM CLI needs to upload the assets first.

As for intrinsic functions that should resolve to local paths, unless the property implements some ad-hoc intrinsic function resolution, there's no great workaround apart from hardcoding. This is something we could explore in SAM CLI so that e.g. it also resolves intrinsic functions before passing to the SAM transform.

[tariromukute]

Hi @hoffa, thank you for clarifying the above. Is there an issue or work being done on this (we could explore in SAM CLI so that e.g. it also resolves intrinsic functions before passing to the SAM transform) or there is need for this to be created. This will make development and deployment easier. We tend to have javascript developers that do not have a lot of cloud experience and knowledge. Allowing SAM applications to be set up end-to-end locally using docker-compose (configure SAM app and local stack etc) will enable the developers to work on the lambda code without needing to know or deal with the cloud and SAM configurations.

[hoffa]

@tariromukute If I'm understanding correctly the problem you're encountering is !Ref to local paths in AWS::Serverless::LayerVersion not working. I was able to reproduce it, and confirmed it works with CodeUri in AWS::Lambda::Function. I've created https://github.com/aws/aws-sam-cli/issues/4767 for the issue. There's not much else that can be done on the SAM transform side as the intrinsic resolution to the local path must happen before the SAM transform receives it. While not ideal, as workaround in the meantime you could use sed or similar to replace the text.

[tariromukute]

@tariromukute If I'm understanding correctly the problem you're encountering is !Ref to local paths in AWS::Serverless::LayerVersion not working. I was able to reproduce it, and confirmed it works with CodeUri in AWS::Lambda::Function. I've created aws/aws-sam-cli#4767 for the issue. There's not much else that can be done on the SAM transform side as the intrinsic resolution to the local path must happen before the SAM transform receives it. While not ideal, as workaround in the meantime you could use sed or similar to replace the text.

Awesome. Thanks @hoffa for reproducing the issue and for creating an issue for it. I will try the sed route on the CI pipeline.

[garretwilson]

This breaks CodeUri when deploying AWS lambda from a prebuilt JAR file using !Sub (see aws/serverless-application-model#271). Let's say I'm deploying some lambda function I built with Maven, and I don't want to hard-code the JAR version in my SAM template.

Resources:
  FooFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: !Sub "target/foo-bar-functions-${Ver}.jar"
      Handler: com.example.FooBar::foo

SAM gives me an error:

'CodeUri' requires Bucket and Key properties to be specified.

If I try the workaround noted here of adding AWS::LanguageExtensions, that SAM will at least upload my JAR to the S3 bucket. But then when it fails with:

'CodeUri' is not a valid S3 Uri of the form 's3://bucket/key' with optional versionId query parameter.

The problem is that SAM is not updating the CodeUri value before it uploads the template to S3, as it would if the value were a literal string. Shouldn't SAM know the dereferenced value at the time of uploading the template? Hasn't the template already been processed? There must be a bug where SAM forgets it needs to update the CodeUri value before uploading the template if !Ref was used. I filed bug aws/aws-sam-cli#5249.

[smaly-amazon]

If the AWS::LanguageExtensions transform is the solution, is that transform available in all regions where the serverless transform is available?

[davidchatz]

I've been using the language extension to workaround this issue with using !If in AutoPublishAlias, but this does not work with the new sam list command, which outputs: Error: [InvalidResourceException('SnsTopicConsumer', "'AutoPublishAlias' must be a string or a Ref to a template parameter")] ('SnsTopicConsumer', "'AutoPublishAlias' must be a string or a Ref to a template parameter")

[GavinZZ]

Hi @davidchatz, feel free to create an issue in aws-sam-cli repository on this.

[ozcsdog]

I am coming from https://github.com/aws/serverless-application-model/issues/1616. Is there any way to set string to whole Step Functions TimeoutSeconds? Just like TimeoutSecondsPath for Task. Now the whole Step Functions TimeoutSeconds is only for int, and can't get string from path.

[anashakt]

I just now ran into same situation while trying to pass "TimeoutSeconds" as integer in definitionSubstitutions

[carlosleon-wl]

Hi, I'm having a very simple problem:

      Events:
        Status:
          Type: HttpApi
          Properties:
            ApiId: !Ref HttpApi
            Method: get
            Path: !Sub "/${Version}/status"

I cannot use !Sub to substitute Version parameter. This is a long-standing problem according to what I've seen. Is there anything I can do apart from hardcoding the Version?

[moelasmar]

@carlosleon-wl .. you can use the workaround mentioned in this issue.

Add the AWS::LanguageExtensions transform

The AWS::LanguageExtensions transform will resolve intrinsic functions if the value is known when Transforms are run.

Replace:

Transform: AWS::Serverless-2016-10-31

With:

Transform:
  - AWS::LanguageExtensions
  - AWS::Serverless-2016-10-31