Serverless Transform failing when using Intrinsic function within value for "ContentUri" property for "AWS::Serverless::LayerVersion" resource

razhamma commented 5 years ago

Issue Description

The following snippet fails transformation:

Transform: 'AWS::Serverless-2016-10-31'
Resources:
  MyServerlessLayerVersion:
    Type: 'AWS::Serverless::LayerVersion'
    Properties:
      LayerName: !Sub ${AWS::Region}-my-layer
      Description: !Sub ${AWS::Region}-my-layer-description
      ContentUri: !Sub "s3://some-example-bucket-${AWS::Region}/someKey.zip"
      CompatibleRuntimes:
      - python3.7
      RetentionPolicy: Retain

with the Error as: "Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [MyServerlessLayerVersion9b0b8a66e5] is invalid. 'ContentUri' requires Bucket and Key properties to be specified".

Reasoning

According to this[1], the "ContentUri" itself is transformed using an S3 util[2] where "ContentUri" is passed to this util as it is(whether a string or an object). However, due to intrinsic function "ContentUri" is treated as an object and then this object is passed to S3 util.
Checking into S3 util's code here[3], missing any of the 'Bucket' or 'Key' property of "ContentUri" object results into transformation failure.

Workaround

Using the S3 location object format for ContentUri property and using intrinsic functions inside that S3 location object will work as expected without getting any transformation errors.
By specifying ContentUri as per S3 location object[4], you won't fail any validation during S3 util parser run and SAM translator will simply pass these values for Key and Bucket property to CloudFormation AWS::Lambda::LayerVersion resource.

The correct version for the above mentioned problem facing snippet will look like:

Transform: 'AWS::Serverless-2016-10-31'
Resources:
MyServerlessLayerVersion:
Type: 'AWS::Serverless::LayerVersion'
Properties:
  LayerName: !Sub ${AWS::Region}-my-layer
  Description: !Sub ${AWS::Region}-my-layer-description
  ContentUri: 
    Bucket: !Sub some-example-bucket-${AWS::Region}
    Key: someKey.zip
  CompatibleRuntimes:
  - python3.7
  RetentionPolicy: Retain

Suggestions/Observations

CloudFormation resolves intrinsic functions only after transformation is done. So, parsing the string(containing intrinsic function) correctly into 'S3 Location Object' and keeping the exact intrinsic function specifications for this object's properties 'Key' and 'Bucket' during transformation will solve the issue.
Another suggestion is making AWS::Serverless Transform Macro capable enough to resolve intrinsic functions beforehand during Transformation. However, it won't be a good approach as we'll be performing same task as CloudFormation is expected to do and it doesn't really go with the sole purpose of just performing SAM based resource translations to CloudFormation specific resources.

Regarding Correct Transformation/Parsing

In case if 'ContentUri' string containing intrinsic function was passed to S3 Util paser as a string, it will still fail as S3 util parser itself is using specific python libs(urlparse, parse_qs)[5] and these libs won't be able to parse a string containing CloudFormation specific intrinsic functions.
Thus, forcing the string(containing intrinsic function) to be always considered as a string rather than an object and enabling S3 util parser to parse this string by appropriate inclusion/substitution of intrinsic function inside the values for 'Key' and 'Bucket' properties of 'S3 Location Object' can do the job.

References

[1] https://github.com/awslabs/serverless-application-model/blob/master/samtranslator/model/sam_resources.py#L732 [2] https://github.com/awslabs/serverless-application-model/blob/master/samtranslator/model/s3_utils/uri_parser.py [3] https://github.com/awslabs/serverless-application-model/blob/master/samtranslator/model/s3_utils/uri_parser.py#L61 [4] https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlesslayerversion [5] https://github.com/awslabs/serverless-application-model/blob/master/samtranslator/model/s3_utils/uri_parser.py#L2

mingkun2020 commented 2 years ago

Thanks for reporting the issue, we will be investigating it further. Please watch this channel for more updates, and feel free to reach out.

hoffa commented 1 year ago

You might be able to get this to work by adding AWS::LanguageExtensions to Transform as such:

Transform:
  - AWS::LanguageExtensions
  - AWS::Serverless-2016-10-31

AWS::LanguageExtensions resolves intrinsic functions if the value is known when Transforms are run.

See https://github.com/aws/serverless-application-model/issues/2533 for more information.

hoffa commented 1 year ago

Closing in favor of https://github.com/aws/serverless-application-model/issues/2533.

tariromukute commented 1 year ago

You might be able to get this to work by adding AWS::LanguageExtensions to Transform as such:
Transform:
  - AWS::LanguageExtensions
  - AWS::Serverless-2016-10-31
AWS::LanguageExtensions resolves intrinsic functions if the value is known when Transforms are run.

See #2533 for more information.

Thanks @hoffa for the suggestion. I tried the above but didn't manage to get it working. The deployment fails with error. Any ideas?

"FAILED" Status: FAILED. Reason: Transform AWS::Serverless-2016-10-31 failed with: Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [utilitiesLayerfc27b2746a] is invalid. 'ContentUri' is not a valid S3 Uri of the form 's3://bucket/key' with optional versionId query parameter.

mcrysler-ng commented 10 months ago

Why was this issue closed? I am running into this issue as well so I'm assuming it hasn't been fixed?

ConnorRobertson commented 10 months ago

@mcrysler-ng have you tried what hoffa mentioned here? It was closed in favour of this issue. Which provides workarounds and an explanation of why this issue was closed. Feel free to open a new issue if this problem still persists.

mcrysler-ng commented 10 months ago

Language extensions are the one thing I haven’t tried yet. I tried the other two workarounds and neither worked. Using language extensions worries me because of the comment about possible conflicts.

Matt Crysler | Application Engineer

From: Connor Robertson @.> Sent: Wednesday, November 22, 2023 3:17:28 PM To: aws/serverless-application-model @.> Cc: Matt Crysler @.>; Mention @.> Subject: Re: [aws/serverless-application-model] Serverless Transform failing when using Intrinsic function within value for "ContentUri" property for "AWS::Serverless::LayerVersion" resource (#1159)

@mcrysler-nghttps://github.com/mcrysler-ng have you tried what hoffa mentioned here?https://github.com/aws/serverless-application-model/issues/1159#issuecomment-1281539068 It was closed in favour of this issue.https://github.com/aws/serverless-application-model/issues/2533 Which provides workarounds and an explanation of why this issue was closed. Feel free to open a new issue if this problem still persists.

— Reply to this email directly, view it on GitHubhttps://github.com/aws/serverless-application-model/issues/1159#issuecomment-1823576370, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ATGOQ6MWFEMXW7LHC3WOYUDYFZ2XRAVCNFSM4I2MTLFKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBSGM2TONRTG4YA. You are receiving this because you were mentioned.Message ID: @.***>

GavinZZ commented 10 months ago

@mcrysler-ng would you be able to provide a snippet of your template? Curious why does this workaround not work for you.

ContentUri: 
        Bucket: !Sub some-example-bucket-${AWS::Region}
        Key: someKey.zip

As for the conflict in Language Extension, currently the only known conflict is that you need to depends on ApiGateway Stage resource and you've explicitly used intrinsic function for StageName property. If that doesn't apply to you, please give Language Extension a try too.

mcrysler-ng commented 10 months ago

That’s almost the exact sample I have except the function is on the key name and I get a NoSuchKey error. If I remove the function and replace it with a constant string, it works. So I know it’s not a permissions issue or anything else. I even tried using the non-SAM version and the Content object that expects S3Bucket and S3Key behave the exact same.

Matt Crysler | Application Engineer

From: GZ @.> Sent: Wednesday, November 22, 2023 3:50:58 PM To: aws/serverless-application-model @.> Cc: Matt Crysler @.>; Mention @.> Subject: Re: [aws/serverless-application-model] Serverless Transform failing when using Intrinsic function within value for "ContentUri" property for "AWS::Serverless::LayerVersion" resource (#1159)

@mcrysler-nghttps://github.com/mcrysler-ng would you be able to provide a snippet of your template? Curious why does this workaround not work for you.

ContentUri: Bucket: !Sub some-example-bucket-${AWS::Region} Key: someKey.zip

— Reply to this email directly, view it on GitHubhttps://github.com/aws/serverless-application-model/issues/1159#issuecomment-1823600659, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ATGOQ6MBUY2JPTE4AI7TBJLYFZ6VFAVCNFSM4I2MTLFKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBSGM3DAMBWGU4Q. You are receiving this because you were mentioned.Message ID: @.***>

mcrysler-ng commented 10 months ago

Snippet included below. The Env parameter is defined earlier in the file and used in almost every resource definition so I know that's not the issue.

This is what is NOT working:

MyLibrary:
    Type: AWS::Serverless::LayerVersion
    Metadata:
      BuildMethod: nodejs18.x
      BuildArchitecture: x86_64
    Properties:
      CompatibleArchitectures:
        - x86_64
      CompatibleRuntimes:
        - nodejs18.x
      ContentUri:
        Bucket: my-bucket
        Key: !Sub "${Env}/${Env}-my-library-name.zip"
      Description: Provides shared library for blah
      LayerName: MyLayer
      LicenseInfo: Available under the MIT-0 license
      RetentionPolicy: Retain

This is what IS working:

MyLibrary:
    Type: AWS::Serverless::LayerVersion
    Metadata:
      BuildMethod: nodejs18.x
      BuildArchitecture: x86_64
    Properties:
      CompatibleArchitectures:
        - x86_64
      CompatibleRuntimes:
        - nodejs18.x
      ContentUri:
        Bucket: my-bucket
        Key: Dev/Dev-my-library-name.zip
      Description: Provides shared library for blah
      LayerName: MyLayer
      LicenseInfo: Available under the MIT-0 license
      RetentionPolicy: Retain

mcrysler-ng commented 10 months ago

And this is the error I get with the first version above:

Resource handler returned message: "Error occurred while GetObject. S3 Error Code: NoSuchKey. S3 Error Message: The specified key does not exist. (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: b4dd165b-b25c-43d6-945e-0108695b4027; Proxy: null)"

aws / serverless-application-model