Swagger templates don't get transformed when using DefinitionUri

[jabalsad]

Hi,

I'm trying to package a swagger spec using the flow proposed by SAM, but it doesn't do the transform properly.

When specifying an AWS::Serverless:Api resource, I have two options for the swagger spec:

1. DefinitionUri
2. DefinitionBody

When I use DefinitionUri, then any references to Lambda functions in my swagger document will not get transformed to the actual ARNs of the Lambda functions, e.g. (in swagger.yaml)

...
      x-amazon-apigateway-integration:
        responses:
          default:
            statusCode: 200
        uri: 
          Fn::Sub: "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${LambdaFunction.Arn}/invocations"

According to the swagger CORS example in the SAM examples folder, you use DefinitionBody. The problem with using DefinitionBody is that the aws cloudformation package command does not transform any references to a local swagger file into a remote file in S3.

The workaround is that I actually put the swagger spec in S3 myself, but this is clunky and contrary to the existing model.

[joelfogue]

I'm running into the same issue as well!

[AlexThomas90210]

Same issue here, Trying to figure out the cleanest way to solve this problem.

[dvdmmc]

@AlexThomas90210 Until cloudformation package supports it, I added an aws s3 cp command into our build system (CodeBuild) .

[oharaandrew314]

I've encountered this issue as well. Eagerly waiting for sam to support this. I can't see any reasonable use for DefinitionUri when you would have to hardcode the references to your lambdas. But ideally, I really shouldn't be forced to add API Gateway integration code to the swagger file when I already have the API mappings on my function definitions.

[joelfogue]

@oharaandrew314 I kinda agree; I was able to get things working normally without the swagger way; but instead implicitly defining the api endpoints within the sam template.yml but I just thought it was cleaner with swagger. Now I no longer think so unless I don't understand the swagger example (api_swagger_cors); it looks like you define the lambda function twice:

1. within the sam template under the

   LambdaFunction:
   Type: AWS::Serverless::Function
    ...

2. You then have to call it again in the Swagger file instead of letting the function definition from above call the function:

   paths:
   /:
   x-amazon-apigateway-any-method:
   ...
   Fn::Sub: "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${LambdaFunction.Arn}/invocations"

   My expectation would be that the swagger definition file would take care of creating the API as well as pointing to the right lambdas through the Fn::Sub: right above.

Could anyone tell me why one would choose the Swagger way over the other way which is to just list out all functions and endpoints in the sam template and let sam create the api for you implicitly?

[joelfogue]

@dvdmmc would you be able to share your sam template and swagger file for reference? I'm using CodeStar and I'm pretty confident my setup should work; it's just failing with some INTERNAL ERROR which are not specific enough for me to find out what the issue is!

[AlexThomas90210]

I saw in a few other threads that CORS support is high priority to SAM, so what I ended up doing is just inlining the definition body in the sam template and then just referencing the API in the Events part of a sam function.

It makes the sam template much larger than it should be, but hopefully, sam will support "cors: true" soon the same way the serverless.com framework does. Then I can remove it.

Not really a solution for people who do want to define their own swagger files though.

[ollyjshaw]

This is a current blocker for us too.

There is no way to use aws cloudformation package and DefinitionBody as it's not a field that has it's reference substituted.

https://docs.aws.amazon.com/cli/latest/reference/cloudformation/package.html

[dvdmmc]

Sorry @Joel-fogue this fell off my radar a bit.

In case you or anyone else is still interested, here is how I've set up my CloudFormation template:

Parameters:
  EnvironmentParameter:
    Type: String
    Default: dev
    AllowedValues:
      - dev
      - prod
    Description: dev, prod
  UserPool:
    Type: String
    Description: Cognito User Pool to use with APIGateway
Resources:
  ApiGateway:
    Type: 'AWS::Serverless::Api'
    Properties:
      Name: !Join [ '', [ !Ref EnvironmentParameter, 'ApiGateway'  ] ]
      StageName: !Ref EnvironmentParameter
      DefinitionBody:
          'Fn::Transform':
            Name: 'AWS::Include'
            # Replace <bucket> with your bucket name
            Parameters:
              Location: !Join [ '', [ 's3://bucketprefixofyourchoice-', !Ref EnvironmentParameter, '/apigatewaydefinition.yaml'  ] ]

In your Swagger template, you can then do things like:

      providerARNs:
      - Fn::Sub: "arn:aws:cognito-idp:us-east-1:123456789:userpool/${UserPool}"

the important step is to have the swagger template copied to an S3 bucket, so my build step for CodeBuild looks something like:

build:
    commands:
      - aws s3 cp ./apigatewaydefinition.yaml s3://myproject-api-"$ENVIRONMENT"/
      - aws cloudformation package .........

Hope this helps, sorry for the late response.

[dan-lind]

Having the exact same problem

[sashokbg]

Any news on this issue ? We just came across the same problem.

For me the solution is for the aws cloudformation package command to upload your swagger file the same way it does with the function CodeUri.

[kevinashaw]

Hi all. Its been seven months. Any updates on this? Is the best recommendation to follow the proposal by dvdmmc, or has this been fixed? In my case, I have inlined the Swagger API definition into the SAM yaml template, but I'm still getting Unable to add Cors configuration because 'DefinitionBody' does not contain a valid Swagger. Thanks.

[brettstack]

I've reached out to the API Gateway team to see if they will fix the underlying issue. However, if they are unable to fix it we may be able to do something like this:

1. We allow DefinitionUri to also accept an Object
2. During sam package, SAM will perform operations on DefinitionUri based on the configuration specified

DefinitionUri:
  Path: src/swagger.yml # Also accepts S3
  Inline: true # Default: true
  Upload: false # Default: false

Path will be either a local file path, or an S3 path. If it's an S3 path, it will be downloaded.

Inline: true will replace DefinitionUri with DefinitionBody and inline the contents of the swagger file there. SAM will then perform transforms against that DefinitionBody as normal.

Upload: true will upload to S3. Default to false as I think most don't want this, however, there are cases where you still want to upload to S3 (e.g if you're using that to generate documentation). Upload: true will thrown an error if an S3 path is provided in Path.

We'll create an RFC for this and accept contributions if people think this will solve their problem.

[kevinashaw]

@brettstack Thank you. This will be helpful. In the meantime, what is the best-practice recommendation?
Currently I am think that I need to rip out the Swagger code and rebuild the API definition in pure CloudFormation -- which does not sound like fun. I get the impression there is a reason people prefer to use Swagger.

[brettstack]

It will work if you just stick the Swagger inline in DefinitionBody or use the workaround posted above by dvdmmc

[kevinashaw]

@brettstack Thank you for the response.
I am finding, even with inlined Swagger, that I get the Invalid Swagger error. Here is the response from sam deploy:

Failed to create the changeset: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. 
Reason: Transform AWS::Serverless-2016-10-31 failed with: Invalid Serverless Application Specification document. 
Number of errors found: 1. Resource with id [MyTest] is invalid. Unable to add Auth configuration because 'DefinitionBody' does not contain a valid Swagger

And the response from sam validate:

2019-04-11 10:10:25 Found credentials in shared credentials file: ~/.aws/credentials
Template provided at '/Users/my_dir/my_template.yaml' was invalid SAM Template.
Error: [InvalidResourceException('MyTest', "Unable to add Cors configuration because 'DefinitionBody' does not contain a valid Swagger")] ('MyTest', "Unable to add Cors configuration because 'DefinitionBody' does not contain a valid Swagger")

The offending lines are:

      Cors:  
        AllowOrigin: '*'

and this

      Auth: 
        DefaultAuthorizer: MyLambdaTokenAuthorizer
        Authorizers:
          MyLambdaTokenAuthorizer:
            FunctionPayloadType: TOKEN
            FunctionArn        : arn:aws:lambda:us-west-2:123123123:function:jwtRsaCustomAuthorizer
            FunctionInvokeRole : arn:aws:iam::123123123:role/Auth0Integration

If either block is present, I get the invalid swagger error. If both are commented out, I get: mytemplate.yaml is a valid SAM Template Here is more of the API block of the template file:

  MyTest:
    Type: AWS::Serverless::Api
    Properties:
      Name          : MyTest
      StageName     : !ref StagingParameter
#      Auth: 
#        DefaultAuthorizer: MyLambdaTokenAuthorizer
#        Authorizers:
#          MyLambdaTokenAuthorizer:
#            FunctionPayloadType: TOKEN
#            FunctionArn        : arn:aws:lambda:us-west-2:123123123:function:jwtRsaCustomAuthorizer
#            FunctionInvokeRole : arn:aws:iam::123123123:role/Auth0Integration
#      Cors:  
#        AllowOrigin: '*'
      DefinitionBody:   # Swagger code definition of the API parameters
        openapi: 3.0.0
        info:
          version: "0.1.0"
          title: My Web API
          description: The primary API for Web and Mobile Apps.
        paths:
          /clientlist:
            get:
              security:
                - application:
                    - 'mypermission'
              operationId: getData
              parameters:
                - name: querystring
                  in: query
                  required: true
                  schema:
                    type: array
                    items:
                      type: integer
                    minItems: 1
                  style:  pipeDelimited
              responses:
                '200':
                  description: Success
                  schema:
                    type: array
                    items:
                      type: string
              x-amazon-apigateway-integration:
                httpMethod: get
                type: aws
                requestTemplates:
                  {
                    "application/json":
                      "#set($path = $input.params().path)
                       #set($qs = $input.params().querystring)
                      {
                        \"params\": {
                          #foreach($key in $path.keySet())
                          \"$key\": \"$path.get($key)\"
                            #if($foreach.hasNext), #end
                            #end
                        },
                        \"query\": {
                          #foreach($key in $qs.keySet())
                          \"$key\": \"$qs.get($key)\"
                            #if($foreach.hasNext), #end
                            #end
                        },
                        \"body\": $input.json('$'),
                        \"user_id\": \"$context.authorizer.principalId\",
                        \"scope\": \"$context.authorizer.scope\",
                        \"permissions\": \"$context.authorizer.permissions\"
                    }"
                  }
                uri:
                  Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${getClientList.Arn}/invocations

Perhaps I am doing something wrong. Thanks again! -K P.S. I'm wondering if this should be a fresh issue, since it seems to be a discussion of DefinitionBody and not DefinitionUri

[brettstack]

SAM CLI has a bug in the validate command when you don't specify a DefinitionBody or DefinitionUri https://github.com/awslabs/aws-sam-cli/issues/803. It should deploy fine though?

[jlhood]

Discussed internally. Here are some notes:

1. Need to iterate on the spec definition. Having DefinitionUri take both object and string can be confusing, especially since the default behavior for each usage is different (inline vs upload). Alternative suggested is to add a new property for this behavior instead, e.g., DefinitionInclude or something like that.
2. We should test to see if SAM could add a Fn::Include of the uploaded swagger file rather than having to manually download and inline it. If that's possible, it would be simpler to implement. It would also work around the issue where merging the swagger definition into the template body on the client side exposes you to hitting template size limits when using sam deploy.

[akhilkvpv88]

against that DefinitionBody

Will this also get rid of the following error we get while using Auth: "Auth works only with inline Swagger specified in 'DefinitionBody' property"

Currently, this is not working with the workaround you proposed.

[akhilkvpv88]

@brettstack I tried your approach of Inline:true but sam package didn't replace DefinitionUri with DefinitionBody and it didn't replace it with inline swagger. I am using sam version 0.15.0.

[jlhood]

@akhilkvpv88 The Inline: true property was a proposed new feature to SAM, not an existing feature. Sorry for the confusion. We're still determining the best way to address this issue.

[keetonian]

I did some investigating on having SAM add a Fn::Transform block to the template to use AWS::Include. The result is that this won't work without a change in CloudFormation.

Investigation Details:

I created and tested the new DefinitionInclude property in a few different ways. All of these ways failed in the Create Changeset workflow during CFN deploy. I verified the resulting templates that SAM created by uploading them separately to CFN, and the templates deployed after a local transform was run, but failed when the transform was run in the cloud.

First Test

AWSTemplateFormatVersion: '2010-09-09'
Transform:
  - 'AWS::Serverless-2016-10-31'
Parameters:
  StageName:
    Default: 'prod'
    Type: String
Resources:
  LambdaAPIDefinition:
    Type: AWS::Serverless::Api
    Properties:
      StageName: !Ref StageName
      DefinitionInclude:  s3://bucket/swagger.json

This test transforms into the following JSON API definition:

    "LambdaAPIDefinition": {
      "Type": "AWS::ApiGateway::RestApi", 
      "Properties": {
        "Body": {
          "Fn::Transform": {
            "Name": "AWS::Include", 
            "Parameters": {
              "Location": "s3://bucket/swagger.json"
            }
          }
        }
      }
    }
  }, 

This resulted in the following error:

Failed to create the changeset: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: Template Error: Encountered unsupported function: Fn::Transform Supported functions are: [Fn::Base64, Fn::GetAtt, Fn::GetAZs, Fn::ImportValue, Fn::Join, Fn::Split, Fn::FindInMap, Fn::Select, Ref, Fn::Equals, Fn::If, Fn::Not, Condition, Fn::And, Fn::Or, Fn::Contains, Fn::EachMemberEquals, Fn::EachMemberIn, Fn::ValueOf, Fn::ValueOfAll, Fn::RefAll, Fn::Sub, Fn::Cidr]

It looks like CFN wasn't expecting another transform after the SAM transform, so...

Second Test

My second test included having SAM do the same as the previous step as well as change Transform: 'AWS::Serverless-2016-10-31' to Transform: 'AWS::Include' on the Transform output. This resulted in the following error:

Failed to create the changeset: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: Transform AWS::Include failed with: Transform parameter map is empty or does not contain location parameter.

Third Test

I expanded the Transform section to:

Transform:
  - 'AWS::Serverless-2016-10-31'
  - 'AWS::Include'

This should have forced CFN to run the Include transform after SAM (docs), but it didn't, instead failing with

Failed to create the changeset: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: Transform AWS::Include failed with: Transform parameter map is empty or does not contain location parameter.

Next Steps I'll follow up with CloudFormation to see if it would be possible to run the AWS::Include transform after the SAM transform.

[lukebrowell]

Any update on the proposed new feature is there a PR yet?

[praneetap]

We haven't found a simple way to make this work as part of the SAM transform, we are still following up with CloudFormation team. If that doesn't work out, the next best option would be to change the sam cli package command to inline swagger that is referenced in a separate file.

[timoschilling]

@praneetap any updates on this?

[tyler2cr]

Related: https://github.com/aws/aws-cli/pull/3454

[keetonian]

I'd recommend using the AWS::Include transform to have CloudFormation automatically include the OpenApi file in your template for you. You can still reference an external file, like in DefinitionUri, but without the drawback of not being able to resolve CFN intrinsics in it or having SAM integrations not work.

Here's an example:

  MyApi:
    Type: AWS::Serverless::Api
    Properties:
      DefinitionBody:
        Fn::Transform:
          Name: AWS::Include
          Parameters:
            Location: openapi.yaml

[joquijada]

Sorry @Joel-fogue this fell off my radar a bit.

In case you or anyone else is still interested, here is how I've set up my CloudFormation template:

Parameters:
  EnvironmentParameter:
    Type: String
    Default: dev
    AllowedValues:
      - dev
      - prod
    Description: dev, prod
  UserPool:
    Type: String
    Description: Cognito User Pool to use with APIGateway
Resources:
  ApiGateway:
    Type: 'AWS::Serverless::Api'
    Properties:
      Name: !Join [ '', [ !Ref EnvironmentParameter, 'ApiGateway'  ] ]
      StageName: !Ref EnvironmentParameter
      DefinitionBody:
          'Fn::Transform':
            Name: 'AWS::Include'
            # Replace <bucket> with your bucket name
            Parameters:
              Location: !Join [ '', [ 's3://bucketprefixofyourchoice-', !Ref EnvironmentParameter, '/apigatewaydefinition.yaml'  ] ]

In your Swagger template, you can then do things like:

      providerARNs:
      - Fn::Sub: "arn:aws:cognito-idp:us-east-1:123456789:userpool/${UserPool}"

the important step is to have the swagger template copied to an S3 bucket, so my build step for CodeBuild looks something like:

build:
    commands:
      - aws s3 cp ./apigatewaydefinition.yaml s3://myproject-api-"$ENVIRONMENT"/
      - aws cloudformation package .........

Hope this helps, sorry for the late response.

I did a CodeBuild a few days ago with a template.yml to define a Lambda and an API that references a local swagger file, and the local swagger file reference got automatically transformed into a remote file in S3. Maybe a more recent version of the AWS CLI released after this issue got created that supports this feature. Below is my template.yml and buildspect.yml for reference,

template.yml

AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: Test Pipeline Lambda
Resources:
  RuleServiceApi:
    Type: 'AWS::Serverless::Api'
    Properties:
      StageName: prod
      DefinitionBody:
        'Fn::Transform':
          Name: 'AWS::Include'
          Parameters:
            Location: api-files/swagger.yml

  RuleServiceFunction:
    Type: 'AWS::Serverless::Function'
    Properties:
      Handler: com.exsoinn.ie.rule.lambda.RuleService::handleRequest
      Runtime: java8
      CodeUri: rule-api-sample-app-aws-lambda/target/rule-api-sample-app-aws-lambda-jar-with-dependencies.jar
      Description: ''
      Events:
        RuleServiceApi:
          Type: Api
          Properties:
            RestApiId:
              Ref: RuleServiceApi
            Path: /rule-sample-app/evaluate
            Method: POST

buildspec.yml

version: 0.2

phases:
  install:
    runtime-versions:
      java: corretto8
  build:
    commands:
      - mvn clean package assembly:single
      - export BUCKET=codepipline-rule-service-lambda
      - aws cloudformation package --template-file template.yml --s3-bucket $BUCKET --output-template-file outputtemplate.yml
artifacts:
  type: zip
  files:
    - template.yml
    - outputtemplate.yml

[ckabalan]

I can confirm along with @joquijada this does work with our build pipeline too (via Jenkins, though it uses regular AWS CloudFormation/SAM commands).

Interestingly the AWS::Include documentation says this:

Parameters Location The location is an Amazon S3 URI, with a specific file name in an S3 bucket. For example, s3://MyBucketName/MyFile.yaml.

All the examples show S3, and but it does indeed work with a local URI:

template.yaml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  API:
    Type: AWS::Serverless::Api
    Properties:
      StageName: v1
      DefinitionBody:
        Fn::Transform:
          Name: AWS::Include
          Parameters:
            Location: openapi.yaml

It looks like the CLI tools are now smart enough to upload the file to the same S3 bucket and reference it. This was definitely NOT the case in late July 2019 because I had to write a workaround to upload the Swagger spec to S3, then pass the S3 URI to the stack as a parameter for use in the DefinitionBody section.

CloudFormation Service > Select Stack > Template Tab

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  API:
    Type: AWS::Serverless::Api
    Properties:
      StageName: v1
      DefinitionBody:
        Fn::Transform:
          Name: AWS::Include
          Parameters:
            Location: s3://my-s3-bucket/87e477e3dd00b78ba5ca18b9335a6105

[mvanbaak]

It looks like the CLI tools are now smart enough to upload the file to the same S3 bucket and reference it. This was definitely NOT the case in late July 2019 because I had to write a workaround to upload the Swagger spec to S3, then pass the S3 URI to the stack as a parameter for use in the DefinitionBody section.

I can confirm this as well. I have deployed a solution like that multiple times as well. Good to see the tools are now smarter. Time to try it again :)

[dacgray]

I got everything working with this setup - took me a good few hours to figure everything out - so I hope this saves someone some time.

This demos two endpoints each pointing to different functions.

Dev:

sam local start-api

Deploy:

sam build && sam deploy --profile xxx --region xxx

Delete:

aws cloudformation delete-stack --profile xxx --region xxx --stack-name TfTestStack

Template.yaml

AWSTemplateFormatVersion : "2010-09-09"
Transform: AWS::Serverless-2016-10-31

Globals:

  Function:
    Timeout: 5

Resources:

  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: node-world/
      Handler: app.lambdaHandler
      Runtime: nodejs12.x

  HelloMoonFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: node-moon/
      Handler: app.lambdaHandler
      Runtime: nodejs12.x

  API:
    Type: AWS::Serverless::Api
    Properties:
      StageName: Test
      DefinitionBody:
        Fn::Transform:
          Name: AWS::Include
          Parameters:
            Location: swagger-node.yaml

  HelloWorldFunctionPermission:
    Type: AWS::Lambda::Permission
    DependsOn:
      - API
      - HelloWorldFunction
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !Ref HelloWorldFunction
      Principal: apigateway.amazonaws.com

  HelloMoonFunctionPermission:
    Type: AWS::Lambda::Permission
    DependsOn:
      - API
      - HelloMoonFunction
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !Ref HelloMoonFunction
      Principal: apigateway.amazonaws.com

swagger-node.yaml

swagger: '2.0'
info:
  version: 1.0.0
  title: Test API
  description: Test API
paths:
  /node-world:
    get:
      description: Test Hello World
      produces:
        - application/json
      responses:
        "200":
          description: Test API
          schema:
            "$ref": "#/definitions/Message"
      x-amazon-apigateway-integration:
        type: aws_proxy
        httpMethod: POST
        uri:
          Fn::Sub: "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${HelloWorldFunction.Arn}/invocations"
  /node-moon:
    get:
      description: Test Hello Moon
      produces:
        - application/json
      responses:
        "200":
          description: Test API
          schema:
            "$ref": "#/definitions/Message"
      x-amazon-apigateway-integration:
        type: aws_proxy
        httpMethod: POST
        uri:
          Fn::Sub: "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${HelloMoonFunction.Arn}/invocations"
definitions:
  Message:
    type: object
    properties:
      message:
        type: string

samconfig.toml

version = 0.1
[default]
[default.deploy]
[default.deploy.parameters]
stack_name = "TfTestStack"
s3_bucket = "xxx"
s3_prefix = "TfTestStack"
region = "xxx"
profile = "xxx"
confirm_changeset = true
capabilities = "CAPABILITY_IAM"

/node-<world|moon>/app.js

// const axios = require('axios')
// const url = 'http://checkip.amazonaws.com/';
let response;

exports.lambdaHandler = async (event, context) => {
    try {
        // const ret = await axios(url);
        response = {
            'statusCode': 200,
            'body': JSON.stringify({
                message: 'hello xxx',
                // location: ret.data.trim()
            })
        }
    } catch (err) {
        console.log(err);
        return err;
    }

    return response
};

[hawflau]

Thanks @dacgray for providing a working example for the issue.

Closing as there is already a solution. Please use DefinitionBody with AWS::Include.