DependsOn AWS::Serverless:Api does not wait for all resources

[neonsamurai]

Background

I try deploying a serverless application which has a couple of Lambdas, an API gateway and I want to lock it behind an API key.

When sending the definition to cloudformation with aws cloudformation deploy I notice that DependsOn does wait on the RestApi resource to be ready, but it does not wait on the AWS:ApiGateway::Deployment resource which also is created by SAM (but transparently).

My SAM.yaml for reference:

AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: Some API key protected serverless application
Parameters:
  targetStage:
    Description: Define stage to which Lambdas/API Gateways should be deployed.
    Type: String
    Default: dev
    AllowedValues:
      - dev
      - test
      - prod
    ConstraintDescription: Only stages dev, test, prod are allowed
Globals:
  Function:
    Runtime: nodejs6.10
    MemorySize: 128
    Timeout: 6
Resources:
  createFunction:
    Type: 'AWS::Serverless::Function'
    Properties:
      Handler: create.handler
      CodeUri: ../dist/deployment_package.zip
      FunctionName: !Sub "${AWS::StackName}-createCertificate-${targetStage}"
      Runtime: nodejs6.10
      Description: Create or overwrite an IoT certificate with given params.
      Role: 'arn:aws:iam::1234:role/my-role'
  deleteFunction:
    Type: 'AWS::Serverless::Function'
    Properties:
      Handler: delete.handler
      CodeUri: ../dist/deployment_package.zip
      FunctionName: !Sub "${AWS::StackName}-deleteCertificate-${targetStage}"
      Runtime: nodejs6.10
      Description: Delete an IoT certificate with given params.
      Role: 'arn:aws:iam::1234:role/my-role'
  restApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: !Ref targetStage
      DefinitionBody:
        swagger: "2.0"
        info:
          version: "2017-11-09T13:59:26Z"
          title: !Sub "${AWS::StackName}-api-${targetStage}"
        basePath: !Sub "/${targetStage}"
        schemes:
          - "https"
        paths:
          /certificate:
            post:
              responses: {}
              security:
              - api_key: []
              x-amazon-apigateway-integration:
                uri: !Sub "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${createFunction.Arn}/invocations"
                passthroughBehavior: "when_no_match"
                httpMethod: "POST"
                type: "aws_proxy"
            delete:
              responses: {}
              security:
              - api_key: []
              x-amazon-apigateway-integration:
                uri: !Sub "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${deleteFunction.Arn}/invocations"
                passthroughBehavior: "when_no_match"
                httpMethod: "POST"
                type: "aws_proxy"
        securityDefinitions:
          api_key:
            type: "apiKey"
            name: "x-api-key"
            in: "header"
        x-amazon-apigateway-binary-media-types:
          - "application/octet-stream"
  apiKey:
    Type: "AWS::ApiGateway::ApiKey"
    DependsOn:
      - restApi
    Properties:
      Name: !Sub "${AWS::StackName}-apiKey-${targetStage}"
      Description: !Sub "API key for: ${AWS::StackName} / ${targetStage}"
      Enabled: "true"
      StageKeys:
        - RestApiId: !Ref restApi
          StageName: !Ref targetStage

Expected behaviour

DependsOn used to wait on a AWS::Serverless:Api should wait on all resources created by SAM for the API.

Actual behaviour

This causes CloudFormation to throw a CREATE_FAILED | AWS::ApiGateway::ApiKey | apiKey | Invalid stage identifier specified error during deployment, since the stage which I want to associate with the API key does not exist yet and I have no means of referencing the (future) stage or referencing the AWS::ApiGateway::Deployment in the DependsOn.

Thoughts

The only workaround I can think of for now is to build all AWS::Serverless::Api component resources (RestApi, Deployment, ApiKey) by hand and use DependsOn on the respective resource identifiers.

[michaelj-smith]

I have seen similar behavior. Thanks for the detailed write-up!

[neonsamurai]

So I spend basically the whole day figuring out how to deploy my stack. I found an okay workaround by inspecting the stack log in the CloudFormation console. I noticed that the stage which is associated with the deployment is reliably created last and that this stage always has the name ${AWS::StackName}Stage so that I can reliably use DependsOn with that stage. This of course will only work as long as the naming convention within SAM won't change. So I hope we can get a fix soon.

[jfuss]

@neonsamurai Naming convention will never change. If we change it, all resources will be recreated which is something we never want to happen.

We also documented them here

[jfuss]

So just caught up on the complete issue here.

We should support some way to create ApiKeys. I thought this was part of #248 but doesn't look like it made it there. Maybe it should be?

Maybe another way to help in solving this would be to do something similar to what we did to surface Versions and Aliases for AWS::Serverless::Function (Ref: FucntionLocicalId.Version). I think that would help in being able to make DependsOn for resources we generate but not sure how this really interacts and works outside the the AWS::Serverless::* Types (it's doable but think there is some additional work to make this supported).

The suggestion you made about relying on the generated LogicalIds is the way to go (at least for now). I know of others that have suggested the same thing on other issues here. We have fully documented the generated resources we create here, so it is safe to depend on these naming conventions. Make sure to follow the docs, the stage logicalId is <AWS::Serverless::Api LogicalId>Stage not based on the AWS::StackName. So in your example it would be restApi<whatever Ref: targetStage resolves to>Stage.

Note: AWS::Serverless::API (explicit) and AWS::Serverless::Function with an API Event type have different generates of resources. Please consult the documentation for references.

[michaelj-smith]

This is helpful. Thank you for the docs! That provides a way to work around this. For the long term, I believe it would be best to auto-generate additional DependsOn properties for the auto-generated resources. So, the sam package step would modify any other resources that depend on the SAM resources, and we'd get package output like:

  apiKey:
    Type: "AWS::ApiGateway::ApiKey"
    DependsOn:
      - restApi
      - restApidevStage
      - restApiDeploymentSHA

I recognize this may be difficult, so I submit it an idea only.

[neonsamurai]

A little thing I noticed in the docs referenced by @jfuss :

It says the logical ID of the stage resource would be: <AWS::Serverless::Api.LogicalId><AWS::Serverless::Api.StageName>Stage whereas it appears from the CloudFormation console it is just <AWS::Serverless::Api.LogicalId>Stage as @jfuss mentioned in his post

[sanathkr]

So, both of these are true. If StageName is a straight up string, then it will be <ID><StageName>Stage. If it is an intrinsic function, it will be <ID>Stage. This is because SAM cannot reliably resolve intrinsic functions. SAM runs outside of CloudFormation where the actual intrinsic function resolution happens.

[manishverma18]

@neonsamurai: I am also stuck with same problem Can u please guide me

I can reliably use DependsOn with that stage. This of course will only work as long as the naming convention within SAM won't change. So I hope we can get a fix soon. Or share On what actually u depends on with the snippet

[caliniliescu]

Is this issue going to be addressed? It's been open for almost a year now and it's a roadblock in using SAM template when you want to secure your API endpoints with Cognito.

[caliniliescu]

Is this issue going to be addressed? It's been open for almost a year now and it's a roadblock in using SAM template when you want to secure your API endpoints with Cognito.

I found that the problem was being caused by bad indentation of swagger definition written inline in my yml sam template. The error i was getting from cloudformation made me think it was a dependency issue.

[brettstack]

Does the provided workaround not work for you?

[judahb]

@neonsamurai Naming convention will never change. If we change it, all resources will be recreated which is something we never want to happen.

We also documented them here

This link doesn't work. Can you please provide an updated link?

[keetonian]

Updated link: https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst

[alp-garcia]

So I spend basically the whole day figuring out how to deploy my stack. I found an okay workaround by inspecting the stack log in the CloudFormation console. I noticed that the stage which is associated with the deployment is reliably created last and that this stage always has the name ${AWS::StackName}Stage so that I can reliably use DependsOn with that stage. This of course will only work as long as the naming convention within SAM won't change. So I hope we can get a fix soon.

I've created a workaround solution based on @neonsamurai comments and seems it's working properly. So, I have on my template.yaml something similar to:

MyFunction:
  Type: AWS::Serverless::Function
  Properties:
    Events:
      MyFunctionApi:
        Type: Api
        Properties:
          Path: "/functions"
          Method: POST
          RestApiId: !Ref MyApi  

MyApi:
  Type: "AWS::Serverless::Api"
  Properties:
    StageName: dev 
    Cors: "'*'"  

I was expecting the full API resources created so I would be able to create a "base path mapping" as below:

MyApiBasePathMapping:
    Type: "AWS::ApiGateway::BasePathMapping"
    DependsOn:
      - MyFunctionMyFunctionApiPermissionStage
    Properties:
      BasePath: "services"
      DomainName: "www.example.com"
      RestApiId: !Ref MyApi
      Stage: dev

The AWS::ApiGateway::BasePathMapping expects the API Stages created to work properly. So I realise the stage is created with the following structure after checking on the stack events:

{Function.LogicalId}{Function.Events.LogicalId}PermissionStage

Hopefully, there's a better and more reliable way to implement it soon.

[razhamma]

@jfuss for AWS::Serverless::Api resource now supports getting Stage and Deployment using following:

• !Ref: MyApi.Stage
• !Ref: MyApi.Deployment

However, issue still persists as DependsOn attribute doesn't support usage of an intrinsic function(Ref in our case) for any of its value(s).

Possible approaches towards resolution might include:

1. Let CloudFormation mark Serverless::API complete only if all the associated resources (RestApi,Stage,Deployment) have finished their creation. Not a good approach(even if possible) as AWS::ApiGateway::RestApi resource has it own Lifecycle and CloudFormation actually has nothing to do with Serverless type resources lifecycle but translated/transformed CloudFormation specific resources.
2. Adding support/exception in DependsOn attribute for Serverless type resources so that !Ref can be used to get Stage/Deployment logical Ids.

[azarboon]

I have encountered this issue too. It's very annoying. It's unfortunate that after a year still the problem. Solution provided by @sanathkr and @jfuss sometimes works and sometimes doesn't. My workaround was to depends my resource on a function (which is itself dependent on api gatewqay). Here is an example:

  MyACLassociation:
    Type: AWS::WAFRegional::WebACLAssociation
    DependsOn: ALambdaFunctionWhichIsTriggeredByApiGateway
    Properties: 
      ResourceArn: AnaRN
      WebACLId: !Ref MyWebACL

[ShreyaGangishetty]

@azarboon you can use a reference to the api and the Stage which will add an implicit dependsOn. Here is an example of how to use an api key:

  MyFirstApiKey:
    Type: AWS::ApiGateway::ApiKey
    DependsOn:
      - MyUsagePlan
    Properties:
      Enabled: true
      StageKeys:
        - RestApiId:
            Ref: MyApi
          StageName:
            Ref: MyApi.Stage

Here is another example for using BasePathMapping:

MyApiBasePathMapping:
    Type: "AWS::ApiGateway::BasePathMapping"
    Properties:
      BasePath: "services"
      DomainName: "www.example.com"
      RestApiId: !Ref MyApi
      Stage: !Ref MyApi.Stage

Please let us know if this works for you

[azarboon]

@azarboon you can use a reference to the api and the Stage which will add an implicit dependsOn. Here is an example of how to use an api key:

  MyFirstApiKey:
    Type: AWS::ApiGateway::ApiKey
    DependsOn:
      - MyUsagePlan
    Properties:
      Enabled: true
      StageKeys:
        - RestApiId:
            Ref: MyApi
          StageName:
            Ref: MyApi.Stage

Here is another example for using BasePathMapping:

MyApiBasePathMapping:
    Type: "AWS::ApiGateway::BasePathMapping"
    Properties:
      BasePath: "services"
      DomainName: "www.example.com"
      RestApiId: !Ref MyApi
      Stage: !Ref MyApi.Stage

Please let us know if this works for you

Hi,

Thanks for reply but I'm afraid it doesn't work for my usecase. I cannot use intrinsic function (MyApi.Stage) in DependsOn, as far as I know

[azarboon]

I have encountered this issue too. It's very annoying. It's unfortunate that after a year still the problem. Solution provided by @sanathkr and @jfuss sometimes works and sometimes doesn't. My workaround was to depends my resource on a function (which is itself dependent on api gatewqay). Here is an example:

MyACLassociation: Type: AWS::WAFRegional::WebACLAssociation DependsOn: ALambdaFunctionWhichIsTriggeredByApiGateway Properties: ResourceArn: AnaRN WebACLId: !Ref MyWebACL

Update: seems my workaround fails also. Other workaround is this: deploy the stack without the dependent resource (i.e. AWS::WAFRegional::WebACLAssociation), then add the dependent resource and redeploy the stack. Obviously this is an annoying workaround. Looking forward the SAM team to fix this issue ASAP.

[tomislacker]

Another sad workaround that I've seen function is to take in a parameter like IsNotFirstRun, setup a Conditions on it, and then set a Condition on the resource on that conditional.

[keetonian]

@azarboon If a resource has a reference to another resource, CFN understands that it depends on that other resource and you don't need to add the explicit "DependsOn" to it.

[azarboon]

@azarboon If a resource has a reference to another resource, CFN understands that it depends on that other resource and you don't need to add the explicit "DependsOn" to it.

That's how it should be and that's what I expected; but unfortunately AWS::Serverless::Api doesn't support this model. Despite it creates api stage and deployment; but when a resource of type AWS::Serverless::Api is in "done" status, it doesn't mean that "stage" is created. Yes, that really s..... This is what I experienced and what AWS Support investigated & confirmed too.

[praneetap]

@azarboon Just to make sure I understand, are you saying !Ref Api.Stage is not working? Or are you just referencing the Api !Ref Api? If you can paste your template here, we can try to repro on our end.

[azarboon]

@azarboon Just to make sure I understand, are you saying !Ref Api.Stage is not working? Or are you just referencing the Api !Ref Api? If you can paste your template here, we can try to repro on our end.

I tried following combination (third one was suggested by AWS support) but that didn't work either. As far as I know, for dependson, I must not use !Ref and just need to use logical name of resource (correct me if I'm wrong- I'm not next to my template and have memory glitch :D ):

DependsOn: LogicalNameOfApiResource DependsOn: LogicalNameOfApiResource + Stage DependsOn: LogicalNameOfApiResource.Stage

[praneetap]

@azarboon Look at this https://github.com/awslabs/serverless-application-model/issues/313#issuecomment-368088946 I think you are missing stage name. So try - DependsOn: LogicalNameOfApiResource + StageName + 'Stage'

[azarboon]

@azarboon Look at this #313 (comment) I think you are missing stage name. So try - DependsOn: LogicalNameOfApiResource + StageName + 'Stage'

Thanks. I'll try this and let you know if it works or not.

[julianpitt]

This seemed to work for me

Parameters:
    StageName:
    Default: 'dev'
    Type: String

Resources:
  MyLambdaFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: mylookupidentity
      CodeUri: my_lambda.zip
      Events:
        APIGateway:
          Type: Api
          Properties:
            Path: /myfn
            Method: post
            RestApiId: !Ref APIGateway
  APIGatewayKey:
    Type: AWS::ApiGateway::ApiKey
    DependsOn:
      - APIGatewayStage #LogicalId for API Gateway + 'Stage'
    Properties:
      Name: myAPI-api-key
      Enabled: true
      StageKeys:
        - RestApiId: !Ref APIGateway
          StageName: !Ref StageName

  APIGateway:
    Type: AWS::Serverless::Api
    Properties:
      Name: myAPI
      StageName: !Ref StageName
      MethodSettings:
      - HttpMethod: "*"
        LoggingLevel: INFO
        ResourcePath: "/*"
      DefinitionBody: 
        Fn::Transform:
          Name: AWS::Include
          Parameters:
            Location: docs/api.yaml

[ivoanjo]

Hey @ShreyaGangishetty is this issue really fixed? Could you doublecheck?

[umaragu]

The bold lines fixed the issue for me - ApiId: !Ref ComputeApi Stage: !Ref ComputeApi.Stage

UsagePlan:
    Type: 'AWS::ApiGateway::UsagePlan'
    DependsOn:
      - ComputeApi
    Properties:
      ApiStages:
        - ApiId: !Ref ComputeApi
          Stage: !Ref ComputeApi.Stage
      Description: Customer dubdub usage plan
      Quota:
        Limit: 5000
        Period: MONTH
      Throttle:
        BurstLimit: 200
        RateLimit: 100
      UsagePlanName: Plan_dubdub

[nCubed]

@umaragu how did you come up with Stage: !Ref ComputeApi.Stage? Is the Reference documented anywhere?

Just spent a day trying to understand why my stack kept failing even though I had all the DependsOn identified in the UsagePlan; swapping out the hard coded Stage with the !Ref resolved the issue. Big ol virtual high-five man!

[hoffa]

@umaragu how did you come up with Stage: !Ref ComputeApi.Stage? Is the Reference documented anywhere?

Just spent a day trying to understand why my stack kept failing even though I had all the DependsOn identified in the UsagePlan; swapping out the hard coded Stage with the !Ref resolved the issue. Big ol virtual high-five man!

See https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-specification-generated-resources-api.html; it documents which resources are generated and how they're referenceable.