Parameter count X is greater than max allowed 60.

vicary commented 4 years ago

5th week of headbanging

Totally aware of #15 but eventually find that I was actually hitting another limit --- Parameters. They just happen to have the same limit of 60.

This limit is especially easy to hit when working with serverless-appsync-plugin, where data sources of lambda resolvers, specified as parameters, are created in 1:1 ratio with the lambda themselves.

Tried adding a check in util.js#stackHasRoom, doesn't work, stack.Parameters are empty the whole time.

I suspect that parameters are added after after:aws:package:finalize:mergeCustomProviderResources hook?

Also took a look at serverless-appsync-plugin/index.js, the same hook as above is used, only adds resources and outputs. Parameters only found at the compiled template in deployment bucket.

With enough information I would make a PR, apologies for such an unstructured issue description for now.

dougmoscrop commented 4 years ago

Parameters only found at the compiled template in deployment bucket.

oof that is a pain. I wonder if you could use like Object.defineProperty to define a setter on the resources (Parameters property) and just throw or log the stack trace so you can reverse-engineer where the parameters are coming from as a next step

dougmoscrop commented 4 years ago

oh wait, are you saying only in the bucket literally, not even in .serverless??!

vicary commented 4 years ago

No, not even .serverless.

The JSON files inside has no definition of Parameters, splitting of resources and outputs seems legit thus far.

I honestly would try anything at this stage, will add object setters tmr.

vicary commented 4 years ago

Object proxy shows nothing.

Since my current parameter count is 66, tried to reduce the limit of both resources and outputs to 50 in my fork, it works. At least proven the parameters are in fact directly related to at least one of them.

Not wanting to stop here, but where should I go next?

dougmoscrop commented 4 years ago

What are the parameters? Names, types?

vicary commented 4 years ago

The whole stack mainly goes resolvers, which looks very much like the redacted parameter below.

AppSync resolvers usually repeats until it reaches 60, and it will definitely exceed the limit with data source and role related stuff such as AppSyncLambdaServiceRoleArnParameter, GraphQlApiApiIdParameter... etc.

"AppSyncDataSourcesNestedStack": {
  "Type": "AWS::CloudFormation::Stack",
  "Properties": {
    "Parameters": {
      "[REDACTED]LambdaFunctionArnParameter": {
        "Fn::GetAtt": [
          "FunctionsNestedStack",
          "Outputs.[REDACTED]LambdaFunctionArn"
        ]
      },
      "AppSyncLambdaServiceRoleArnParameter": {
        "Fn::GetAtt": [
          "AppSyncLambdaServiceRole",
          "Arn"
        ]
      },
      "GraphQlApiApiIdParameter": {
        "Fn::GetAtt": [
          "AppSyncNestedStack",
          "Outputs.GraphQlApiApiId"
        ]
      }
    },
    "TemplateURL": {
      "Fn::Join": [
        "/",
        [
          "https://s3.ap-southeast-1.amazonaws.com",
          {
            "Ref": "ServerlessDeploymentBucket"
          },
          "[REDACTED]",
          "cloudformation-template-AppSyncDataSources-nested-stack.json"
        ]
      ]
    }
  },
  "DependsOn": [
    "FunctionsNestedStack",
    "AppSyncLambdaServiceRole",
    "AppSyncNestedStack",
    "FunctionsNestedStack2"
  ]
}

Also attaching my stacks-map.js below,

module.exports = {
  "AWS::AppSync::ApiKey": { destination: "AppSync", allowSuffix: true },
  "AWS::AppSync::DataSource": { destination: "AppSyncDataSources", allowSuffix: true },
  "AWS::AppSync::FunctionConfiguration": { destination: "AppSyncFunctions", allowSuffix: true },
  "AWS::AppSync::GraphQLApi": { destination: "AppSync", allowSuffix: true },
  "AWS::AppSync::GraphQLSchema": { destination: "AppSync", allowSuffix: true },
  "AWS::AppSync::Resolver": { destination: "AppSyncResolvers", allowSuffix: true },
  "AWS::Lambda::Function": { destination: "Functions", allowSuffix: true },
  "AWS::Logs::LogGroup": { destination: "Logs", allowSuffix: true },
};

dougmoscrop commented 4 years ago

The thing I can't get over is that the .serverless side is not parameterized. I would expect it to be a bug in this library (it certainly does not consider parameter count properly or at all).

What other plugins do you have?

vicary commented 4 years ago

My plugins here, just appsync and typescript.

plugins:
  - serverless-dotenv-plugin
  - serverless-plugin-typescript
  - serverless-plugin-optimize
  - serverless-appsync-plugin
  - serverless-plugin-split-stacks

I deployed one more time for staging so I have the most recent templates in .serverless.

I must have missed this before but the data source template in fact has parameters there and the count is 52, still exceed the current output limit of 50.
The template without parameter is LogsNestedStack, it makes #15 without changing resource limit from 200 to 60 which I think is another whole rabbit hole to dig in. Better off focus on parameter first.

tvb commented 4 years ago

I just ran into the same issue and scratching my head..

An error occurred: ApiGatewayMethodNestedStack - Template format error: Parameter count 62 is greater than max allowed 60.

My cloudformation-template-ApiGatewayMethod-nested-stack has many resources which Ref to [REDACTED]LambdaFunctionArnParameter:

    "ApiGatewayMethod[REDACTED]Post": {
      "Type": "AWS::ApiGateway::Method",
      "Properties": {
        "HttpMethod": "POST",
        "ResourceId": {
          "Ref": "ApiGatewayResource[REDACTED]Parameter"
        },
        "RestApiId": {
          "Ref": "ApiGatewayRestApiParameter"
        },
        "Integration": {
          "IntegrationHttpMethod": "POST",
          "Type": "AWS_PROXY",
          "Uri": {
            "Fn::Join": [
              "",
              [
                "arn:",
                {
                  "Ref": "AWS::Partition"
                },
                ":apigateway:",
                {
                  "Ref": "AWS::Region"
                },
                ":lambda:path/2015-03-31/functions/",
                {
                  "Ref": "[REDACTED]LambdaFunctionArnParameter"
                },
                "/invocations"
              ]
            ]
          }
        },
        "MethodResponses": []
      },
      "DependsOn": []
    }

tvb commented 4 years ago

I have temporarily fixed it by setting the following in my stacsk-map.js:

    else if (logicalId.startsWith('ApiGatewayMethodA')) return {
        destination: 'ApiGatewayMethodA'
    };
    else if (logicalId.startsWith('ApiGatewayMethod')) return {
        destination: 'ApiGatewayMethod'
    };

However this adds yet another 3 resources to my parent stack:

Serverless: [serverless-plugin-split-stacks]: Summary: 193 resources migrated in to 36 nested stacks
Serverless: [serverless-plugin-split-stacks]:    Resources per stack:
Serverless: [serverless-plugin-split-stacks]:    - (root): 118

Not long before I run into new problems 😭

morficus commented 4 years ago

I'm running in to the same problem now 😢 has anyone found a good way around this?

edit: after much fiddling with my stacks-map.js I was able to work around the issue (at least for now). what did it for me was combining the AppSync resolver and FunctionConfiguration in to the same stack.

here is what my config looks like:

serverless.yml

  splitStacks:
    perFunction: true
    perType: false
    perGroupFunction: false

split-stack.js

module.exports = {
  'AWS::AppSync::ApiKey': { destination: 'AppSyncApiKey', allowSuffix: true },
  'AWS::AppSync::DataSource': { destination: 'AppSyncDataSources', allowSuffix: true },
  'AWS::AppSync::GraphQLApi': { destination: 'AppSyncApi', allowSuffix: true },
  'AWS::AppSync::GraphQLSchema': { destination: 'AppSyncSchema', allowSuffix: true },
  'AWS::Lambda::Function': { destination: 'LambdaFunctions', allowSuffix: true },
  'AWS::Logs::LogGroup': { destination: 'Logs', allowSuffix: true },
  'AWS::AppSync::FunctionConfiguration': { destination: 'AppSyncResolversAndFns', allowSuffix: true },
  'AWS::AppSync::Resolver': { destination: 'AppSyncResolversAndFns', allowSuffix: true }
}

vicary commented 4 years ago

@morficus This is the first step we all hit before taking another hit when these logical groups are also filled-up.

Root cause is somehow from CloudFormation and its underlying services, moving resources between sub-stacks are not allowed, recreating a resource may sometimes mean deleting a whole database... etc.

You will eventually need to squeeze both resources and outputs under the number of 52 (it was 200 resources and 60 outputs before) to do incremental deploys reliably until you hit the 200 * 52 resource limit.

I ended up forked and changed the numbers at utils.js:195, using git-style-path for version in package.json to directly reference that change.

Mudassir-23 commented 3 years ago

By splitting stacks in the right way through stacks-map.js, this issue can be handled to a good extent https://medium.com/faun/overcoming-cloud-formation-limit-hurdles-with-the-serverless-framework-and-app-sync-4ed91d4ffee9

vicary commented 3 years ago

Note that when you finally go production and add provisionedConcurrency to resolvers, the count will be at least tripled, having AWS::Lambda::Version and AWS::Lambda::Alias on top of the original AWS::Lambda::Function. Parameters and Outputs will explode out of control because they're added after stack nesting.

thadeu commented 3 years ago

By splitting stacks in the right way through stacks-map.js, this issue can be handled to a good extent https://medium.com/faun/overcoming-cloud-formation-limit-hurdles-with-the-serverless-framework-and-app-sync-4ed91d4ffee9

@Mudassir-23 If I adjust an existing stacks-map.js this way depends on recreating the resource?

My stacks-map.js looks like this

module.exports = {
    'AWS::AppSync::ApiKey': { destination: 'AppSync', allowSuffix: true },
    'AWS::AppSync::DataSource': { destination: 'AppSync', allowSuffix: true },
    'AWS::AppSync::FunctionConfiguration': { destination: 'AppSync', allowSuffix: true },
    'AWS::AppSync::GraphQLApi': { destination: 'AppSync', allowSuffix: true },
    'AWS::AppSync::GraphQLSchema': { destination: 'AppSync', allowSuffix: true },
    'AWS::AppSync::Resolver': { destination: 'AppSync', allowSuffix: true },
    'AWS::CloudWatch::Alarm': { destination: 'Alarms', allowSuffix: true,  force: true}
 }

I may not recreate my stack! :(

dougmoscrop / serverless-plugin-split-stacks

Parameter count X is greater than max allowed 60. #91