search

dougmoscrop / serverless-plugin-split-stacks

A plugin to generate nested stacks to get around CloudFormation resource/parameter/output limits
299 stars 71 forks source link

Using perFunction, Successfully deployed first time. But can't second time. #33

Closed aki-webii closed 6 years ago

aki-webii commented 6 years ago

Encountered the error like blow.

> SLS_DEBUG=* sls deploy --stage dev -v

...

Serverless: Packaging service...
Serverless: Remove /MY_PRJ_DIR/.webpack
Serverless: Invoke aws:package:finalize

  Serverless Error ---------------------------------------

  Rate exceeded

  Stack Trace --------------------------------------------

ServerlessError: Rate exceeded
    at BbPromise.fromCallback.catch.err (/Users/mikami/.nodebrew/node/v9.7.0/lib/node_modules/serverless/lib/plugins/aws/provider/awsProvider.js:258:33)
From previous event:
    at persistentRequest (/Users/mikami/.nodebrew/node/v9.7.0/lib/node_modules/serverless/lib/plugins/aws/provider/awsProvider.js:247:13)
    at doCall (/Users/mikami/.nodebrew/node/v9.7.0/lib/node_modules/serverless/lib/plugins/aws/provider/awsProvider.js:205:9)
    at BbPromise (/Users/mikami/.nodebrew/node/v9.7.0/lib/node_modules/serverless/lib/plugins/aws/provider/awsProvider.js:216:14)
From previous event:
    at persistentRequest (/Users/mikami/.nodebrew/node/v9.7.0/lib/node_modules/serverless/lib/plugins/aws/provider/awsProvider.js:203:38)
    at Object.request.requestQueue.add [as promiseGenerator] (/Users/mikami/.nodebrew/node/v9.7.0/lib/node_modules/serverless/lib/plugins/aws/provider/awsProvider.js:237:49)
    at Queue._dequeue (/Users/mikami/.nodebrew/node/v9.7.0/lib/node_modules/serverless/node_modules/promise-queue/lib/index.js:153:30)
    at /Users/mikami/.nodebrew/node/v9.7.0/lib/node_modules/serverless/node_modules/promise-queue/lib/index.js:160:26
From previous event:
    at Queue._dequeue (/Users/mikami/.nodebrew/node/v9.7.0/lib/node_modules/serverless/node_modules/promise-queue/lib/index.js:155:18)
    at /Users/mikami/.nodebrew/node/v9.7.0/lib/node_modules/serverless/node_modules/promise-queue/lib/index.js:109:18
From previous event:
    at Queue.add (/Users/mikami/.nodebrew/node/v9.7.0/lib/node_modules/serverless/node_modules/promise-queue/lib/index.js:94:16)
    at AwsProvider.request (/Users/mikami/.nodebrew/node/v9.7.0/lib/node_modules/serverless/lib/plugins/aws/provider/awsProvider.js:237:39)
    at listStackResources (/MY_PRJ_DIR/node_modules/serverless-plugin-split-stacks/lib/utils.js:261:23)
    at ServerlessPluginSplitStacks.getStackSummary (/MY_PRJ_DIR/node_modules/serverless-plugin-split-stacks/lib/utils.js:273:12)
    at Promise.all.nestedStacks.map.stack (/MY_PRJ_DIR/node_modules/serverless-plugin-split-stacks/lib/migrate-existing-resources.js:19:21)
    at Array.map (<anonymous>)
    at getStackSummary.catch.then.then.nestedStacks (/MY_PRJ_DIR/node_modules/serverless-plugin-split-stacks/lib/migrate-existing-resources.js:15:39)
    at runCallback (timers.js:763:18)
    at tryOnImmediate (timers.js:734:5)
    at processImmediate (timers.js:716:5)
    at process.topLevelDomainCallback (domain.js:101:23)
From previous event:
    at ServerlessPluginSplitStacks.getCurrentState [as migrateExistingResources] (/MY_PRJ_DIR/node_modules/serverless-plugin-split-stacks/lib/migrate-existing-resources.js:14:6)
    at Promise.resolve.then (/MY_PRJ_DIR/node_modules/serverless-plugin-split-stacks/split-stacks.js:54:24)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:118:7)

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           9.7.0
     Serverless Version:     1.26.1

I have 54 funcitons which is combination of Lambda and API Gateway, and want to create more functions. So decided to migrate from perType to perFunction strategy. Using perFunction splitting from today. First deployment is successfully done. After that, I modify my code and try to update my stacks by sls deploy, but encountered the error.

Is this AWS's API limit exceeded? Because this plugin is invoking CF's listStackResources API per nested stack at here. I have 54 nested stack, so doing 54 API request in parallel.

Any ideas how can I fix? or should I not use perFunction yet?

environemnt serverless v1.26.1 serverless-plugin-split-stacks latest master branch other plugin: serverless-webpack v4.2.0

aki-webii commented 6 years ago

I'M SORRY! I didn't see the lower part of README.

This example is for the project using perFunction. Isn't it?

const ServerlessPluginSplitStacks = require('serverless-plugin-split-stacks');

ServerlessPluginSplitStacks.resolveMigration = function (resource, logicalId, serverless) {
  if (logicalId.startsWith("Foo")) return { destination: 'Foo' };

  // Fallback to default:
  return this.stacksMap[resource.Type];
};

I tried this configure, but there are still problem. I added some configures like below.

const ServerlessPluginSplitStacks = require('serverless-plugin-split-stacks');

ServerlessPluginSplitStacks.resolveMigration = function(
  resource,
  logicalId,
  serverless
) {
  if (logicalId.startsWith('account')) return { destination: 'Account' };
  if (logicalId.startsWith('programs')) return { destination: 'Programs' };
  ...

  // Fallback to default:
  return this.stacksMap[resource.Type];
};

then, encountered new error.

Serverless: Packaging service...
Serverless: Remove /Users/mikami/Projects/ap2/client-api/.webpack

  Type Error ---------------------------------------------

  Cannot read property 'AWS::Logs::LogGroup' of undefined

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           9.7.0
     Serverless Version:     1.26.1

Is this problem occurred by combination of split plugin and webpack plugin? If so, I will change my project build&deploy architecture. or is this caused by something wrong configure of split plugin?

aki-webii commented 6 years ago

Sorry, I solved my problem myself.

I misunderstood. Correctly, logicalId.startsWith() based custom split is for perType split strategy. Don't set the custom.splitStacks.perFunction flag on serverless.yml as true, and make configures as below, my stacks are deployed as I expected.

const ServerlessPluginSplitStacks = require('serverless-plugin-split-stacks');

ServerlessPluginSplitStacks.resolveMigration = function(
  resource,
  logicalId,
  serverless
) {
  if (logicalId.startsWith('account')) return { destination: 'Account' };
  if (logicalId.startsWith('programs')) return { destination: 'Programs' };
  ...

  // Fallback to default:
  return this.stacksMap[resource.Type];
};

However, I think the first problem on this issue is still remaining. Many functions(for example, my 54 functions) can not be managed by this plugin with perFunction split strategy.(The second deployment may be fail.) Of course, perFunction split strategy is newest feature, and still under development. I look forward to use this feature smartly, in near future. 😃

Sorry for the redundant issue, and thank you for your nice plugin!

dougmoscrop commented 6 years ago

The RateExceeded seems like a fair bug, and we could probably do something insidethe summary to reduce the rate that we call. I will reopen this and take a look next week.