dougmoscrop / serverless-plugin-split-stacks

A plugin to generate nested stacks to get around CloudFormation resource/parameter/output limits
297 stars 68 forks source link

No difference between perLambda and perGroup options #124

Open jagadish432 opened 4 years ago

jagadish432 commented 4 years ago

I have tried deploying my lambda functions (testing) using perLambda and perGroup (one at a time), I haven't seen any difference because both of them creating a separate stacks for each lambda functions defined in yml file.

Can someone please let me know how to distinguish and use it at their best?

tonivdv commented 4 years ago

Hi @jagadish432 ,

If you already deployed the first time with "PerFunction" then indeed you will not see any difference as it keeps (if I'm not mistaken) the same split strategy as per:

There are built-in migration strategies that can be turned on or off as well as defining your own custom migrations. It is a good idea to select the best strategy for your needs from the start because the only reliable method of changing strategy later on is to recreate the deployment from scratch

So if you want to see how it goes with another strategy you'll have to recreate the deploy from scratch. What I do to test the different strategies is executing the "package" command. There you'll see that it differs.

Hope it helps.

dougmoscrop commented 4 years ago

@tonivdv is spot on - moving resources around in CloudFormation was at the time I wrote this almost impossible; there is now support in the CFn API to do some import/migration of resources.. but it would be tricky. I would welcome any PR that could add this.. even if it had to be done one resource at a time, something like serverless split-stacks migrate --resource={logicalId} --to-stack={stackName}

Edit:

Thinking mostly off the top of my head, it might work like this (and it would likely be a multi-step operation)

Edit2: Actually, if we can do all that, we can probably just treat all forced migrations as a subset of this problem, no need for specifying a resource id?

I don't think we would do this as part of a normal deploy, but maybe the plugin could print out a warning that says there are resources that have a 'pending migration' or something like that, and you run the split-stacks migrate helper to do this. My guess is this type of operation would take a while so it would not be reasonable to do it as part of the normal deploy in CI/CD, although again, an option like --do-migrations could make it seamless.

I've got to be honest, I am super busy with my day job, receive no compensation for any of this and I have a half dozen startups I wish I could pursue.. I need help!

jagadish432 commented 4 years ago

Hi @jagadish432 ,

If you already deployed the first time with "PerFunction" then indeed you will not see any difference as it keeps (if I'm not mistaken) the same split strategy as per:

There are built-in migration strategies that can be turned on or off as well as defining your own custom migrations. It is a good idea to select the best strategy for your needs from the start because the only reliable method of changing strategy later on is to recreate the deployment from scratch

So if you want to see how it goes with another strategy you'll have to recreate the deploy from scratch. What I do to test the different strategies is executing the "package" command. There you'll see that it differs.

Hope it helps.

@tonivdv ,

I have removed the stacks and alos delete .serverless folder as well, and the tried but still, I couldn't see any difference. (please find the screenshots attached".

Capture1 Capture2

if possible, can you please let me know how to separate lambda functions using perGroupFunction ? like "how the plugin distinguishes? based on what parameter?"

Here is my serverless.yml file code

` service: ${self:custom.variables.application}-ABC

provider: name: aws runtime: python3.6 region: us-east-2 stage: dev timeout: 30

apiGateway: restApiId: 'Fn::ImportValue': 'apigateway-${self:custom.variables.application}-${opt:stage, self:provider.stage}' restApiRootResourceId: 'Fn::ImportValue': 'apigateway-root-${self:custom.variables.application}-${opt:stage, self:provider.stage}'

custom: splitStacks: nestedStackCount: 20 perFunction: false perType: false perGroupFunction: true variables: ${file(../serverless.env.yml)}

functions: abc1: handler: handler.abc1 events:

plugins:

tonivdv commented 4 years ago

Hi @jagadish432

I'm only using this plugin for a couple of days so I wouldn't know how the internals work. But on my project where I do have +10 functions I did see a difference in perType, PerFunction and PerFunctionGroup

Now since your serverless has only 2 functions it's possible that it doesn't change a lot. But normally 'naming wise' it does change if you go from perFunction to perFunctionGroup ... as in the former it uses the name of the function to create a template while in the other it generates a name with numbers (as in your screenshot)

For the perType I noticed that it simply groups per type resources. Hence all my functions were put into one template.

Maybe @dougmoscrop can share some more on it, and maybe this can also evolve to improve a bit the documentation as it was for me also not clear what the differences were.

jagadish432 commented 4 years ago

Hi @jagadish432

I'm only using this plugin for a couple of days so I wouldn't know how the internals work. But on my project where I do have +10 functions I did see a difference in perType, PerFunction and PerFunctionGroup

Now since your serverless has only 2 functions it's possible that it doesn't change a lot. But normally 'naming wise' it does change if you go from perFunction to perFunctionGroup ... as in the former it uses the name of the function to create a template while in the other it generates a name with numbers (as in your screenshot)

For the perType I noticed that it simply groups per type resources. Hence all my functions were put into one template.

Maybe @dougmoscrop can share some more on it, and maybe this can also evolve to improve a bit the documentation as it was for me also not clear what the differences were.

@tonivdv , yes , the doc was not so clear. @dougmoscrop , it would be helpful if you please update the documentation in you're free time, to understand the diff between perLambda and perGroup .

stephde commented 2 years ago

The docs are really unclear about how this perFunctionGroup works. I would also like to know if there is a way to predict or define manually which functions will be put into the same group? Is it for example based on the function names or resources they use? I imagine its not pure random. Would also be awesome to have an option to tag functions manually and have them put into groups based on the tag.

The code looks to me like it just randomly distributes the function by hashing their names to the number of stacks that are defined in the configuration. Can anyone confirm this?

https://github.com/dougmoscrop/serverless-plugin-split-stacks/blob/eb92ae90562cfdeed12c0dd13ba8c2cf37ba7b9d/lib/migration-strategy/per-group-function.js#L28-L30