V0 Goals

[austencollins]

The workflow of larger teams vary, but the general problem teams are running into is this:

• Most developers in a team, working on the same project, will have their own stage which they use while developing. When a new developer joins, they git clone the serverless project, run project init, then create their own stage. • However, some project stages are shared across developers and CI/CD systems (e.g., test, beta, production). • For the developers and CI/CD systems who are sharing stages (e.g., test, beta, production), they need a secure way to sync/pull the meta data for those shared stages, outside of their version control system (e.g., Git).

We need a Serverless Plugin that allows them to do that. It should simply sync the variable files in the _meta/variables folder with copies located on an S3 bucket. Here's how it should work in more detail:

• [x] It should introduce a single meta sync command. This syncs project, stage, and stage/region variables found in Serverless Project's _meta folder, depending on how it's called.
• [x] In s-project.json a custom.meta property should contain the following settings:

"meta": {
   "name": , // Name of the S3 bucket that holds the meta data.  Project variables can be used here.
   "region": // The region the S3 bucket resides in.  Project variables can be used here.
}

• [x] When called, the first thing that happens is a find or create option for the S3 bucket that is specified in the custom.meta data. This ensures that if different AWS accounts are used with different stages, the bucket is auto-created if it does not yet exist, without bothering the developer to do this manually.
• [x] When used programmatically, MetaSync should simply overwrite the local project variables with the variables on the S3 Bucket.
• [x] When used interactively, sls meta sync should download the variable files on S3 and prompt the user with a select option, whenever it finds a conflict. The first option should be the local option, the second option should be the remote option (on the S3 bucket). When all conflicts are resolved, the action will then persist the updated variables to the project, then replace the file with the updated one on S3.
• [x] MetaSync with a stage option, should sync the s-variables-stage.json file.
• [x] MetaSync with a stage and region option, should sync the s-variables-stage-region.json file

[erikerikson]

maybe MetaSync without options, should sync the s-variables-common.json file. MetaSync with a stage option, should sync the s-variables-common.json and s-variables-stage.json files. MetaSync with a stage and region option, should sync the s-variables-common.json, s-variables-stage.json, and s-variables-stage-region.json files.

after all, the stage scoped variables include the s-variables-common.json defined variables, et cetera. Not sure I've fully thought that through.

[erikerikson]

Our planned but unimplemented CI/CD strategy is to have a stage per branch rather than per developer. We see the branch as the macro unit of testing need.

As for what to do about sensitive information, we are currently taking the position that the variable should always remain insensitive. After all, although not in a source control system, it will be on multiple disks as a result of Amazon's handling so it would have to be encrypted to be in the environment securely. We accomplish this by placing only a pointer to the sensitive asset in our _meta configuration files. This has advantages like allowing _meta/resources and _meta/variables to be checked in. Another important advantage for the paranoid among us is providing a control plane for allowing engineers to use dev/test certificates but not the production ones. This leads to an easily solvable potential race condition to ensure loading completion (of course) but the many advantages outweigh that small amount of complexity and additional cold start latency.

To be completely honest, we're still working through the "what is driven by SLS as opposed to our scripts?" question. For example, trying to plug CI/CD into SLS seems like folly as our CI/CD is the orchestrator that knows how to direct SLS. Trying to make SLS the tool for everything seems to be asking for trouble, at best, for at least the short term. That said, I believe there will be teams that will want everything driven by SLS and this use case seems a valid choice for an organization, I'm just not sure of the priority of this as a feature.

[austencollins]

@erikerikson Please check out the new demo of this plugin and we can discuss all of this tonight. Thanks as always for your input :)