Allow to create reusable cross-stage deployment ready build artifacts

[HyperBrain]

With the new package/deploy semantics, the builds and deployments can be distributed on different servers. The created artifacts can then be deployed to AWS (using the CF template created in the build step).

Normally you construct your services, so that some properties are set by variables defined in the serverless.yml that are set depending on the service target (e.g. stage or region). A prominent example for that is the setting of Lambda environment variables.

Imagine you have the following service definition:

service: stashimi-bot-dash

provider:
  name: aws
  runtime: nodejs6.10
  stage: dev
  region: us-east-1
  ...

  environment:
    MY_VAR: ${self:custom.${self:custom.stage}.myVar}
    ...

custom:
  stage: ${opt:stage, self:provider.stage}
  ...
  dev:
    myVar: "My-3rdParty-Auth-Id-For-Dev"
  prod:
    myVar: "My-3rdParty-Auth-Id-For-Prod"

The problem with the package command is now, that the variables are already substituted there and the CF template that is transferred to the deploy stage already contains the resolved variables. As a consequence it is not possible to deploy a built/packaged service to different stages, i.e. the stage used to build also has to be used to deploy.

A workaround is to build all stages/regions independently on the build server, create artifacts for each stage and deploy the corresponding artifact on the deployment server.

A proper solution would be that the build phase creates an artifact, that is independent from the environment that the artifact is deployed to.

Proposal

The build phase should keep the literal variable references in the artifact's CF template and the deploy phase should do the variable substitution depending on the stage/region selected at the time of deployment. This would mean, that only the variable subsitution functionality has to be moved. Of course only variables that are used/placed in the generated CF template should be deferred. So maybe the generateXXXTemplate() function that is run in the build phase would be the target for a viable implementation - then only the right variables would be affected. On the deploy phase the substitution has to be made as soon as the CF template from the artifact is loaded, to have the real values available for all plugins running in the deployment context.

Care has to be taken for plugins that might use the variable information during the build phase (imo that's wrong anyway).

[mike-suggitt]

FWIW I've come across this same issue for our builds. I've managed to create a workaround for the time being using a conditional package.artifact value.

As above, imagine a serverless.yml with

service: bob
custom:
  defaultPackage: ./.serverless
package:
  artifact: ${opt:artifact, self:custom.defaultPackage}/${self:service}.zip
provider:
    Stage: ${opt:env}

or similar.

I then run sls package --package bob --env dev Followed by

sls deploy --artifact bob --env dev
sls deploy --artifact bob --env qa
sls deploy --artifact bob --env live

The conditional use of package.artifact allows us to do an initial build without an artifact but then subsequently ignore cloudformation and recreate them at deploy time

[bishwash-devops]

This seems like a major issue, a package should be immutable and reusable. Where does this feature belong in the priority?

[bishwash-devops]

I went ahead with the suggested workaround, and it works fine for now. I have tried to document it here as well, https://medium.com/@bishwash.aryal/serverless-framework-build-immutable-package-for-ci-cd-pipeline-949b49f7df91

[alfaproject]

I'm having a similar issue but on removing a service instead because it references a CF variable of another service that might have been destroyed first.

To work-around that I created this patch and apply it with patch-package:

diff --git a/node_modules/serverless/lib/classes/Variables.js b/node_modules/serverless/lib/classes/Variables.js
index 10036b5..c3505a9 100644
--- a/node_modules/serverless/lib/classes/Variables.js
+++ b/node_modules/serverless/lib/classes/Variables.js
@@ -684,6 +684,9 @@ class Variables {
   }

   getValueFromCf(variableString) {
+    if (this.serverless.processedInput.options['ignore-cf-variables']) {
+      return BbPromise.resolve(undefined);
+    }
     const regionSuffix = variableString.split(':')[0].split('.')[1];
     const variableStringWithoutSource = variableString.split(':')[1].split('.');
     const stackName = variableStringWithoutSource[0];

That allows me to do serverless remove --ignore-cf-variables so that serverless doesn't try to get the CloudFormation variables from another service that might have been destroyed already.

Maybe we could introduce the concept of delayed variables, whereby they are only computed IIF they are needed.

[kabo]

I can't get the workaround to work :/

service:
  name: bob
custom:
  stage: "${opt:stage, self:provider.stage}"
package:
  individually: false
  artifact: ${opt:artifact, "./.serverless"}/${self:service.name}.zip
provider:
    stage: test

serverless package --package ./package -s test

I now see package/bob.zip. Great! I run

serverless deploy -s test --artifact ./package

But I still get

Serverless: Bundling with Webpack...

and a bunch of webpack output. It's still creating a new zip :(

Also, it's not clear to me how to get this to work with

package:
  individually: true

[medikoo]

This is a bit tricky with current design.

Problem is that variables by design can be used anywhere and can refer to anything, e.g. imagine following in service config:

package:
  artifact: ${self:provider.stage}.zip

or

package:
  include: ${self:provider.stage}/**

and I believe we can come up with trickier to handle cases.

This means that to create a build through which we can deploy any stage or region would require preserving whole service directory as is, and simply re-initializing a packaging logic as whole. That's obviously not what we're after.

To achieve the goal I think some internal design changes need to be made:

1. Clearly dinstinguish certain processing steps (handled at https://github.com/serverless/serverless/issues/8364)
2. Introduce another variables namespace (some special prefix), to be resolved prior deployment only. Through that namespace we could indicate variables that we do not which to resolve in packaging phase, but only in deployment phase.

[corydorning53]

I can't get the workaround to work :/

service:
  name: bob
custom:
  stage: "${opt:stage, self:provider.stage}"
package:
  individually: false
  artifact: ${opt:artifact, "./.serverless"}/${self:service.name}.zip
provider:
    stage: test

serverless package --package ./package -s test

I now see package/bob.zip. Great! I run

serverless deploy -s test --artifact ./package

But I still get

Serverless: Bundling with Webpack...

and a bunch of webpack output. It's still creating a new zip :(

Also, it's not clear to me how to get this to work with

package:
  individually: true

@kabo ever figure this out for individually packaged functions?

[bcraft]

I can't get the workaround to work :/

service:
  name: bob
custom:
  stage: "${opt:stage, self:provider.stage}"
package:
  individually: false
  artifact: ${opt:artifact, "./.serverless"}/${self:service.name}.zip
provider:
    stage: test

serverless package --package ./package -s test

I now see package/bob.zip. Great! I run

serverless deploy -s test --artifact ./package

But I still get

Serverless: Bundling with Webpack...

and a bunch of webpack output. It's still creating a new zip :( Also, it's not clear to me how to get this to work with

package:
  individually: true

@kabo ever figure this out for individually packaged functions?

I have been looking into this in some detail over the past couple of weeks and unfortunately whilst the serverless package event itself will acknowlege the artifact option by design the webpack plugin begins its own packaging process prior to that of serverless.

The solution I have is to remove the webpack plugin (and typescript if applicable) from the serverless.yml file which is included with the artifacts which are intended for reuse. I have been playing with a custom plugin which loads webpack/typescript dynamically (ommited from plugins in config), however its easier and more reliable to go with option 1.

[corydorning53]

@bcraft You can't have the artifact set to anything unless it exists. therefore you need to use logic that consumes the --artifact flag or sets it nothing...try this:

service:
  name: bob
custom:
  stage: ${opt:stage, self:provider.stage}
package:
  individually: false
  artifact: ${opt:artifact, ""}
provider:
    stage: test

then execute serverless package --package ./package -s test which will create your package in ./package and you can then run serverless deploy -s test --artifact ./package/bob.zip

[mike-suggitt]

This is a bit tricky with current design.

Problem is that variables by design can be used anywhere and can refer to anything, e.g. imagine following in service config:

package:
  artifact: ${self:provider.stage}.zip

or

package:
  include: ${self:provider.stage}/**

and I believe we can come up with trickier to handle cases.

This means that to create a build through which we can deploy any stage or region would require preserving whole service directory as is, and simply re-initializing a packaging logic as whole. That's obviously not what we're after.

To achieve the goal I think some internal design changes need to be made:

1. Clearly dinstinguish certain processing steps (handled at #8364)
2. Introduce another variables namespace (some special prefix), to be resolved prior deployment only. Through that namespace we could indicate variables that we do not which to resolve in packaging phase, but only in deployment phase.

It's maybe worth clarifying that I don't propose utilising this workaround as core behaviour of serverless package and deploy. My personal opinion is that serverless should ideally entirely separate out it's code packaging with it's deploy steps. I'm sure i'm not alone in advocating the advantages of "build once deploy everywhere". With the defafult behaviour of serverless, this isn't possible AFAIK

[bryantbiggs]

any updates on this - is it even on the roadmap @medikoo ?

3421

3696

4715

5150

6145

7085

tl;dr - I think all are looking for serverless to be able to support artifact promotion (build once, deploy many) across environments even when packaging individually

[medikoo]

@bryantbiggs We have that in mind, but it requires a significant change to Framework internals.

Currently compilation of end CF template and generation of packaging artifacts is mixed into same process flow, while to achieve this those have to be treated as totally separate.

We need to provide a mean to package artifacts, without a requirement for full configuration resolution and without attempting to generate a CF template.

Issue in which we want to tackle that: https://github.com/serverless/serverless/issues/8499 Still it wouldn't be delivered earlier than v3 of the Framework, and may require further major release (v4)

[erstaples]

Hi @medikoo - is the roadmap published anywhere? I'm curious to see what's prioritized ahead of this. I don't understand how a design flaw of this magnitude could go unaddressed for such a long time despite a lot of noise from the community to fix it (e.g. https://github.com/serverless/serverless/issues/4715).

[relaxdiego]

Hi. Is there any update to this issue? Thanks.

[medikoo]

Unfortunately at this point, there are no plans to work on that (we're a small team and currently our priorities are laid in other areas)