Serverless variables aren't being resolved anymore

[JaroVDH]

Describe the bug Since version 1.2 serverless variables (eg. ${self:stage.provider}) aren't being resolved to their values anymore.

To Reproduce Environment: Bitbucket Pipeline CI with Docker image node:14.15.0

Versions:

"serverless": "^2.38.0",
"serverless-scriptable-plugin": "^1.2.1",

Relevant serverless.yml sections:

provider:
  name: aws
  runtime: nodejs14.x
  stage: ${opt:stage, 'nonprod'}

plugins:
  - serverless-scriptable-plugin

custom:
  scriptHooks:
    after:package:initialize: STAGE=${self:provider.stage} npm run build:views && STAGE=${self:provider.stage} npm run build:editor

Running a serverless package results in:

Running command: STAGE=${self:provider.stage} npm run build:views && STAGE=${self:provider.stage} npm run build:editor
/bin/sh: 1: Bad substitution

Expected behavior It used to properly parse them. Pipeline run from ~24 hours ago:

Running command: STAGE=nonprod npm run build:views && STAGE=nonprod npm run build:editor

Additional context It also seems to be happening when running locally on my macOS Catalina.

[weixu365]

I can't reproduce it either under macOS Catalina or docker, can you reproduce the issue by creating a new repo and show outputs like the following?

Here's the console output of running inside docker on my local:

$ docker run -it -v `pwd`:/work -w /work node:14.15.0 bash
root@23ee18ed0412:/work# ls
node_modules  package-lock.json  package.json  serverless.yml

root@23ee18ed0412:/work# cat serverless.yml 
service: plugintest

provider:
  name: aws
  runtime: nodejs12.x
  region: ap-southeast-2
  stage: ${opt:stage, 'nonprod23'}

plugins:
  - serverless-scriptable-plugin

custom:
  scriptHooks:
    after:package:initialize: STAGE=${self:provider.stage} npm run build:views && STAGE=${self:provider.stage} npm run build:editor

functions:
  test:
    handler: handler.handle

root@23ee18ed0412:/work# cat package.json 
{
  "name": "plugin-test",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "serverless": "^2.38.0",
    "serverless-scriptable-plugin": "^1.2.1"
  }
}

root@23ee18ed0412:/work# npm i
npm WARN plugin-test@1.0.0 No description
npm WARN plugin-test@1.0.0 No repository field.

audited 644 packages in 8.615s

35 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

root@23ee18ed0412:/work# npx serverless package
Running command: STAGE=nonprod23 npm run build:views && STAGE=nonprod23 npm run build:editor
npm ERR! missing script: build:views

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2021-04-29T11_33_32_812Z-debug.log

[JaroVDH]

I'll try and set up a reproducible minimal setup, but it might take a while to find time, sorry about that. 🙏

I've locked it @ v1.1.0 for us and that fixed it for now.

[RLRabinowitz]

This regression is happening to me as well, and prevents us from being able to upgrade from v1.1.0 to v1.2.1

[weixu365]

Hi @RLRabinowitz and @JaroVDH , is that possible for you to create a minimum reproducible project like what I did above? And paste the content of two files: serverless.yml, package.json.

I can't fix it if I don't know how to reproduce the issue.

[weixu365]

I still could not reproduce it by using nodejs 14, serverless: ^2.39.1.

Could you please do the following changes and paste the output? and a screenshot of the related settings in the config file

1. Change serverless to: ^2.39.1

2. Change config file

   service: plugintest
   unresolvedVariablesNotificationMode: error
   variablesResolutionMode: 20210326

And change the variable in the hook script to something not exists, then it would show the following message on my machine:

  Cannot resolve serverless.yml: Variables resolution errored with:
    - Cannot resolve variable at "custom.scriptHooks.after:package:initialize": Value not found at "self" source

[JaroVDH]

Sorry for the delay, I managed to get some time for this. It seems like it might not be directly related to serverless-scriptable-plugin configuration.

I've managed to boil it down to this:

It seems that if you have a variable with an empty default anywhere in your serverless.yml, it'll stop working. Removing that empty default value (eg. replacing it with ${self:provider.stage, 'anything'} or ${self:provider.stage}) makes it work again.

[weixu365]

Glad that you figured out this hard-to-find root cause 😄 , and updated it here.

In the latest version of serverless, if you add variablesResolutionMode: 20210326 like https://github.com/weixu365/serverless-scriptable-plugin/issues/81#issuecomment-831228818, it will print an error and stop which would save you a lot of time, and this would be the default behavior of version 3.x.

Example of the error output is:

 Serverless Error ----------------------------------------

   Cannot resolve serverless.yml: Variables resolution errored with:
    - Variable syntax error at "provider.environment.test": Invalid variable source at index 22 in "${self:provider.stage,}"

[weixu365]

Hi @RLRabinowitz, Could you please check if the error you got caused by the same reason? if yes, I would close this ticket, thanks.

[RLRabinowitz]

I'll try that out now

[RLRabinowitz]

Adding variablesResolutionMode: 20210326 doesn't solve this for me. Also, in my case those are not variables with empty default values, those values actually exist.

I'm getting the following when running my command:

Running command: REGION=${self:provider.region} STAGE=${self:custom.stage} yarn docker:build

[weixu365]

@RLRabinowitz In that case, I need reproducible steps to understand this issue, what you can do is find the root cause by removing all unrelated settings and make sure the issue still exists then post the config files here

https://github.com/weixu365/serverless-scriptable-plugin/issues/81#issuecomment-831145431

[github-actions[bot]]

This issue has now been marked as stale because there hasn't been any activity on this issue recently. It will be closed if no further activity occurs. Thank you for your contributions.

[bitsofinfo]

experiencing same problem

[weixu365]

@bitsofinfo , Could you please provide reproducible steps for me to understand your issue, what you can do is find the root cause by removing all unrelated settings and make sure the issue still exists then post the config files here

You can reference what I tried to reproduce the issue at here: https://github.com/weixu365/serverless-scriptable-plugin/issues/81#issuecomment-831145431

[bitsofinfo]

i found the issue and its unrelated

[RLRabinowitz]

I wan't able to reproduce this in a smaller repo, but I've also experienced ${aws:region} variable being resolved to an empty string

I have found a fix though, changing the hook definition to be an array solved the variable resolution for me.

So instead of:

custom:
  scriptHooks:
    after:package:initialize: STAGE=${self:provider.stage} npm run build:views && STAGE=${self:provider.stage} npm run build:editor

I write:

custom:
  scriptHooks:
    after:package:initialize: 
      - STAGE=${self:provider.stage} npm run build:views && STAGE=${self:provider.stage} npm run build:editor

That solved it for me, not sure why though