serverless [service]:package fails if another service refers as params

mokuzuu commented 2 years ago

Are you certain it's a bug?

[X] Yes, it looks like a bug

Are you using the latest version?

[X] Yes, I'm using the latest version

Is there an existing issue for this?

[X] I have searched existing issues, it hasn't been reported yet

Issue description

What I want to achieve

I want to package each service first to test before deployment. So I planned to package a service that outputs referred by another service first then a service depending on. However, the first packaging fails because another service depends on the output but I feel it shouldn't bother.

issue details

When service-b refers to the output of service-a as params in serverless-compose.yml, packaging service-a fails.

e.g. I have the following serverless-compose.yml

services:
  service-a:
    path: service-a 
  service-b:
    path: service-b
    params:
      value1: ${service-a.value1}

I ran the following command

npm run sls -- service-a:package

// package.json
"scripts": {
  "sls": "serverless"
}

I get following error

Error:
The variable "${service-a.value1}" cannot be resolved: the referenced output does not exist.

Service configuration (serverless-compose.yml) content

services:
  service-a:
    path: service-a
  service-b:
    path: service-b
    params:
      value1: ${service-a.value1}
      value2: ${service-a.value2}

Command name and used flags

npm run sls -- service-a:package --stage [stage-name] // package.json "scripts": { "sls": "serverless" }

Command output

> package-name@0.0.1 sls
> serverless "service-a:package" "--stage" "[stage-name]"

Environment: darwin, node 16.15.1, compose 1.2.4
Docs:        slss.io/docs-compose
Bugs:        github.com/serverless/compose/issues

Error:
The variable "${service-a.value1}" cannot be resolved: the referenced output does not exist.

Verbose logs are available in ".serverless/compose.log"

mokuzuu commented 2 years ago

Actually, deployment for service-a also fails but I feel it shouldn't...?

command npm run sls -- service-a:deploy

error log

> package-name@0.0.1 sls
> serverless "service-a:deploy"

Environment: darwin, node 16.15.1, compose 1.2.4
Docs:        slss.io/docs-compose
Bugs:        github.com/serverless/compose/issues

Error:
The variable "${service-a.value1}" cannot be resolved: the referenced output does not exist.

mnapoli commented 2 years ago

Could it be that the service has been deployed from another machine? (https://www.serverless.com/framework/docs/guides/compose#refreshing-outputs)

You might want to refresh the outputs stored locally with:

serverless refresh-outputs

mokuzuu commented 2 years ago

@mnapoli Sorry for the late reply. I ran the command below but it didn't work...

npm run sls:refresh -- --stage [stage-name] // sls:refresh: "serverless refresh-outputs"

> project@0.0.1 sls:refresh
> serverless refresh-outputs "--stage" "[stage-name]"

Refreshing outputs

    ✔  service-a › outputs refreshed › 3s
    ✔  service-b › outputs refreshed › 4s

npm run sls:deploy -- service-a:deploy -- --stage [stage-name] // sls:deploy: "serverless deploy"

> project@0.0.1 sls:deploy
> serverless deploy "service-a:deploy"

Environment: darwin, node 16.15.1, compose 1.2.4
Docs:        slss.io/docs-compose
Bugs:        github.com/serverless/compose/issues

Error:
The variable "${service-a.value1}" cannot be resolved: the referenced output does not exist.

Verbose logs are available in ".serverless/compose.log"

mnapoli commented 2 years ago

Thanks. Could it be that the output simply doesn't exist? Could you run the command sls outputs?

semiautomatixfu commented 2 years ago

This is a bit of a chicken-and-egg situation for package where a service has params.

$ ./node_modules/serverless/bin/serverless.js package --service service-b --stage dev
Environment: linux, node 14.19.2, compose 1.2.4
Docs:        slss.io/docs-compose
Bugs:        github.com/serverless/compose/issues

Error:
The variable "${shared.dataTableName}" cannot be resolved: the referenced output does not exist.

Verbose logs are available in ".serverless/compose.log"

$ ./node_modules/serverless/bin/serverless.js package --service service-b --stage dev --param dataTableName=""
(node:87066) UnhandledPromiseRejectionWarning: ServerlessError: The "--param" option is not supported (yet) in Serverless Framework Compose
You can search and/or open feature requests here: https://github.com/serverless/compose'
    at /node_modules/@serverless/compose/src/validate-options.js:12:13
    at Array.forEach (<anonymous>)
    at validateCliOptions (/node_modules/@serverless/compose/src/validate-options.js:10:31)
    at runComponents (/node_modules/@serverless/compose/src/index.js:125:3)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:87066) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:87066) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

tl;dr if we define params in the serverless.yml file, then it is neither possible to use package with or without specifying the params.

services:
  service-a:
    path: service-a
  service-b:
    path: service-b
    params:
      value1: ${service-a.value1}
      value2: ${service-a.value2}

pgrzesik commented 2 years ago

Hey @semiautomatixfu, the --param flag is not supported on Compose level, but the locally resolved params are still going to be passed. Could you run sls refresh-outputs first so the outputs of service-a are actually populated locally?

semiautomatixfu commented 2 years ago

@pgrzesik Additionally we're running dynamodb-local and offline to add to the issues. I'm unsure how packaging service-a seperately would require a reference to it's own output :shrug:

$ ./node_modules/serverless/bin/serverless.js package --service service-a --stage ${USER} --verbose
Environment: linux, node 14.19.2, compose 1.2.4
Docs:        slss.io/docs-compose
Bugs:        github.com/serverless/compose/issues

Error:
The variable "${service-a.value1}" cannot be resolved: the referenced output does not exist.
ServerlessError: The variable "${service-a.value1}" cannot be resolved: the referenced output does not exist.
    at Object.<anonymous> (/node_modules/@serverless/compose/src/ComponentsService.js:45:15)
    at walker (/node_modules/traverse/index.js:190:22)
    at /node_modules/traverse/index.js:208:29
    at Array.forEach (<anonymous>)
    at forEach (/node_modules/traverse/index.js:298:31)
    at walker (/node_modules/traverse/index.js:203:13)
    at /node_modules/traverse/index.js:208:29
    at Array.forEach (<anonymous>)
    at forEach (/node_modules/traverse/index.js:298:31)
    at walker (/node_modules/traverse/index.js:203:13)

Verbose logs are available in ".serverless/compose.log"
$ ./node_modules/serverless/bin/serverless.js  refresh-outputs "--stage" "dev"

Refreshing outputs

    ✖  shared › error › 3s

Error:
Stack with id service-a-dev does not exist

Thanks!

pgrzesik commented 2 years ago

That's very surprising - do you have a chance to provide a small reproducible example that I could run on my side to evaluate these issues? I couldn't reproduce it with my test services

tibbe commented 1 year ago

One thing I noticed is that, for me, sls package only works if I provide --stage.

The behavior seems stateful as well. sls package --stage=foo works if I previously deployed foo. refresh-outputs doesn't work and fails with the same

The variable "${backend.emailBucketName}" cannot be resolved: the referenced output does not exist.

error.

tibbe commented 1 year ago

Some more testing suggests that sls package/sls refresh-outputs only works with a previously successfully deployed stage.

darylteo commented 1 year ago

For those looking for a workaround on this (similar to my suggested proposal here https://github.com/serverless/compose/issues/149) , I ended up creating 2 serverless-compose.{stage}.yml, and then manually copying them to serverless-compose.yml in as package.json scripts.

serverless / compose