[Bug] Documentation issues for CI/CD pipeline

tdmalone commented 2 years ago

Describe the bug The documentation for the CI/CD pipeline appears to be incorrect:

pnpx sls deploy is referenced a few times - this actually runs the 'structured liste' package (https://www.npmjs.com/package/sls) and thus doesn't deploy anything; it should probably be pnpx serverless deploy instead?
using pnpx serverless deploy instead then comes across an error accessing the @amzn/base-serverless-settings-helper package, which appears to not exist (either at https://www.npmjs.com/package/@amzn/base-serverless-settings-helper or https://www.npmjs.com/package/@amzn-oss/base-serverless-settings-helper, or anywhere else on npm that I can find)

To Reproduce Steps to reproduce the behavior:

Follow the instructions in https://github.com/awslabs/service-workbench-on-aws/blob/mainline/docs/docs/best_practices/cicd.md#deploying-the-cicd-pipeline

Expected behavior The instructions to work.

Actual behavior

 ~/service-workbench-on-aws/main/cicd/cicd-source $ pnpx sls deploy --stage dev       
.../Library/pnpm/store/v3/tmp/dlx-75388  |   +1 +
Packages are hard linked from the content-addressable store to the virtual store.
  Content-addressable store is at: /Users/tim/Library/pnpm/store/v3
  Virtual store is at:             ../../../../../../../../../Library/pnpm/store/v3/tmp/dlx-75388/node_modules/.pnpm
.../Library/pnpm/store/v3/tmp/dlx-75388  | Progress: resolved 1, reused 1, downloaded 0, added 1, done
./
 ├─config/
 │ ├─infra(+)/
 │ └─settings(+)/
 ├─README.md
 ├─package.json
 ├─serverless.yml
 └─yarn-error.log

 ~/service-workbench-on-aws/main/cicd/cicd-source $ pnpx serverless deploy --stage dev
.../Library/pnpm/store/v3/tmp/dlx-75420  |  WARN  deprecated querystring@0.2.1
.../Library/pnpm/store/v3/tmp/dlx-75420  |  WARN  deprecated querystring@0.2.0
.../Library/pnpm/store/v3/tmp/dlx-75420  |  WARN  deprecated superagent@7.1.6
.../Library/pnpm/store/v3/tmp/dlx-75420  | +414 +++++++++++++++++++++++++++++++++++++++++
Packages are hard linked from the content-addressable store to the virtual store.
  Content-addressable store is at: /Users/tim/Library/pnpm/store/v3
  Virtual store is at:             ../../../../../../../../../Library/pnpm/store/v3/tmp/dlx-75420/node_modules/.pnpm
.../Library/pnpm/store/v3/tmp/dlx-75420  | Progress: resolved 457, reused 414, downloaded 0, added 414, done
Environment: darwin, node 18.5.0, framework 3.21.0, plugin 6.2.2, SDK 4.3.2
Docs:        docs.serverless.com
Support:     forum.serverless.com
Bugs:        github.com/serverless/serverless/issues

Error:
Cannot resolve serverless.yml: Variables resolution errored with:
  - Cannot resolve variable at "custom.settings": Cannot load "config/settings/.settings.js": Initialization error: Error: Cannot find module '@amzn/base-serverless-settings-helper'
Require stack:
- /Users/tim/service-workbench-on-aws/main/cicd/cicd-source/config/settings/.settings.js
- /Users/tim/Library/pnpm/store/v3/tmp/dlx-75420/node_modules/.pnpm/serverless@3.21.0/node_modules/serverless/lib/configuration/variables/sources/file.js
- /Users/tim/Library/pnpm/store/v3/tmp/dlx-75420/node_modules/.pnpm/serverless@3.21.0/node_modules/serverless/scripts/serverless.js
- /Users/tim/Library/pnpm/store/v3/tmp/dlx-75420/node_modules/.pnpm/serverless@3.21.0/node_modules/serverless/bin/serverless.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:946:15)
    at Module._load (node:internal/modules/cjs/loader:787:27)
    at Module.require (node:internal/modules/cjs/loader:1012:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/Users/tim/service-workbench-on-aws/main/cicd/cicd-source/config/settings/.settings.js:16:25)
    at Module._compile (node:internal/modules/cjs/loader:1112:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1166:10)
    at Module.load (node:internal/modules/cjs/loader:988:32)
    at Module._load (node:internal/modules/cjs/loader:834:12)
    at Module.require (node:internal/modules/cjs/loader:1012:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at /Users/tim/Library/pnpm/store/v3/tmp/dlx-75420/node_modules/.pnpm/serverless@3.21.0/node_modules/serverless/lib/configuration/variables/sources/file.js:100:22
    at Object.resolve (/Users/tim/Library/pnpm/store/v3/tmp/dlx-75420/node_modules/.pnpm/serverless@3.21.0/node_modules/serverless/lib/configuration/variables/sources/file.js:144:7)
    at Object.defineProperties.memoizeMethods.resolveSource.d.normalizer.type.type (/Users/tim/Library/pnpm/store/v3/tmp/dlx-75420/node_modules/.pnpm/serverless@3.21.0/node_modules/serverless/lib/configuration/variables/resolve.js:561:60)
    at VariablesResolver.<anonymous> (/Users/tim/Library/pnpm/store/v3/tmp/dlx-75420/node_modules/.pnpm/memoizee@0.4.15/node_modules/memoizee/lib/configure-map.js:61:24)
    at /Users/tim/Library/pnpm/store/v3/tmp/dlx-75420/node_modules/.pnpm/serverless@3.21.0/node_modules/serverless/lib/configuration/variables/resolve.js:275:31
    at VariablesResolver.resolveVariableSource (/Users/tim/Library/pnpm/store/v3/tmp/dlx-75420/node_modules/.pnpm/serverless@3.21.0/node_modules/serverless/lib/configuration/variables/resolve.js:305:11)
    at async VariablesResolver.resolveVariable (/Users/tim/Library/pnpm/store/v3/tmp/dlx-75420/node_modules/.pnpm/serverless@3.21.0/node_modules/serverless/lib/configuration/variables/resolve.js:192:24)
    at async /Users/tim/Library/pnpm/store/v3/tmp/dlx-75420/node_modules/.pnpm/serverless@3.21.0/node_modules/serverless/lib/configuration/variables/resolve.js:118:9
    at async Promise.all (index 0)
    at async VariablesResolver.resolveVariables (/Users/tim/Library/pnpm/store/v3/tmp/dlx-75420/node_modules/.pnpm/serverless@3.21.0/node_modules/serverless/lib/configuration/variables/resolve.js:112:5)
    at async VariablesResolver.self (/Users/tim/Library/pnpm/store/v3/tmp/dlx-75420/node_modules/.pnpm/serverless@3.21.0/node_modules/serverless/lib/configuration/variables/resolve.js:457:9)
 ERROR  Command failed with exit code 1: /Users/tim/Library/pnpm/store/v3/tmp/dlx-75420/node_modules/.bin/serverless deploy --stage dev

pnpm: Command failed with exit code 1: /Users/tim/Library/pnpm/store/v3/tmp/dlx-75420/node_modules/.bin/serverless deploy --stage dev
    at makeError (/Users/tim/.config/yarn/global/node_modules/pnpm/dist/pnpx.cjs:16516:17)
    at handlePromise (/Users/tim/.config/yarn/global/node_modules/pnpm/dist/pnpx.cjs:17087:33)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.handler [as dlx] (/Users/tim/.config/yarn/global/node_modules/pnpm/dist/pnpx.cjs:183479:7)
    at async /Users/tim/.config/yarn/global/node_modules/pnpm/dist/pnpx.cjs:192493:21
    at async run (/Users/tim/.config/yarn/global/node_modules/pnpm/dist/pnpx.cjs:192467:34)
    at async /Users/tim/.config/yarn/global/node_modules/pnpm/dist/pnpx.cjs:192539:5

tdmalone commented 2 years ago

EDIT: This comment is now out-of-date - see the next comment.

The second error appears to be fixable with:

cd ../../../addons/addon-base/packages/serverless-settings-helper
yarn install
yarn link
cd ../../../../main/cicd/cicd-pipeline
yarn link @amzn/base-serverless-settings-helper
pnpx serverless deploy --stage dev

This then results in:

Error:
Cannot resolve serverless.yml: Variables resolution errored with:
  - Cannot resolve variable at "custom.settings": Cannot resolve "merged" out of ".settings.js": Received rejection: TypeError: Cannot read properties of undefined (reading 'options')
    at /Users/tim/service-workbench-on-aws/addons/addon-base/packages/serverless-settings-helper/lib/index.js:137:40

which can be resolved by changing line 137 in the above file:

-    const stage = serverless.variables.options.s || serverless.variables.options.stage || undefined;
+    const stage = serverless.options.s || serverless.options.stage || undefined;

This then results in:

Error:
Cannot resolve serverless.yml: Variables resolution errored with:
  - Cannot resolve variable at "custom.settings": Cannot resolve "merged" out of ".settings.js": Received rejection: TypeError: Cannot read properties of undefined (reading 'parse')
    at /Users/tim/service-workbench-on-aws/addons/addon-base/packages/serverless-settings-helper/lib/index.js:83:40

tdmalone commented 2 years ago

Ok - eventually got there with a suggestion from a colleague - just run pnpm i in the root folder, and this appears to fill in dependencies for all other packages as well.

I presume that may have also resolved most (all?) of the above errors.

So I guess the bug is, at least, that the step to run pnpm i in the root folder is missing from the docs :)

tdmalone commented 2 years ago

Another step missing - config files are gitignored, so they never get pushed to the codecommit repo. You need to comment out the relevant lines in the root .gitignore.

tdmalone commented 2 years ago

Serverless also appears to not be able to find credentials from the ECS task metadata endpoint in CodeBuild:

 CredentialsError: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1

Prepending the following to the pre_build stage in the buildspec seems to work around this:

      - CREDS="$(curl "http://169.254.170.2${AWS_CONTAINER_CREDENTIALS_RELATIVE_URI}")"
      - aws configure set aws_access_key_id "$(echo "${CREDS}" | jq -r .AccessKeyId)"
      - aws configure set aws_secret_access_key "$(echo "${CREDS}" | jq -r .SecretAccessKey)"
      - aws configure set aws_session_token "$(echo "${CREDS}" | jq -r .Token)"

maghirardelli commented 2 years ago

Hey @tdmalone I am not sure if you got this working or not. I was just able to successfully deploy the CI/CD pipeline using these instructions: https://github.com/awslabs/service-workbench-on-aws/tree/mainline/main/cicd. I believe the file you were linking is for our old documentation set when we used docusaurus for delivery.

I believe many of the errors you ran into were because you were using Serverless Framework V3 (Environment: darwin, node 18.5.0, framework 3.21.0, plugin 6.2.2, SDK 4.3.2) while before SWB v5.2.3, we only supported using Serverless Framework v1. Upgrading to SWB v5.2.3 might fix lots of the errors you mentioned here. Would you be able to try and let me know?

I am going to add a backlog item to review the documentation on our side, but any information you can give us will help us in this process.

Thanks for bringing this all to our attention, Marianna

tdmalone commented 2 years ago

Hi @maghirardelli,

Thanks for your reply. We did eventually get it working after making lots of tweaks and fixes, some of which are outlined in this issue.

I believe the file you were linking is for our old documentation set when we used docusaurus for delivery.

Can I suggest that if documentation is no longer correct, it is removed from the repository.

I believe many of the errors you ran into were because you were using Serverless Framework V3...

If I missed this in the docs, apologies, however if it wasn't in the docs, obviously I would suggest that the version constraint should be added. I would presume it's quite natural to install the latest available version unless explicitly advised not to.

awslabs / service-workbench-on-aws

[Bug] Documentation issues for CI/CD pipeline #1016