Closed chrismilleruk closed 2 years ago
kaustavghosh06
commented
2 years ago @chrismilleruk Could you please use npm install -g @aws-amplify/cli instead of using yarn and if you're still experiencing this issue? We've seen issues with yarn installations in the past like this one - https://github.com/aws-amplify/amplify-cli/issues/8942
chrismilleruk
commented
2 years ago @kaustavghosh06 yes, I switched to yarn after npm i -g @aws-amplify/cli failed, here is the buildspec and logs from that run:
kaustavghosh06
commented
2 years ago @chrismilleruk Does it work with the default Node version which Amplify build provides and NPM CLI installation?
chrismilleruk
commented
2 years ago It's a reasonable question but it might be trying to create a workaround / resolve a different issue. Specifically (& rather sarcastically), "...upgrade Node AND Amplify prior to amplifyPush in build" definitely does not work if we use the "default Node version which Amplify build provides and NPM CLI installation." 😁
... however, for your info, my specific use case & background:
In our case, the correct & specific version of tsc is required for this build step:
The specific tsc version we need is stored in the package.json file so a simple npm install -g typescript against node 12 doesn't really work for me as it will introduce inconsistencies.
npm install with the default 'NPM CLI installation' produces errors due to a package-lock.json version conflict.
I'm not too keen to look at any temporary workarounds for this issue since a recent opaque change to _LIVE_UPDATES caused a build pipeline to fail spectacularly, in prod, 24h before a conference.
I've disabled lambda pre-build for now and I'm looking for a robust way to upgrade the build container to the correct versions prior to running the build. Preferably without installing the same thing multiple times against different versions.
I hope this helps & provides the background context you need to understand why I'm attempting to upgrade node & reinstall amplify at this point in the build.
chrismilleruk
commented
2 years ago @kaustavghosh06 thanks in advance for your help and let me know if I can add any more info for you.
lazpavel
commented
2 years ago Hi @chrismilleruk, is it possible for you to use node 14? Please let us know if you're still having issues after that.
josefaidt
commented
2 years ago Hey @chrismilleruk :wave: as a workaround can you try adding the string "packaged-node-modules" to the pluginDirectories section of the ~/.amplify/plugins.json file and try to redeploy?
{
"pluginDirectories": [
"cli-local-node-modules",
"cli-parent-directory",
"global-node-modules",
+ "packaged-node-modules"
],
// ...
}For reference: https://github.com/josefaidt/9209/blob/main/scripts/patch.js
chrismilleruk
commented
2 years ago @lazpavel, @josefaidt 👋 Apologies for the slow response. I've created a ticket to try these approaches and will let you know how it goes.
josefaidt
commented
2 years ago Hey @chrismilleruk :wave: just wanted to follow-up on this one and see if you'd had a chance to try out the workaround posted above?
chrismilleruk
commented
2 years ago Hi @josefaidt, @lazpavel, I've had a chance to revisit this and test some more this evening, with mixed results. There seem to be a few things overlapping here so I did some brute force testing to try and isolate the key factors.
A quick reminder that the goal (for me at least) is to upgrade node (Active LTS: v16, Current: v17) with a working amplify CLI at preBuild. The two workarounds mentioned are "use node 14" and "add the string "packaged-node-modules" to the pluginDirectories section of the ~/.amplify/plugins.json".
The patch seems to be non-impacting from what I can tell.
The patch works for nvm install v14 and nvm install v16 when using yarn
The target version of Node (and NPM) coupled with the amplify-cli install method seems to have the greatest impact & correlation.
I see also that the main build provision script has been updated and the default version of node is now v14.18.1 instead of v12.21.0. A great improvement 👏. We need to nvm install something in order to cause the issue so I have just performed a nvm install 14.18.3.
NO PATCH OR node install-9209-patch.js
nvm install 16 OR nvm install 14.18.3 OR nvm use 16
NO AMPLIFY INSTALL OR npm install -g @aws-amplify/cli@latest OR yarn global add @aws-amplify/cli OR npm config set user 0 && npm config set unsafe-perm true && npm install -g @aws-amplify/cli@latest
amplify --version
If you want a minor or patch update to v14, use npm install -g.
node install-9209-patch.js + nvm install 14.18.3 + npm install -g @aws-amplify/cli@latest
NO PATCH + nvm install 14.18.3 + npm install -g @aws-amplify/cli@latest
Success 🆗 node v14.18.3 (npm v6.14.15)
version: 1
backend:
phases:
preBuild:
commands:
- "# Check Versions"
- node --version && npm --version && yarn --version && amplify --version
- "# Upgrade from 14.18.1 to 14.x"
- nvm install 14
- npm install -g @aws-amplify/cli@latest
- amplify --versionIf you want to upgrade to v14 or v16 with yarn global add, use the patch from @josefaidt above (https://github.com/josefaidt/9209/blob/main/scripts/patch.js).
node install-9209-patch.js + nvm install 14 + yarn global add @aws-amplify/cli
node install-9209-patch.js + nvm install 16 + yarn global add @aws-amplify/cli
Success 🆗 node v16.13.2 (npm v8.1.2) yarn@1.22.0
version: 1
backend:
phases:
preBuild:
commands:
- "# Check Versions"
- node --version && npm --version && yarn --version && amplify --version
- "# Upgrade from 14.18.1 to 16.x"
- node install-9209-patch.js
- nvm install 16
- yarn global add @aws-amplify/cli
- amplify --versionIt is also possible to nvm _use_ 16 to access v16.13.0 since it is preinstalled but it is not recommended.
Using npm config set user 0 && npm config set unsafe-perm true && npm install -g @aws-amplify/cli@latest to update amplify
josefaidt
commented
2 years ago Hey @chrismilleruk thank you for taking the time to include those details! The "packaged-node-modules" line added to the plugins JSON file should be all we need to resolve the "missing plugins..." error regardless of Node version, though I'm unsure what issues you may have encountered with the preinstalled Node version 16. In any event I'm glad to hear that you're unblocked 🙂
chrismilleruk
commented
2 years ago Hey @josefaidt, No problem, sorry for the big blogpost..
Just checking again with a clear head and I can see I was wrong (copy paste error).
The patch in fact DID work for both v16 and v14 when using Yarn to upgrade amplify.
Thanks again Chris
chrismilleruk
commented
2 years ago I've updated the summary comment above for future readers.
l20DfX35JnKBfRn
commented
8 months ago Future readers:
I've managed to get next 14 working on amplify with cloudformation without needing to create custom image hosted on ECR...
Just use the public.ecr.aws/sam/build-nodejs[version].x:latest images.
E.g., 18.x latest
EnvironmentVariables:
- Name: _CUSTOM_IMAGE
Value: "public.ecr.aws/sam/build-nodejs18.x:latest"*Thanks to https://www.cloudypandas.ch/posts/for-when-you-need-a-custom-amplify-build-image/
I hope this finds you well :)
Before opening, please confirm:
How did you install the Amplify CLI?
npm & yarn
If applicable, what version of Node.js are you using?
node 16
Amplify CLI Version
7.3.3 - 7.3.5
What operating system are you using?
Amazon Linux
Amplify Categories
Not applicable
Amplify Commands
push, Not applicable
Describe the bug
In order to support typescript compilation in lambda functions (https://docs.amplify.aws/cli/function/build-options/) I need to upgrade
node&npmprior toamplifyPush --simple(which has a dependency on@aws-amplify/cli).Most folks seem to do
nodeupgrades in thefrontEnd-preBuildvianvmand while this destroysamplifycli, this goes unnoticed since thebackEnd-buildhas already taken place.I tried a few different methods to get this working (below) but the simplest repro steps below (
nvm install 16,yarn global add @aws-amplify/clicreate the following error:Other techniques attempted
nvm install 16nvm install 16 --reinstall-packages-from=nodenpm install -g @aws-amplify/cli@latestyarn global add @aws-amplify/cli@aws-amplify/cli@latestin_LIVE_UPDATESnvm install 16via_LIVE_UPDATESSide note. I had managed to workaround this issue previously by upgrading both
npmand@aws-amplify/cliin_LIVE_UPDATES, which was just enough at the time, but that stopped working last week. I suspect the root cause of failure was a change (Tue 16th between 12:00 and 18:00 GMT) to run the npm live updates in parallel instead of in the supplied order. it's obviously not possible to upgrade npm while npm is installing something else so my hack failed. Regardless, the Cloning Repository step was marked as successful even though it failed with errors.Expected behavior
It should be possible to upgrade
nodeusingnvmand install@aws-amplify/cli@latestprior toamplifyPushReproduction steps
GraphQL schema(s)
Log output
Additional information
Above is extracted from this build: eu-west-1#/d16xma096tmdk6/feature-sc-1422/21 The previous 20 builds may also be relevant as they were other attempts to solve the same issue.
Side Issue: Cloning Repository Step marked as successful
Logs from: eu-west-1#/d16xma096tmdk6/prod/15
``` # Put your logs below this line 2021-11-16T19:34:40.245Z [INFO]: Git SSH Key acquired 2021-11-16T19:34:40.321Z [INFO]: # Cloning repository: git@github.com:Akeero/akeero-admin.git 2021-11-16T19:34:40.374Z [INFO]: Agent pid 136 2021-11-16T19:34:40.396Z [INFO]: Identity added: /root/.ssh/git_rsa (/root/.ssh/git_rsa) 2021-11-16T19:34:40.583Z [INFO]: Cloning into 'akeero-admin'... 2021-11-16T19:34:41.097Z [INFO]: Warning: Permanently added the ECDSA host key for IP address '140.82.121.4' to the list of known hosts. 2021-11-16T19:34:43.412Z [INFO]: # Switching to commit: a723c66142bb1f4aca3306c0f418274016c3ca3f 2021-11-16T19:34:43.442Z [INFO]: Agent pid 149 2021-11-16T19:34:43.516Z [INFO]: Identity added: /root/.ssh/git_rsa (/root/.ssh/git_rsa) Note: switching to 'a723c66142bb1f4aca3306c0f418274016c3ca3f'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example: git switch -c