Deploy fails with "[ERROR]: Failed to deploy"

[desertmonkey]

Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

1. Start a build on our master branch
2. See error during the Deploy step.

Expected behavior The build should succeed, as it has every other time before today.

Additional context We have a stage branch and a master branch in Amplify. We merge GitHub release branches into the stage branch first to verify it all looks good. The builds on stage work flawlessly. Then we merge that same release branch into the master branch and the build fails to deploy.

I thought this could be related to this issue but I reduced the size of our build by a couple hundred MBs, and it still fails to deploy.

[rickiesmooth]

we have the same problem currently.. our app id is d3rfa777e3c1ou in ap-southeast-2.

don't know if this is relevant,

but we noticed that the error comes when we changed the cypress version in yarn lock. If we use 3.4.1, the tests are running. (sometimes it freezes but it's running). When we upgraded our cypress version to 3.6.1 in package.json, it fixes the freeze.

But when we changed our yarn.lock to include that 3.6.1, it doesn't even run the test. These are the last log from test tab

cypress run --reporter mochawesome --reporter-options "reportDir=cypress/report/mochawesome-report,overwrite=false,html=false,json=true,timestamp=mmddyyyy_HHMMss"

and then it goes straight to deployment

2019-11-21T04:40:01 [INFO]: Starting Deployment 2019-11-21T04:40:01 [ERROR]: Failed to deploy 2019-11-21T04:40:01 [INFO]: Deployment finished.

Update

I can confirm that the cypress version in yarn.lock is causing it.

Because when I rollback to:

cypress@3.4.1:
  version "3.4.1"
  resolved "https://registry.yarnpkg.com/cypress/-/cypress-3.4.1.tgz#ca2e4e9864679da686c6a6189603efd409664c30"
  integrity sha512-1HBS7t9XXzkt6QHbwfirWYty8vzxNMawGj1yI+Fu6C3/VZJ8UtUngMW6layqwYZzLTZV8tiDpdCNBypn78V4Dg==
  dependencies:
    "@cypress/listr-verbose-renderer" "0.4.1"
    "@cypress/xvfb" "1.2.4"
    arch "2.1.1"
    bluebird "3.5.0"
    cachedir "1.3.0"
    chalk "2.4.2"
    check-more-types "2.24.0"
    commander "2.15.1"
    common-tags "1.8.0"
    debug "3.2.6"
    execa "0.10.0"
    executable "4.1.1"
    extract-zip "1.6.7"
    fs-extra "5.0.0"
    getos "3.1.1"
    is-ci "1.2.1"
    is-installed-globally "0.1.0"
    lazy-ass "1.6.0"
    listr "0.12.0"
    lodash "4.17.15"
    log-symbols "2.2.0"
    minimist "1.2.0"
    moment "2.24.0"
    ramda "0.24.1"
    request "2.88.0"
    request-progress "3.0.0"
    supports-color "5.5.0"
    tmp "0.1.0"
    url "0.11.0"
    yauzl "2.10.0"

it works.

Previously, we used:

cypress@3.6.1:
  version "3.6.1"
  resolved "https://registry.yarnpkg.com/cypress/-/cypress-3.6.1.tgz#4420957923879f60b7a5146ccbf81841a149b653"
  integrity sha512-6n0oqENdz/oQ7EJ6IgESNb2M7Bo/70qX9jSJsAziJTC3kICfEMmJUlrAnP9bn+ut24MlXQST5nRXhUP5nRIx6A==
  dependencies:
    "@cypress/listr-verbose-renderer" "0.4.1"
    "@cypress/xvfb" "1.2.4"
    "@types/sizzle" "2.3.2"
    arch "2.1.1"
    bluebird "3.5.0"
    cachedir "1.3.0"
    chalk "2.4.2"
    check-more-types "2.24.0"
    commander "2.15.1"
    common-tags "1.8.0"
    debug "3.2.6"
    execa "0.10.0"
    executable "4.1.1"
    extract-zip "1.6.7"
    fs-extra "5.0.0"
    getos "3.1.1"
    is-ci "1.2.1"
    is-installed-globally "0.1.0"
    lazy-ass "1.6.0"
    listr "0.12.0"
    lodash "4.17.15"
    log-symbols "2.2.0"
    minimist "1.2.0"
    moment "2.24.0"
    ramda "0.24.1"
    request "2.88.0"
    request-progress "3.0.0"
    supports-color "5.5.0"
    tmp "0.1.0"
    untildify "3.0.3"
    url "0.11.0"
    yauzl "2.10.0"

note: that we're still using 3.6.1 in package.json. And we can confirm amplify is using 3.6.1 when running the test.

I tried this and now I’m not generating the aws-imports file anymore while I am doing the amplify simplePush step for the backend. Would be great if someone from aws could offer some assistance while I’m unable to deploy my app since October

[kahdojay]

@ascorbic the issue I'm referring to is not specific to PR previews but with large deploy artifacts. Do you have another GitHub issue I could look at that describes your problem? If not please feel free to open a new one so we can get some more context.

[kahdojay]

@kkesley-shine @rickiesmooth can you both please share your respective buildspecs (if possible, from the time you were experiencing the issue, or a buildspec with which you are able to reproduce the issue)?

Also @rickiesmooth if you could tell me a specific build number (for dmm4bu2tim2c; eu-central-1) that I can look at when this happens, it would help a lot. (We're going to start asking for these for new issues to save time)

[kahdojay]

@desertmonkey same for you if you are able, please - specific build number, as well as Amplify ARN (for the app id and region), and a buildspec with which you can reproduce the issue.

[kkesley-shine]

Hi @kahdojay

Here's our amplify.yml

version: 0.1
backend:
  phases:
    build:
      commands:
        - "# Execute Amplify CLI with the helper script"
        - amplifyPush --simple
    postBuild:
      commands:
        - nvm use $VERSION_NODE_12
        - yarn install
        - yarn run test:scripts
        - yarn run data-loader
        - yarn run test:integration
frontend:
  phases:
    preBuild:
      commands:
        - nvm use $VERSION_NODE_12
        - yarn install
        - yarn run test
    build:
      commands:
        - yarn run build
  artifacts:
    baseDirectory: build
    files:
      - "**/*"
  cache:
    paths:
      - node_modules/**/*
test:
  artifacts:
    baseDirectory: cypress
    configFilePath: '**/mochawesome.json'
    files:
      - '**/*.png'
      - '**/*.mp4'
  phases:
    preTest:
      commands:
        - nvm use $VERSION_NODE_12
        - yarn install
    test:
      commands:
        - npm run ci:e2e
    postTest:
      commands:
        - npx mochawesome-merge --reportDir cypress/report/mochawesome-report > cypress/report/mochawesome.json

and scripts in our package.json

"scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test --testPathIgnorePatterns=\"src/integration\"",
    "test:scripts": "jest scripts/",
    "test:lambda": "jest --testPathPattern=\"amplify/backend/function\" --config=amplify/backend/function/jest.config.json",
    "test:integration": "react-scripts test src/integration",
    "test:all": "jest scripts/ && yarn test:lambda && react-scripts test",
    "ci:cypress": "cypress run --reporter mochawesome --reporter-options \"reportDir=cypress/report/mochawesome-report,overwrite=false,html=false,json=true,timestamp=mmddyyyy_HHMMss\"",
    "ci:e2e": "start-server-and-test start 3000 ci:cypress",
    "eject": "react-scripts eject",
    "lint": "eslint . --ext .js,.jsx,.ts,.tsx",
    "data-loader": "node ./scripts/src/data-loader/main.js"
  },

Thanks!

And we also noticed that cypress 3.6.1 is getting stuck recently. And coincidently, there's a 3.7.0 out last week.

We also noticed amplify is installing cypress globally when provisioning the build. Do we have to use the one installed globally?

UPDATE

I tried installing 3.7.0 but it's still stuck in the testing phase.. Last log is:

Running: signup_spec.js (4 of 4)

It's master build #47 if you need the details..

But updating cypress to v3.7.0 in yarn.lock does not give me "Failed to deploy" error.

[wajjs]

I opened issue #287 "@wajjs how big is your node_modules folder? We have a known issue in #115 which we are actively working on. The temporary workaround is to delete unused branches in your app."

For me the deploy to hangs around 14000 files. My node_modules folder is 350 MB.

[desertmonkey]

@desertmonkey same for you if you are able, please - specific build number, as well as Amplify ARN (for the app id and region), and a buildspec with which you can reproduce the issue.

Thanks @kahdojay, as my colleague @cawleyflower mentioned above, we changed our build procedures (including moving all our binary files to a CDN) and haven't seen the issue since.

For the record, our ARN is arn:aws:amplify:us-west-2:054500231008:apps/d1ev2ep2azk4pn

[kahdojay]

@kkesley-shine Thanks for sharing. For those having build issues with Cypress, please use version 3.4.1 as a workaround for now. We’ll update this thread once we’ve added full support to properly execute later versions. If builds continue to fail with 3.4.1, please let us know. @rickiesmooth

[ascorbic]

@kahdojay I'm getting failures during deploy with the same error, but not on the master branch. I'm not using Cypress, and the site is around 13k files. My ARN is arn:aws:amplify:eu-west-1:615693933970:apps/d301w3ojpkhzjt.

[kahdojay]

@ascorbic will take a look, can you also share the build number? @wajjs will take a look, thanks for sharing your details in the other thread

[kahdojay]

@wajjs in #287 you provided a job arn that has a different account and app id than the app id provided. Can you confirm which app, branch, and job id we should look at?

[alana314]

Hi, we are also having this issue, deploy stuck for three days, app d3mckmn7bfao41 branch master build 4 in oregon

[kahdojay]

@jordan314 there are a few different underlying issues being discussed in this thread. I should've asked for this earlier, but can you (and anyone requesting assistance) please provide more details, as provided in our updated issue template

This will help us save time in investigating and allow us to help more people. Thanks!

[alana314]

Hi, we've provided more details in support ticket 6655323281, a lot of the content is sensitive

[ascorbic]

@kahdojay See build d301w3ojpkhzjt/search-page/7 for an example

[kahdojay]

@ascorbic This looks like a different issue, I see the build failed outright, you should see it in your console - if not please share a screenshot from that build of what you see. I also see an error regarding 'gatbsy-source-drupal' in the log that I believe you should as well.

[ascorbic]

@kahdojay No, the build succeeded. You must be looking at the wrong one. This is the one: https://eu-west-1.console.aws.amazon.com/amplify/home?region=eu-west-1#/d301w3ojpkhzjt/search-page/7

[kahdojay]

@ascorbic thanks, you're right I was looking at the wrong one. It looks like you're running into the large artifact issue as well. We are actively working on a fix for this, in the meantime if you have any unused branches, you can try deleting them. Another potential workaround would be to deploy to a separate Amplify app.

[ajhool]

@kahdojay

It seems like there are multiple issues going on in this thread and the request to "provide more details" is difficult when the only details provided from the Console logs are:

2019-12-28T21:31:00 [INFO]: Starting Deployment 2019-12-28T21:31:00 [ERROR]: Failed to deploy 2019-12-28T21:31:00 [INFO]: Deployment finished.

We're deploying a basic Vue.js app (the app is generated using the official vue-cli provided by the Vue.js team). We are using one branch although would like to use many branches for gitflow as advertised by Amplify Console (https://dev.to/kkemple/branch-based-deployment-strategies-with-aws-amplify-console-1n3c)

Our app id is: d1v4fadcbu6950

Desired functionality:

• The app deployment should work correctly or basic debugging output should be provided
• A troubleshooting page should be available for known issues (eg. deployment size, cypress cache, etc.)
• Amplify Console should be proactive in providing error feedback, rather than coming to github to post an AppId to find out which known-but-not-documented issue we are running into.
• Cypress should not interfere with deployment, as advertised by Amplify Console, here: https://docs.aws.amazon.com/amplify/latest/userguide/running-tests.html

[hellokingdom]

arn:aws:amplify:eu-west-2:673653084816:apps/d26tysq5iwv37v

Failing to deploy if tests are enabled.

The tests complete successfully but instant fail on deploy step.

[germain-receeve]

Same issue here, downgrading to cypress 3.4.1 didn't help. Did anybody figured out a way around this issue?

[swaminator]

@ajhool @germain-receeve @hellokingdom we are aware of the Cypress issue causing deploy failures. Please refer to #283. We are actively working on a fix here and will let you know when that is rolled out. Apologies for the inconvenience. In the meantime to unblock yourself, set an environment variable called USER_DISABLE_TESTS to true to disable your tests from running.

[cslogan-red]

Quick update: fixes are pending release. This issue has two parts (and each applied only when using some type of test suite via the test phase); an improperly bubbled exception scenario resulting in builds that either; hung until the build timeout, or builds that showed "successful" in the Console and immediately failed to deploy, each without logging the full exception details, as well as a command update.

For existing apps, the latest version of mochawesome-merge required a slight change to the final postTest command. Below is the correct test section edit for Cypress 3.4.1^ & mochawesome-merge 3^ (our docs are being updated to reflect this and you can apply this now to get tests working with these versions):

Final postTest command should be updated to:

npx mochawesome-merge cypress/report/mochawesome-report/mochawesome*.json > cypress/report/mochawesome.json

Additionally, the first preTest command has been changed from "npm install" to "npm ci" to ensure the Cypress binary is always downloaded as this seems to be more reliable. These changes will get Cypress correctly running again, using their kitchen-sink repo as an example.

As stated above, these fixes are pending release and should be GA by 1/23. These updates will cause builds with actual exceptions that occur during the test step to be correctly logged to the Console's Test log, and the Test step will fail as desgined, as opposed to the previous behavior observed.

[spencer-aerian]

@kahdojay No, the build succeeded. You must be looking at the wrong one. This is the one: https://eu-west-1.console.aws.amazon.com/amplify/home?region=eu-west-1#/d301w3ojpkhzjt/search-page/7

Taking the mantel from @ascorbic as he's moved over to other things , we are STILL seeing this Failed to Deploy error in the Deploy console. Whats' the actual ETA of this getting into production? Tests disabled, verify is disabled etc.

[z3by]

We are facing the same issue here, is this gonna be solved soon?

[eterry1388]

Same issue for us. Build works fine without specifying test. But as soon as we put in test in the amplify.yml config in any fashion, it always fails.

[MrBokeh]

I found a working solution by running custom script such as node cypress.js and then merge the report with mochawesome-merge... which is quite hacky...

test:
  artifacts:
    baseDirectory: cypress
    configFilePath: '**/mochawesome.json'
    files:
      - '**/*.png'
      - '**/*.mp4'
  phases:
    preTest:
      commands:
        - npm install
        - npm install wait-on
        - 'npm start & npx wait-on http://localhost:3000'
    test:
      commands:
        - npm run cypress:custom
    postTest:
      commands:
        - npx mochawesome-merge --reportDir mochawesome-report > cypress/report/mochawesome.json

[AndriiChubariev]

Hi, I have started to experience the same issue today. The build is green, failed to deploy.

Using Gatsby site with a lot of images. Used to work fine on Saturday.

[sashasochka]

We are having the same issue.

[z3by]

Please re-open this Issue, has it been solved?

[AndriiChubariev]

For more details, my build artifact is about ~1Gb, a lot of images. I had two environments: dev and prod. After I removed dev and left only one environment the issue just fixed itself and not reproducing anymore.

I understand that it is not an option for many users to delete a dev environment. It worked in my case just because it is a hobby project.

[deybhayden]

Hey there - are there any updates on this issue? We have a build artifact that's 55 MB but is still suffering the same fate. Is there a recommended artifact size to avoid this problem?

[cslogan-red]

This issue has become a placeholder for three distinct issues that all caused similar behavior in Amplify Console, a Deploy step with a sometimes immediate (and sometimes after deploying many files) "Failed to deploy" message.

The first issue has been fixed and released as described above and was related to the use of the Test step, exceptions here weren't always being caught and bubbled up to your build logs properly, so the Test step was actually failing and it was not being bubbled up, resulting in the "failed to deploy" behavior. Note this doesn't mean you'll never have failed test steps, this step can fail for dozens and dozens of reasons, you're just going to see it as a test failure with proper logs now

The remaining two issues are being actively worked on, the highest priority of which has a release coming soon, and we'll update this thread once the release is available in all regions.

For these remaining two issues, the failure to deploy is compounded by two things, the number of branches you have connected to a single Amplify Console app, and the number of deployed files you have in your app.

We're seeing this issue most commonly in (very) large static Gatsby sites with many static images; the suggested workaround is the same no matter what, reduce the number of branches you have connected to each app down to a single Production branch if necessary (you can connect multiple apps using the same repo using a single branch per app if you have to, in order to still test each branch). The forthcoming update will resolve this limitation.

[deybhayden]

Hey there - thanks for the update @cslogan-red! I wanted to drop a note that the interim fix did work for our project - deleting everything but the single branch. This included the PR preview branches as well.

[gianlucapisati]

Hi I have the same error, even if deleting all the other branch except the production one... this is my arn arn:aws:amplify:eu-west-1:135446448955:apps/d1xjyanq2k93xx

[FemkeBuijs]

I also still experience the same problem. With cypress tests the error 020-03-19T10:45:32 [ERROR]: Failed to deploy remains, while the tests are successful. Build settings:

 test:
   artifacts:
     baseDirectory: cypress
     files:
       - '**/*.png'
   phases:
     test:
       commands:
         - 'npx cypress run'

When commenting out these settings there is no error and the deploy is successful.

[alana314]

Honestly our solution was to stop using amplify and use codebuild / S3 / cloudfront. Something like this
https://medium.com/@muaohua/deploy-your-single-page-application-to-s3-and-cloudfront-with-code-build-and-code-pipeline-4ae6e5c12018

[robmarshall]

I am still getting this issue

[amuresia]

+1

[grudra7714]

+1

[plumbis]

Is there an ETA for a fix for this issue? The current workaround of deleting the app and recreating it is a major challenge since it requires updating DNS, environmental variables and more. This has impacted our production environment multiple times now.

[litwicki]

@plumbis the team is actively working on this as I type and planning to have an update in the next several days. I or the team will update this thread as soon as we have a more information to share.

[robmarshall]

I needed to deploy my clients projects sooner rather than later so cracked on with Codepipeline. This walks through the steps needed: https://thoughtsandstuff.com/deploy-gatsby-aws-with-ci-without-amplify

[adriencg]

We're currently working on testing & deploying the fix, and sincerely apologize for all the inconvenience this has caused.

[stigalvestad]

Is there any news on this case, @adriencg ?

[michalmikolajczyk]

Hi we have the same problem. Our app ARN arn:aws:amplify:us-west-2:329268793000:apps/d2o4y68d25axi5

This seems related to the tests. The problem only started showing after I added the test: paragraph to amplify.yml and when it was removed, the build deploys. With that section, I get

2020-05-05T08:48:40 [INFO]: Starting Deployment
2020-05-05T08:48:40 [ERROR]: Failed to deploy
2020-05-05T08:48:40 [INFO]: Deployment finished.

I worked around this problem by moving the commands for tests into the preBuild phase for frontend (I just added one command yarn test tbh)

[d-luk]

Thanks for the workaround @michalmikolajczyk.

We are having the exact same problem. With the test step the deployment fails, but without it it succeeds. We are also just running yarn test on our Create React App project.

@adriencg Can you give an update on the fix you mentioned?

[steinybot]

Are the remaining issue related to the size of the deployment artifacts size? As far as I know my artifacts are not large.

My cache is probably very large though. Is there even a way to invalidate/expire the cache? The docs are very light on the subject.

I respect that fixing this issue may be complicated but this is a "P0 - loss of major functionality - do not pass Go do not collect $200" kind of bug and it has been quite a long time. Can we at least get a more detailed error message so we have some hope of working around it ourselves?

I really want to like amplify but I have encountered so many different bugs and usability issues in the past 2 weeks of using it that I'm not sure it is worth the hassle. Most of the time I have just resorted to deleting everything and starting again which sadly works but my good will is running low.

[steinybot]

I noticed something odd with my builds.

The first one which failed with the infamous "Failed to deploy" message actually has the Test phase marked as pending. I'm not sure if that has been mentioned before or not.

The next one where I tried removing parts of the test phase shows the opposite problem. It shows the build phase as having failed but it was actually the test phase (I had not specified a baseDirectory for the artifacts).

In case anyone wants to take a closer look the App ARN is arn:aws:amplify:ap-southeast-2:844681392632:apps/d1855l7ntw7ham, preview build pr-5, builds 3 and 4.

[simpson]

We've been experiencing this over the past few weeks