Frontend is deploying when tests fail

[AndyW22]

Before opening, please confirm:

• [X] I have checked to see if my question is addressed in the FAQ.
• [X] I have searched for duplicate or closed issues.
• [X] I have read the guide for submitting bug reports.
• [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

App Id

d29mr6lvexeqms

Region

us-east-1

Amplify Console feature

E2E tests, Logging

Describe the bug

if the E2E tests on my app fail, the app gets deployed regardless. The UI will show the test has failed and the "deploy" stage hasn't started:

But the site has been deployed, it has all the changes in the commit on the frontend, and the Next JS lambdas also get deployed.

I have since remade the entire amplify app, but still have the same problem.

Expected behavior

Frontend doesnt deploy if the tests fail

Reproduction steps

Unfortunately I'm unsure what causes this. I've tried removing certain resources but the problem was likely there when the tests were passing so I'm yet to find the cause.

Build Settings

version: 1
backend:
  phases:
    build:
      commands:
        - amplifyPush --simple
frontend:
  phases:
    preBuild:
      commands:
        - nvm install 16
        - yarn install
        - yarn run lint
    build:
      commands:
        - yarn next build
  artifacts:
    baseDirectory: .next
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*
      - .next/cache/**/*
test:
  phases:
    preTest:
      commands:
        - nvm install 16
        - yarn add mocha mochawesome mochawesome-merge mochawesome-report-generator
    test:
      commands:
        - yarn start-test 'NEXT_PUBLIC_ISTEST=true next build && next start' 3000 'CYPRESS_TEST_USER=$TEST_USER CYPRESS_ENVIRONMENT=staging ELECTRON_ENABLE_LOGGING=1 yarn cypress run --reporter mochawesome --reporter-options "reportDir=cypress/report/mochawesome-report,overwrite=false,html=false,json=true,timestamp=mmddyyyy_HHMMss"'
    postTest:
      commands:
        - yarn mochawesome-merge cypress/report/mochawesome-report/mochawesome*.json > cypress/report/mochawesome.json
  artifacts:
    baseDirectory: cypress
    configFilePath: '**/mochawesome.json'
    files:
      - '**/*.png'
      - '**/*.mp4'
  cache:
    paths:
      - node_modules/**/*
      - .next/cache/**/*

### Additional information

_No response_

[ghost]

HI @AndyW22 👋🏽 thank you for raising this! There may be a bug where some edge cases allow this to happen. I will reproduce this behavior and determine if there is a workaround.

[ghost]

@AndyW22 apologies for the delay, can you try implementing your tests using these build scripts for the test phase:

test:
  phases:
    preTest:
      commands:
        - npm install
        - npm install start-server-and-test cypress mocha mochawesome mochawesome-merge mochawesome-report-generator --save-dev
    test:
      commands:
        - npm run build
        - npx start-test 'npm start' 3000 'npx cypress run --reporter mochawesome --reporter-options "reportDir=cypress/report/mochawesome-report,overwrite=false,html=false,json=true,timestamp=mmddyyyy_HHMMss"'
    postTest:
      commands:
        - npx mochawesome-merge cypress/report/mochawesome-report/mochawesome*.json > cypress/report/mochawesome.json
  artifacts:
    baseDirectory: cypress
    configFilePath: '**/mochawesome.json'
    files:
      - '**/*.png'
      - '**/*.mp4'

[AndyW22]

@AndyW22 apologies for the delay, can you try implementing your tests using these build scripts for the test phase:

test:
  phases:
    preTest:
      commands:
        - npm install
        - npm install start-server-and-test cypress mocha mochawesome mochawesome-merge mochawesome-report-generator --save-dev
    test:
      commands:
        - npm run build
        - npx start-test 'npm start' 3000 'npx cypress run --reporter mochawesome --reporter-options "reportDir=cypress/report/mochawesome-report,overwrite=false,html=false,json=true,timestamp=mmddyyyy_HHMMss"'
    postTest:
      commands:
        - npx mochawesome-merge cypress/report/mochawesome-report/mochawesome*.json > cypress/report/mochawesome.json
  artifacts:
    baseDirectory: cypress
    configFilePath: '**/mochawesome.json'
    files:
      - '**/*.png'
      - '**/*.mp4'

I've tried this script and has the same result, frontend is still getting deployed when the tests fail