Cypress will not stop test run if errors in before hook

[andreiashu]

• Operating System: osx
• Cypress Version: 2.0.2
• Browser Version: Canary 66

Is this a Feature or Bug?

Bug

Current behavior:

If an error is raised in the before hook for tests, Cypress will not stop the test run and seems to continue indefinitely.

This behaviour seems identical to #674

Desired behavior:

Tests should fail with a descriptive error message.

How to reproduce:

Run the below code

Test code:

describe('Demo throw in before hook', () => {
  before(() => {
    cy.exec('echo "test test"').then(() => {
      throw new Error('whoa something broke');
    });
  });

  it('visits url', () => {
    cy.get('/');
  });
  // NB: for some reason if I comment out the below test
  // Cypress UI will fail the test correctly.
  it('gets element', () => {
    cy.get('.someelem');
  });
});

Additional Info (images, stack traces, etc)

Below is a screenshot that shows that although the error is thrown, the spinner still continues to... spin :)

[andreiashu]

haven't tested but I suspect that after() hook has a similar issue.

[brian-mann]

Can confirm - but this is only in the before. Changing to after does not create the same problem.

I'm investigating to understand if this is a regression. To be clear, it only happens in cypress open mode, cypress run does not hang.

[brian-mann]

Okay did my homework

• this is not a regression - it exists in 1.4.2
• it's purely a visual bug. the test "appears" to hang, but actually its not really hanging
• other tests outside of the suite that used the hook will continue to run
• cypress run will run to completion

All that's a problem is that we're not visually showing the error and the test's state is not displayed. The test continues to "spin" but it should be errored, and we should show the error that explains that the remaining tests in the suite have been skipped.

[brian-mann]

The root cause of this is a bug in mocha - it's short circuiting some login in certain situations and we don't account for it.

Mocha is not correctly running root level after hooks in the case where a before fails.

Additionally this bug only crops up when the test has siblings - if its the last test in a suite, it will work correctly.

[brian-mann]

Nevermind - mocha is not the problem here. We are just not taking into account the edge case where a test fails in a before hook and has sibling tests. We assume they will run, but they won't since mocha skips them. We just need a new branch of logic to take this into account.

[brian-mann]

This PR should fix this. Thank you for opening a good issue.

[brian-mann]

Released in 2.0.3.

[quantizor]

FWIW I'm seeing this in "run" mode as well as a result of a page timeout during before. The screenshot gets written but the process never exits.

[podarok]

I can confirm this is still exists even in 3.1.0

tail -n10000 -f cypress.stdout 

================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:  3.1.0                                                            │
  │ Browser:  Electron 59 (headless)                                           │
  │ Specs:    1 found (hello_world_spec.js)                                    │
  └────────────────────────────────────────────────────────────────────────────┘

────────────────────────────────────────────────────────────────────────────────

  Running: hello_world_spec.js...                                      (1 of 1) 
    <section class="suite">
      <h1>My First Test</h1>
      <dl>
        <dt class="error">Does not do much!</dt>
TypeError: Cannot read property 'replace' of undefined
    at Object.exports.clean (/root/.cache/Cypress/3.1.0/Cypress/resources/app/packages/server/node_modules/mocha/lib/utils.js:265:5)
    at Runner.<anonymous> (/root/.cache/Cypress/3.1.0/Cypress/resources/app/packages/server/node_modules/mocha/lib/reporters/doc.js:58:35)
    at emitTwo (events.js:130:20)
    at Runner.emit (events.js:213:7)
    at Reporter.emit (/root/.cache/Cypress/3.1.0/Cypress/resources/app/packages/server/lib/reporter.js:239:55)
    at Object.server.startWebsockets.onMocha (/root/.cache/Cypress/3.1.0/Cypress/resources/app/packages/server/lib/project.js:296:22)
    at Socket.<anonymous> (/root/.cache/Cypress/3.1.0/Cypress/resources/app/packages/server/lib/socket.js:237:36)
    at emitThree (events.js:135:13)
    at Socket.emit (events.js:216:7)
    at /root/.cache/Cypress/3.1.0/Cypress/resources/app/packages/socket/node_modules/socket.io/lib/socket.js:503:12
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickCallback (internal/process/next_tick.js:180:9)

TypeError: Cannot read property 'replace' of undefined
    at Object.exports.clean (/root/.cache/Cypress/3.1.0/Cypress/resources/app/packages/server/node_modules/mocha/lib/utils.js:265:5)
    at Runner.<anonymous> (/root/.cache/Cypress/3.1.0/Cypress/resources/app/packages/server/node_modules/mocha/lib/reporters/doc.js:58:35)
    at emitTwo (events.js:130:20)
    at Runner.emit (events.js:213:7)
    at Reporter.emit (/root/.cache/Cypress/3.1.0/Cypress/resources/app/packages/server/lib/reporter.js:239:55)
    at Object.server.startWebsockets.onMocha (/root/.cache/Cypress/3.1.0/Cypress/resources/app/packages/server/lib/project.js:296:22)
    at Socket.<anonymous> (/root/.cache/Cypress/3.1.0/Cypress/resources/app/packages/server/lib/socket.js:237:36)
    at emitThree (events.js:135:13)
    at Socket.emit (events.js:216:7)
    at /root/.cache/Cypress/3.1.0/Cypress/resources/app/packages/socket/node_modules/socket.io/lib/socket.js:503:12
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickCallback (internal/process/next_tick.js:180:9)

      </dl>
    </section>
q^C

cypress waiting for the ^C to stop command to run

CYPRESS_baseUrl="http://gggg/build${PARENT_BUILD_NUMBER}/docroot" node_modules/cypress/bin/cypress run --reporter=doc > ../build_reports/cypress.stdout

[jennifer-shehane]

HEy @podarok, could you provide a reproducible example? Can you share you test code?

[podarok]

code

const paragraphType = 'Teaser';
const contentTitle = '4D FITNESS GOALS';
const contentImg = '010619 4d fitness goals';
const contentDescription = 'Throughout the year, Y locations offer: Baseball/Softball/T-ball, Basketball, Cheerleading, Dodgeball, Flag Football, Floor Hockey, Lacrosse, Martial Arts, Running Club, Soccer, Special Olympics Young Athletes Program, Track & Field, Tumbling, Swim Team and Volleyball.';
const contentUrl = '/';
const contentUrlText = 'About';
it('Create landing page with ' + paragraphType + ' paragraph type in all areas', function () {

  cy.visit('user/login', {
    auth: {
      username: Cypress.env('basic_auth_username'),
      password: Cypress.env('basic_auth_passwd')
    }
  }).wait(2000);

  cy.get('#edit-name').type(Cypress.env('drupal_auth_username'), {force: true})
    .get('#edit-pass').type(Cypress.env('drupal_auth_passwd'), {force: true})
    .get('#edit-submit').click({force: true}).wait(2000);

  cy.visit('/node/add/landing_page', {
    auth: {
      username: Cypress.env('basic_auth_username'),
      password: Cypress.env('basic_auth_passwd')
    }
  }).wait(2000);

  cy.get('#edit-title-0-value').type('Style guide for '+ paragraphType + ' paragraph type in all areas', {force: true})
    .get('#edit-field-lp-layout').select('One Column', {force: true});

  cy.get('#edit-group-content-area > summary').click({force: true}).wait(2000)
    .get('#edit-group-content-area > div > div > div > div > div > div > ul > li').contains('Add ' + paragraphType).click({force: true}).wait(2000)
    .get('#edit-group-content-area').contains('Title').siblings('input').type(contentTitle, {force: true}).wait(2000)
    .get('#edit-group-content-area').contains('Image').click({force: true}).wait(2000)
    .get('#edit-group-content-area').contains('Select images').click({force: true}).wait(4000)
    .get('#entity_browser_iframe_images_library').then(function ($iframe) {
    let $body = $iframe
      .contents().find('body');
    cy.wrap($body).find('a').contains('All Images').click({force: true}).wait(4000)
      .get('#entity_browser_iframe_images_library').then(function ($iframe) {
      let $body = $iframe
        .contents().find('body');
      cy.wrap($body).find('#edit-name').type(contentImg, {force: true}).wait(2000)
        .wrap($body).find('#edit-submit-images-library').click({force: true}).wait(2000)
        .wrap($body).find('img').click({force: true}).wait(2000)
        .wrap($body).find('#edit-submit').click({force: true}).wait(2000)
    });
  })
    .get('#edit-group-content-area').contains('Text format').siblings('select').select('Code', {force: true}).wait(2000)
    .get('#edit-group-content-area').contains('Description').siblings('div').children('textarea').type(contentDescription, {force: true}).wait(2000)
    .get('#edit-group-content-area').contains('URL').siblings('input').type(contentUrl, {force: true}).wait(2000)
    .get('#edit-group-content-area').contains('Link text').siblings('input').type(contentUrlText, {force: true}).wait(2000)

  cy.get('#edit-moderation-state-0-state').select('Published', {force: true});

  cy.get('#edit-submit').click({force: true}).wait(2000);
});

error

TypeError: Cannot read property 'replace' of undefined
    at Object.exports.clean (/root/.cache/Cypress/3.1.0/Cypress/resources/app/packages/server/node_modules/mocha/lib/utils.js:265:5)

[jennifer-shehane]

@podarok This issue involved a similar error message: https://github.com/cypress-io/cypress/issues/1669

Unfortunately, I am unable to run the test code provided locally and track this down, since there is no base url to visit provided.