cy.route2 errors leak between tests

[tony-g]

Current behavior

I have a few tests that I'd like to share some network stubs. My first thought was to use before:

context('Context', () => {
  before(() => {
    cy.route2(...)
  })

  it('test a', () => {
    doSomethingThatShouldTriggerStub()
    // stub is triggered
  })

  it('test b', () => {
    doSomethingThatShouldTriggerStub()
    // stub is not triggered
  })
})

But the stubbed route isn't getting triggered on the 2nd test. I wasn't sure if that was intentional but I assumed it was either intentional or WIP and moved to beforeEach:

context('Context', () => {
  beforeEach(() => {
    cy.route2(...)
  })
  ...
})

beforeEach solved the problem with the stubs only working on the first test, but seemed to create a sort of race condition; (I think) when a stubbed request happens in between tests. (e.g. if the app is polling in the background)

Different errors depending on the timing

I don't understand the cypress code well enough to figure out how to control the timing for a consistent repro, but I was able to track down a couple different places where it's failing.

Cannot access 'continueSent' before initialization

• Looks like the function can bail before continueSent gets initialized?

https://github.com/cypress-io/cypress/blob/052892d79f95f9792e00d426722a9d904b572df4/packages/driver/src/cy/net-stubbing/events/request-received.ts#L51-L102

ReferenceError: The following error originated from your test code, not from Cypress.

  > Cannot access 'continueSent' before initialization

When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.
    at sendContinueFrame (http://localhost:5000/__cypress/runner/cypress_runner.js:164953:5)
    at onRequestReceived (http://localhost:5000/__cypress/runner/cypress_runner.js:164976:12)
    at http://localhost:5000/__cypress/runner/cypress_runner.js:164832:14
From previous event:
    at $Cypress.<anonymous> (http://localhost:5000/__cypress/runner/cypress_runner.js:164830:56)
    at $Cypress.../driver/node_modules/eventemitter2/lib/eventemitter2.js.EventEmitter.emit (http://localhost:5000/__cypress/runner/cypress_runner.js:85891:19)
    at $Cypress.parent.<computed> [as emit] (http://localhost:5000/__cypress/runner/cypress_runner.js:173409:33)
    at Socket.<anonymous> (http://localhost:5000/__cypress/runner/cypress_runner.js:199986:17)
    at Socket.../socket/node_modules/component-emitter/index.js.Emitter.emit (http://localhost:5000/__cypress/runner/cypress_runner.js:187739:20)
    at Socket.../socket/node_modules/socket.io-client/lib/socket.js.Socket.onevent (http://localhost:5000/__cypress/runner/cypress_runner.js:194467:10)
    at Socket.../socket/node_modules/socket.io-client/lib/socket.js.Socket.onpacket (http://localhost:5000/__cypress/runner/cypress_runner.js:194425:12)
    at Manager.<anonymous> (http://localhost:5000/__cypress/runner/cypress_runner.js:187597:15)
    at Manager.../socket/node_modules/component-emitter/index.js.Emitter.emit (http://localhost:5000/__cypress/runner/cypress_runner.js:187739:20)
    at Manager.../socket/node_modules/socket.io-client/lib/manager.js.Manager.ondecoded (http://localhost:5000/__cypress/runner/cypress_runner.js:193923:8)
    at Decoder.<anonymous> (http://localhost:5000/__cypress/runner/cypress_runner.js:187597:15)
    at Decoder.../socket/node_modules/component-emitter/index.js.Emitter.emit (http://localhost:5000/__cypress/runner/cypress_runner.js:187739:20)
    at Decoder.../socket/node_modules/socket.io-circular-parser/index.js.Decoder.add (http://localhost:5000/__cypress/runner/cypress_runner.js:192571:12)
    at Manager.../socket/node_modules/socket.io-client/lib/manager.js.Manager.ondata (http://localhost:5000/__cypress/runner/cypress_runner.js:193913:16)
    at Socket.<anonymous> (http://localhost:5000/__cypress/runner/cypress_runner.js:187597:15)
    at Socket.../socket/node_modules/engine.io-client/node_modules/component-emitter/index.js.Emitter.emit (http://localhost:5000/__cypress/runner/cypress_runner.js:190216:20)
    at Socket.../socket/node_modules/engine.io-client/lib/socket.js.Socket.onPacket (http://localhost:5000/__cypress/runner/cypress_runner.js:188283:14)
    at WS.<anonymous> (http://localhost:5000/__cypress/runner/cypress_runner.js:188100:10)
    at WS.../socket/node_modules/engine.io-client/node_modules/component-emitter/index.js.Emitter.emit (http://localhost:5000/__cypress/runner/cypress_runner.js:190216:20)
    at WS.../socket/node_modules/engine.io-client/lib/transport.js.Transport.onPacket (http://localhost:5000/__cypress/runner/cypress_runner.js:188726:8)

And then I was also sometimes experiencing it here, where request can apparently be undefined:

https://github.com/cypress-io/cypress/blob/052892d79f95f9792e00d426722a9d904b572df4/packages/driver/src/cy/net-stubbing/events/response-received.ts#L45-L51

Confusing things

Easy for me to imagine this is all my user error, but a few confusing things:

• The error message is a bit confusing, as the errors seem to be coming from cypress code (though obviously caused by my test code)
• The error seems to leak into other commands like click shown above, even though it seems to be coming from the network stubbing.
• The error was also sometimes causing click to fail silently (cypress said it had clicked, but didn't perform the click). This seems to happen if the error occurs in between the "actionability" checks.

state('error') was returning the stubbing error, and click was then failing silently (the cypress UI says it completed, but there's no click)

https://github.com/cypress-io/cypress/blob/052892d79f95f9792e00d426722a9d904b572df4/packages/driver/src/cy/retries.js#L101

Desired behavior

• Ideally I'd like to be able to define stubs for a set of tests in the before hook.
• If that's not possible or intended, it would be nice if there was a way to prevent these errors. (It would be pretty simple to initialize continueSent earlier and put a null check around request.response=, but I obviously don't understand the code remotely well enough to know if that creates other issues :)
• Or if this is outside the intended usage, might be nice if the error exploded things more (like if you try to run cy.etc outside a test)

Test code to reproduce

Because of the finicky timing, I was unable to get a consistent minimal repro, but this repros about 50% of the time for me. The crux seems to be that the app fires off a stubbed request in between the tests.

"App"

<div data-cy="buttons">
  <button id="aa">button aa</button>
  <button id="bb">button bb</button>
  <button id="cc">button cc</button>
  <button id="clear">clear</button>
</div>
<div id="result" data-cy="result"></div>
<script>
  document.getElementById('clear').addEventListener('click', () => {
    document.getElementById('result').innerHTML = ''
  })
  ;['aa', 'bb', 'cc'].forEach((id) => {
    document.getElementById(id).addEventListener('click', async () => {
      const response = await window.fetch(`/${id}`)
      const { data } = await response.json()
      console.log('click', id, { data })

      // Fire off a bunch of extra requests to try to trigger the error
      for (let i = 0; i < 5 ; i++) {
        setTimeout(() => {
          window.fetch(`/${id}`)
        }, i * 20)
      }

      document.getElementById('result').innerHTML = data
    })
  })
</script>

Test code

const url = 'http://localhost:5000'

context('Trying to repro', () => {
  before(() => {
    cy.visit(url)
  })

  beforeEach(() => {
    cy.route2({ method: 'GET', pathname: /[a-z]+/ }, (req) => {
      req.reply({ data: req.url })
    })
  })
  ;['aa', 'bb'].forEach((data) => {
    it(`clicks button ${data}`, () => {
      cy.get(`#clear`).click()
      cy.get(`[data-cy=buttons]`).contains(data).click()
      cy.get(`[data-cy=result]`).contains(`${url}/${data}`)
    })
  })
})

Versions

Cypress 5.4.0 Electron 85 macOS 10.15.7

PS: Cypress is amazing, thanks!

[jennifer-shehane]

I can reproduce this pretty consistently with the example below - meaning it happens in 1/20 of the tests or so. The presense of a beforeEach seems important to reproduce.

index.html

<div data-cy="buttons">
  <button id="aa">button aa</button>
  <button id="clear">clear</button>
</div>
<div id="result" data-cy="result"></div>
<script>
  document.getElementById('clear').addEventListener('click', () => {
    document.getElementById('result').innerHTML = ''
  })
  document.getElementById('aa').addEventListener('click', async () => {
    const response = await window.fetch(`/${'aa'}`)
    const { data } = await response.json()
    console.log('click', 'aa', { data })

    // Fire off a bunch of extra requests to try to trigger the error
    for (let i = 0; i < 5; i++) {
      setTimeout(() => {
        window.fetch(`/${'aa'}`)
      }, i * 20)
    }

    document.getElementById('result').innerHTML = data
  })
</script>

spec.js

context('Trying to repro', () => {
  // the presense of the beforeEach seems required to reproduce
  beforeEach(() => {}) 

  Array(20).fill(0).forEach((val, i) => {
    it(`test ${i}`, () => {
      cy.visit('index.html')
      cy.route2({ method: 'GET', pathname: /[a-z]+/ }, (req) => {
        req.reply({ data: req.url })
      })
      cy.get(`#clear`).click()
      cy.get(`[data-cy=buttons]`).contains('aa').click()
      cy.get(`[data-cy=result]`).contains('aa')
    })
  })
})

[tony-g]

Phew, thanks! - I was afraid it would be hard to repro.

Here are a couple possibly related repros that might be useful for further investigation - (They may be separate bugs, but I didn't want to spam you and your team with issues before understanding if I am using it wrong)

cy.route2 in each test

Your mention of beforeEach being important reminded me of another thing I tried - putting the cy.route2 call in each test:

index.html (same as for the beforeEach scenario)

<div data-cy="buttons">
  <button id="aa">button aa</button>
  <button id="bb">button bb</button>
  <button id="cc">button cc</button>
  <button id="clear">clear</button>
</div>
<div id="result" data-cy="result"></div>
<script>
  document.getElementById('clear').addEventListener('click', () => {
    document.getElementById('result').innerHTML = ''
  })
  ;['aa', 'bb', 'cc'].forEach((id) => {
    document.getElementById(id).addEventListener('click', async () => {
      const response = await window.fetch(`/${id}`)
      const { data } = await response.json()
      console.log('click', id, { data })

      // Fire off a bunch of extra requests to try to trigger the error
      for (let i = 0; i < 5 ; i++) {
        setTimeout(() => {
          window.fetch(`/${id}`)
        }, i * 20)
      }

      document.getElementById('result').innerHTML = data
    })
  })
</script>

spec.js

const url = 'http://localhost:5000'

context('Trying to repro', () => {
  before(() => {
    cy.visit(url)
  })
  ;['aa', 'bb'].forEach((data) => {
    it(`clicks button ${data}`, () => {
      // put the route2 call in each test
      cy.route2({ method: 'GET', pathname: /[a-z]+/ }, (req) => {
        req.reply({ data })
      })
      cy.get(`#clear`).click()
      cy.get(`[data-cy=buttons]`).contains(data).click()
      cy.get(`[data-cy=result]`).contains(data)
    })
  })
})

Cypress reports that both tests succeed, but it's not running any of the commands in the 2nd test.

cy.route2 in before

The request in the 2nd test isn't intercepted so the request 404s and the test fails.

index.html (The beforeEach version should still work, but this scenario repros without firing off any extra requests in between tests.)

<div data-cy="buttons">
  <button id="aa">button aa</button>
  <button id="bb">button bb</button>
  <button id="clear">clear</button>
</div>
<div id="result" data-cy="result"></div>
<script>
  // clear button just so tests don't accidentally pass when nothing happens
  document.getElementById('clear').addEventListener('click', () => {
    document.getElementById('result').innerHTML = ''
  })
  // wire up a fetch to click for each button
  ;['aa', 'bb'].forEach((id) => {
    document.getElementById(id).addEventListener('click', async () => {
      const response = await window.fetch(`/${id}`)
      const { data } = await response.json()
      document.getElementById('result').innerHTML = data
    })
  })
</script>

spec.js

const url = 'http://localhost:5000'

context('Trying to repro', () => {
  before(() => {
    cy.visit(url)
    cy.route2({ method: 'GET', pathname: /[a-z]+/ }, (req) => {
      console.log('INTERCEPTED', req.url)
      req.reply({ data: req.url })
    })
  })
  ;['aa', 'bb'].forEach((data) => {
    it(`clicks button ${data}`, () => {
      cy.get(`#clear`).click()
      cy.get(`[data-cy=buttons]`).contains(data).click()
      cy.get(`[data-cy=result]`).contains(`${url}/${data}`)
    })
  })
})

[jennifer-shehane]

• I've opened a new issue for the 2nd test not logging commands: https://github.com/cypress-io/cypress/issues/8940
• @flotwig Are cy.route2() definitions meant to be reusable when defined in a before? We have some other places where this doesn't work - it's defined once and not accessible after. So I'm not sure if this is expected or not.

[flotwig]

@jennifer-shehane route2 definitions are cleared between each test no matter what, if they're defined in a before they'll be removed on the start of the 2nd test

[sainthkh]

This issue contains 2 problems:

• cy.route2 doesn't work in before. => This is the intended feature as @flotwig said above.
• Cannot access 'continueSent' before initialization error. => Fixed in #8978.

As for the workaround, you can define a function like below:

context('Trying to repro', () => {
  before(() => {
    cy.visit(url)
  })

  const setupRoutes = () => {
    cy.route2({ method: 'GET', pathname: /[a-z]+/ }, (req) => {
      req.reply({ data: req.url })
    })
  }

  ;['aa', 'bb'].forEach((data) => {
    it(`clicks button ${data}`, () => {
      setupRoutes()

      cy.get(`#clear`).click()
      cy.get(`[data-cy=buttons]`).contains(data).click()
      cy.get(`[data-cy=result]`).contains(`${url}/${data}`)
    })
  })
})

[ChrisSargent]

@jennifer-shehane route2 definitions are cleared between each test no matter what, if they're defined in a before they'll be removed on the start of the 2nd test

Would be good to document this behaviour in the route2 page (I don't think it currently mentions it)

[rjdestigter]

I've been running into this as well. I think it is due to the client code failing. I'm using a non-async version of @xstate/test to generate model based tests and route2 to intercept APIS.

There are cases where a test is done. I expect that Cypress is already cancelling API interception but the browsers is still making a few additional requests. Because previous related requests have been intercepted payloads between the intercepted requests and the now non-intercepted requests cause client code to throw errors causing the route2 API to fail?

Order of execution would look something like:

• Cypress runs a test iteration: it ( ... ).
• Tests run okay, intercepted requests are mocked, all good.
• Cypress cleans up before next iteration. Request interception is cancelled.
• Visited page is still active and makes additional requests that normally would be intercepted
• Previous mocked response payloads and additional payloads don't match up and unexpected errors occur in the client code.
• For some reason this breaks route2?
• Additional tests don't work properly

I've been able to circumvent this by running:

cy.window().then(win => win.location.href = "about:blank")

for now effectivly killing the client before moving on the next iteration.

[cypress-bot[bot]]

The code for this is done in cypress-io/cypress#8978, but has yet to be released. We'll update this issue and reference the changelog when it's released.

[flotwig]

@rjdestigter it could be that there's some race condition where the Cypress proxy is still intercepting those routes after the Cypress driver has already cleared them. Additionally, there's not yet any special provision for clearing async callbacks created in cy.route2 handlers, so it's possible they could leak between tests. Feel free to open a new issue if you have a separate error from the OP here.

[cypress-bot[bot]]

Released in 5.6.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to Cypress v5.6.0, please open a new issue.