OIDC Login in e2e Test works, but only the first time after npx cypress open, not unless we close all cypress windows

[dlatikaynen]

Current behavior

npx cypress run --> test fails instantly, from the video it can be seen that there is a timeout after 70 seconds with the message

(page load) --waiting for new page to load--

npx cypress open --> test always works fast and perfectly well on the initial run in a pristine cypress window. but as soon as we repeat the test run, it fails. It also fails when we only close the cypress browser window. Only if we also close the cypress main window (which ends the cypress process), then it works again once on the next open.

Desired behavior

The test should always succeed, without the need to terminate the cypress process inbetween runs. The cypress run approach should always work.

Test code to reproduce

cypress-issue-MCVE.zip cypress_debug_output.zip

Full MCVE in attached ZIP file. Relevant test code:

describe('lets log into caats', () => {
  it('(1) open the login page and login', () => {
    cy.clearAllSessionStorage();
    cy.clearAllLocalStorage();
    cy.clearAllCookies();
    cy.visit('https://loidl-consulting-d.caats.io');
    cy.get('#Input_Username').type('DevOp@loidl-consulting.at');
    cy.get('#Input_Password').type('********');
    cy.get('.btn').click();
    cy.url().should('contain', 'https://loidl-consulting-d.caats.io');
    cy.get('.mat-expansion-panel-header-description').contains('Short name');
  })
})

Cypress Version

12.15.0

Node version

18.14.2

Operating System

Windows 10 21H2 19044-3086

Debug Logs

C:\lc\devops\testing>set DEBUG=cypress:*
C:\lc\devops\testing>cypress run
  cypress:cli:cli cli starts with arguments ["C:\\Program Files\\nodejs\\node.exe","C:\\lc\\devops\\testing\\node_modules\\cypress\\bin\\cypress","run"] +0ms
  cypress:cli NODE_OPTIONS is not set +0ms
  cypress:cli:cli program parsing arguments +2ms
  cypress:cli:cli running Cypress with args [ Command { _events: [Object: null prototype] { 'option:auto-cancel-after-failures': [Function (anonymous)], 'option:browser': [Function (anonymous)], 'option:ci-build-id': [Function (anonymous)], 'option:component': [Funct
ion (anonymous)], 'option:config': [Function (anonymous)], 'option:config-file': [Function (anonymous)], 'option:e2e': [Function (anonymous)], 'option:env': [Function (anonymous)], 'option:group': [Function (anonymous)], 'option:key': [Function (anonymous)], 'option:
headed': [Function (anonymous)], 'option:headless': [Function (anonymous)], 'option:no-exit': [Function (anonymous)], 'option:parallel': [Function (anonymous)], 'option:port': [Function (anonymous)], 'option:project': [Function (anonymous)], 'option:quiet': [Function
 (anonymous)], 'option:record': [Function (anonymous)], 'option:reporter': [Function (anonymous)], 'option:reporter-options': [Function (anonymous)], 'option:spec': [Function (anonymous)], 'option:tag': [Function (anonymous)], 'option:dev': [Function (anonymous)] }, 
_eventsCount: 23, _maxListeners: undefined, commands: [], options: [ [Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option]
, [Option], [Option], [Option] ], parent: Command { _events: [Object: null prototype], _eventsCount: 1, _maxListeners: undefined, commands: [Array], options: [Array], parent: null, _allowUnknownOption: false, _args: [], rawArgs: [Array], _scriptPath: 'C:\\lc\\devops\
\testing\\node_modules\\cypress\\bin\\cypress', _name: 'cypress', _optionValues: {}, _storeOptionsAsProperties: true, _storeOptionsAsPropertiesCalled: true, _passCommandToAction: true, _actionResults: [], _actionHandler: null, _executableHandler: false, _executableFi
le: null, _defaultCommandName: null, _exitCallback: null, _aliases: [], _combineFlagAndOptionalValue: true, _hidden: false, _hasHelpOption: true, _helpFlags: '-h, --help', _helpDescription: 'display help for command', _helpShortFlag: '-h', _helpLongFlag: '--help', _h
asImplicitHelpCommand: undefined, _helpCommandName: 'help', _helpCommandnameAndArgs: 'help [command]', _helpCommandDescription: 'display help for command', _usage: '<command> [options]', args: [Array], [Symbol(kCapture)]: false }, _allowUnknownOption: false, _args: [
], rawArgs: null, _scriptPath: null, _name: 'run', _optionValues: {}, _storeOptionsAsProperties: true, _storeOptionsAsPropertiesCalled: false, _passCommandToAction: true, _actionResults: [], _actionHandler: [Function: listener], _executableHandler: false, _executable
File: null, _defaultCommandName: null, _exitCallback: null, _aliases: [], _combineFlagAndOptionalValue: true, _hidden: false, _hasHelpOption: true, _helpFlags: '-h, --help', _helpDescription: 'display help for command', _helpShortFlag: '-h', _helpLongFlag: '--help', 
_hasImplicitHelpCommand: 0, _helpCommandName: 'help', _helpCommandnameAndArgs: 'help [command]', _helpCommandDescription: 'display help for command', _usage: '[options]', _description: 'Runs Cypress tests from the CLI without the GUI', _argsDescription: undefined, ex
it: true, args: [], [Symbol(kCapture)]: false } ] +1ms
  cypress:cli:cli variable-length opts parsed { args: [ 'C:\\Program Files\\nodejs\\node.exe', 'C:\\lc\\devops\\testing\\node_modules\\cypress\\bin\\cypress', 'run' ], opts: Command { _events: [Object: null prototype] { 'option:auto-cancel-after-failures': [Function 
(anonymous)], 'option:browser': [Function (anonymous)], 'option:ci-build-id': [Function (anonymous)], 'option:component': [Function (anonymous)], 'option:config': [Function (anonymous)], 'option:config-file': [Function (anonymous)], 'option:e2e': [Function (anonymous
)], 'option:env': [Function (anonymous)], 'option:group': [Function (anonymous)], 'option:key': [Function (anonymous)], 'option:headed': [Function (anonymous)], 'option:headless': [Function (anonymous)], 'option:no-exit': [Function (anonymous)], 'option:parallel': [F
unction (anonymous)], 'option:port': [Function (anonymous)], 'option:project': [Function (anonymous)], 'option:quiet': [Function (anonymous)], 'option:record': [Function (anonymous)], 'option:reporter': [Function (anonymous)], 'option:reporter-options': [Function (an
onymous)], 'option:spec': [Function (anonymous)], 'option:tag': [Function (anonymous)], 'option:dev': [Function (anonymous)] }, _eventsCount: 23, _maxListeners: undefined, commands: [], options: [ [Option], [Option], [Option], [Option], [Option], [Option], [Option], 
[Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option], [Option] ], parent: Command { _events: [Object: null prototype], _eventsCount: 1, _maxListeners: undefined, commands:
 [Array], options: [Array], parent: null, _allowUnknownOption: false, _args: [], rawArgs: [Array], _scriptPath: 'C:\\lc\\devops\\testing\\node_modules\\cypress\\bin\\cypress', _name: 'cypress', _optionValues: {}, _storeOptionsAsProperties: true, _storeOptionsAsProper
tiesCalled: true, _passCommandToAction: true, _actionResults: [], _actionHandler: null, _executableHandler: false, _executableFile: null, _defaultCommandName: null, _exitCallback: null, _aliases: [], _combineFlagAndOptionalValue: true, _hidden: false, _hasHelpOption:
 true, _helpFlags: '-h, --help', _helpDescription: 'display help for command', _helpShortFlag: '-h', _helpLongFlag: '--help', _hasImplicitHelpCommand: undefined, _helpCommandName: 'help', _helpCommandnameAndArgs: 'help [command]', _helpCommandDescription: 'display he
lp for command', _usage: '<command> [options]', args: [Array], [Symbol(kCapture)]: false }, _allowUnknownOption: false, _args: [], rawArgs: null, _scriptPath: null, _name: 'run', _optionValues: {}, _storeOptionsAsProperties: true, _storeOptionsAsPropertiesCalled: fal
se, _passCommandToAction: true, _actionResults: [], _actionHandler: [Function: listener], _executableHandler: false, _executableFile: null, _defaultCommandName: null, _exitCallback: null, _aliases: [], _combineFlagAndOptionalValue: true, _hidden: false, _hasHelpOptio
n: true, _helpFlags: '-h, --help', _helpDescription: 'display help for command', _helpShortFlag: '-h', _helpLongFlag: '--help', _hasImplicitHelpCommand: 0, _helpCommandName: 'help', _helpCommandnameAndArgs: 'help [command]', _helpCommandDescription: 'display help for
 command', _usage: '[options]', _description: 'Runs Cypress tests from the CLI without the GUI', _argsDescription: undefined, exit: true, args: [], [Symbol(kCapture)]: false } } +167ms
  cypress:cli parsed cli options {} +4s
  cypress:cli verifying Cypress app +0ms
  cypress:cli checking environment variables +1ms
  cypress:cli checking if executable exists C:\Users\dlatikay\AppData\Local\Cypress\Cache\12.15.0\Cypress\Cypress.exe +3ms
  cypress:cli Binary is executable? : true +3ms
  cypress:cli binaryDir is  C:\Users\dlatikay\AppData\Local\Cypress\Cache\12.15.0\Cypress +0ms
  cypress:cli Reading binary package.json from: C:\Users\dlatikay\AppData\Local\Cypress\Cache\12.15.0\Cypress\resources\app\package.json +0ms
  cypress:cli Found binary version 12.15.0 installed in: C:\Users\dlatikay\AppData\Local\Cypress\Cache\12.15.0\Cypress +6ms
  cypress:cli { verified: true } +7ms
  cypress:cli is Verified ? true +3ms
  cypress:cli:run processing run options { key: null, spec: null, reporter: null, reporterOptions: null, project: 'C:\\lc\\devops\\testing' } +0ms
  cypress:cli:run --key is not set, looking up environment variable CYPRESS_RECORD_KEY +1ms
  cypress:cli:run run to spawn.start args ["--run-project","C:\\lc\\devops\\testing"] +0ms
  cypress:cli needs to start own Xvfb? false +0ms
  cypress:cli spawning, should retry on display problem? false +0ms
  cypress:cli spawn args [ '--', '--run-project', 'C:\\lc\\devops\\testing', '--cwd', 'C:\\lc\\devops\\testing', '--userNodePath', 'C:\\Program Files\\nodejs\\node.exe', '--userNodeVersion', '18.14.2' ] { detached: false, stdio: 'pipe', windowsHide: false } +6ms     
  cypress:cli spawning Cypress with executable: C:\Users\dlatikay\AppData\Local\Cypress\Cache\12.15.0\Cypress\Cypress.exe +1ms
  cypress:cli piping process STDIN into child STDIN +11ms
  cypress:cli piping child STDOUT to process STDOUT +1ms
  cypress:cli piping child STDERR to process STDERR +915ms

Other

We would look into other ways to get the token, but in our use case it absolutely makes sense to include the actual authentication flow in the e2e testing because the identity server is part of the test candidate project. We would not regard this as a bug, had it not worked on initial cypress start. The fact that it does so, points to some kind of state being persisted and not cleared after the initial run. Cookies and localstore were inconspicious, additionally calling clearallcookies, clearalllocalstorage and clearallsessionstorage changed nothing. Also, tests with other browsers than electron remained inconclusive. we did not pursue this further; we had the impression that on chrome, it never worked properly but then there were certificate errors and other stuff that electron did not have. so the MCVE included with this report assumes electron only.

github tells me "body is too long". so we included the full debug output as another attachment, the text in "Debug Logs" is truncated.

[warrensplayer]

I was able to reproduce the error. There seems to be some flake on when the final page that is being navigated to after submitting the login form is loaded. I could get the test to pass when Cypress first loads in open mode, then it would fail if rerun. However, if I just navigated to another page (like Settings) and back to run the spec, then it would pass again ever so often. Sometimes it seemed like every other time.

The load window event is not being fired in the cases it fails so the logic navigation.ts never fires to cancel the timeout that is waiting for the page to load here: https://github.com/cypress-io/cypress/blob/b3fd048fbc1b920251e145b050494909449005c9/packages/driver/src/cy/commands/navigation.ts#L354

I will forward this ticket to the appropriate team. They will soon evaluate the priority of this ticket and consider their capacity to pick it up. Please note that this does not guarantee that this issue will be resolved. The ticket will indicate status changes during evaluation, so we ask that you please refrain from asking for updates. Thanks!

[fischermatte]

Is there any workaround for this? We experiencing maybe something similar with our oidc flow. In our case it the first describe section works with cypres run. All following are running into

(page load) --waiting for new page to load--

This happens mostly only on the pipeline, not when running the test from the local machine.