search

cypress-io / cypress

Fast, easy and reliable testing for anything that runs in a browser.
https://cypress.io
MIT License
46.71k stars 3.16k forks source link

Issue With Skipping Tests Resurfaced in 13.8.1 #29461

Open wakeboardfit opened 4 months ago

wakeboardfit commented 4 months ago

Current behavior

We run in an Azure pipeline and after upgrading to 13.8.1, we are experiencing that many tests are just being skipped. This did not happen before and the code did not change outside of a few locator updates and whatnot. image

When running in 13.6.4, our executions would look like image

You can see the drop off where we upgraded versions image

I saw there was an issue that was closed regarding something similar back in version 10.

Desired behavior

Cypress should not just skip tests for not reason.

Test code to reproduce

Everything runs locally so this seems only on Azure. We are running "windows-latest" agent.

Cypress Version

13.8.1

Node version

18.18.2

Operating System

Windows-latest Azure Pipeline

Debug Logs

I will have to get this setup in our pipeline and will add when I can.

Other

No response

jennifer-shehane commented 4 months ago

@wakeboardfit Are you recording to the Cloud? Could you link to a URL of the run if so?

Could you provide the command you use to run Cypress?

I'm not familiar with anything that would cause this. This is very strange. Can you link to the old issue?

wakeboardfit commented 4 months ago

Old issue: https://github.com/cypress-io/cypress/issues/22276

Command: 

npx cypress run --spec cypress/e2e/*/**/*.feature --config video=true,screenshotOnRunFailure=true --env username=$(USERNAME),insurerUser=$(INSUREDNAME),password=$env:MAPPED_PASS,ls_api_username=$(USERNAME),ls_api_password=$env:MAPPED_PASS,tags=@admitted

We are not using the cypress cloud. I am still trying to figure out how to set the Debug flag in Azure pipeline as it says you need to run in command line (not powershell) and I need to see if I can make that run as admin.

wakeboardfit commented 4 months ago

Unless you guys have another way, I am not seeing a way to run Command Line as and Admin in Azure without self hosted agents. I put the debug command in our Power Shell step but I don't believe it did anything and your documentation pretty specifically says not Power Shell.

jennifer-shehane commented 4 months ago

@wakeboardfit I do see you are passing a tags flag. What plugin are you using to read in tags and run/not run tests based on that? Is that was would be causing some tests to be skipped? If they don't match the tag?

wakeboardfit commented 4 months ago

I need to keep digging in to this. The Pending correctly indicates all the tests that are != tag. It gets weird though with the skipping and potentially it's a error that is happening silently. I have yet to be able to trigger it on a local run but also need have more time to experiment and try to see what's going on. I am reverting our pipeline to 12.17.4 for now and will experiment with 13.8.X to see why that is failing and skipping when it didn't in the past.

wakeboardfit commented 3 months ago

Finally getting a chance to get back to this. I noticed this when examining the latest Skipped test. image image While executing, the "Mocha Tests" will just fire and then all tests after that are Skipped. I'm theorizing it has something to do with a failure although if it were consistent then we would never get executions on a set of tests with 1 or more failures. I am currently attempting to remove any code from the After in the tests just to see if that is potentially causing the tests to consider themselves "Done" in the middle of a run.

codeustad commented 2 months ago

@jennifer-shehane and @wakeboardfit we are experiencing the same issue but only in CI. Howevery, after many hours of investigating, I could form a hypothesis: I believe it is because the processess or threads in CI, due to perhaps low resources like CPU and Memory, sometimes can get paused and continued afterwards. It might be that Cypress internally then runs into a timeout and stops the runner. But I am not sure about that.

@jennifer-shehane, can you pin-point in the Cypress source code if and when Cypress decides to e.g. fire Cypress.runner.stop()?

Thanks in advance!

codeustad commented 2 months ago

Finally getting a chance to get back to this. I noticed this when examining the latest Skipped test. -- IMAGE -- While executing, the "Mocha Tests" will just fire and then all tests after that are Skipped. I'm theorizing it has something to do with a failure although if it were consistent then we would never get executions on a set of tests with 1 or more failures. I am currently attempting to remove any code from the After in the tests just to see if that is potentially causing the tests to consider themselves "Done" in the middle of a run.

@wakeboardfit in your tests above, are you running any cy.visit(...) inside beforeEach, before or after that could have been run into a time out due to loading HTML?

wakeboardfit commented 2 months ago

@codeustad So sorry for such a long time before replying, had a vacation thrown in there :D We do not have cy.visit in beforeEach however it is in a Background step so it does satisfy your question of if it runs each test. I'm not sure if that would fire of the stop test execution we are experiencing or not. My removal of code that might be throwing an exception was not fruitful. We are still experiencing this issue but again I've been super busy and can't dig into this right now.

jennifer-shehane commented 3 weeks ago

We'll need a way to reproduce this in order to look into this.