search

karma-runner / karma-jasmine

A Karma plugin - adapter for Jasmine testing framework.
MIT License
542 stars 162 forks source link

Tests are run multiple times #260

Closed provegard closed 4 years ago

provegard commented 4 years ago

Occasionally one of my test suits will run tests multiple times. It may be that certain tests are run 2 times, or 3 times, or perhaps even more.

Here's what a test run may look like:

HeadlessChrome 79.0.3945 (Windows 10.0.0): Executed 6855 of 6641 (skipped 44) SUCCESS (0 secs / 1 min 17.505 secs)
HeadlessChrome 79.0.3945 (Windows 10.0.0): Executed 12588 of 6641 (skipped 78) SUCCESS (0 secs / 2 mins 26.782 secs)

In this case, I aborted the test run before it ran to completion, so I don't know what it would have reached.

I've tested with version 3.1.0, and the output above is from that version. The subsequent run was successful. Previously I was on 2.0.1, also with intermittent "test overruns".

I'ved tried reducing the suite size - the suite before was well over 10000 tests.

I've tried setting concurrency to 1, without success.

I've looked for places where my code reloads the page, because I saw that mentioned in some other issue. But in all places where that happens, the tests mock the reload action.

Do you have any suggestions on how to track down the issue? Since the overrun doesn't happen all the time, I would like to have some sort of trace log to look at. The standard Karma debug log is not helpful.

Thank you in advance!

johnjbarton commented 4 years ago

I've not seen anything similar. Maybe you can 'bisect', splitting tests in half to see if the issue is in one or the other, recursively.

provegard commented 4 years ago

Yes, that's a good idea!

However, I finally figured it out without resorting to bisecting - it was a reload after all, caused by an unfortunate/accidental dependency between two separate test suites.

Now I need to figure out why I didn't see the Some of your tests did a full page reload message. But that's a Karma thing, so I'm closing this issue.