search

garris / BackstopJS

Catch CSS curve balls.
http://backstopjs.org
MIT License
6.69k stars 604 forks source link

The word "fail" in the label causes a scenario to be marked failed #1407

Closed fastfedora closed 2 years ago

fastfedora commented 2 years ago

I had two scenarios whose labels included the text "fail" (specifically the words "Failure" and "Failed"). These were flagged as failing tests even though they had diffs of 0%. When I renamed so the text "fail" was not in the label, they passed.

The workaround is to simply rename your scenarios, but since I didn't see it documented anywhere, I'm opening this ticket in case others run into the same problem.

garris commented 2 years ago

Wow Really? I just searched the codebase and didn't see anything that could behave that way. But I have seen weirder things. Thanks for reporting!

fastfedora commented 2 years ago

I re-ran the tests several more times under different conditions and think I mis-read what was happening. I just installed this today, so didn't understand enough about the UI to debug it properly.

When I originally ran it, I got 6 failures (2 scenarios vs 3 viewports). I then typed "fail" into the search box rather than clicking the "6 failed" button (I didn't realize it was a button). This showed me the reports with "fail" in the label. I then changed the labels and re-ran and everything passed.

However, after your reply, I went back and ran each of those scenarios using the filter option, with and without "fail" in the name and they all passed. Then I went and ran my entire test suite again and got 6 failures.

It looks like I have some timing issues which is causing 2 of my scenarios to fail sometimes. It just so happened that the number of scenarios matched the number of scenarios with the test "fail" and when I re-ran, they all passed. Pure coincidence and lack of understanding on my part of how to properly filter for failed tests.

I'm going to try to resolve the timing issues with the delay property, and in the meantime, I'm closing this issue, since it appears to be a false alarm. Thanks for the speedy response though!