search

pitest / pitclipse

Mutation testing for Java in Eclipse IDE. Based on PIT (Pitest).
https://pitest.org
Apache License 2.0
59 stars 17 forks source link

UI tests randomly fail #81

Closed echebbi closed 3 years ago

echebbi commented 5 years ago

When executed, some UI tests can fail unexpectedly. The following failures have been observed:

I believe that SWTBot is not always used properly, e.g. tests do not always wait for shells to close before moving on.

echebbi commented 5 years ago

I believe that high priority should be given to this issue, so I'm going to investigate it.

echebbi commented 5 years ago

It seems that I managed to fix the random exceptions, but the build still hangs from time to time. Logs don't show anything useful so I don't know why yet; I will try to reproduce the problem locally.

echebbi commented 5 years ago

After a few tests it seems that the test hangs because the SummaryView is never updated with Pitest's results:

I'll try to figure out why.

echebbi commented 5 years ago

Reporting progress so far:

  1. Managed to display PIT logs during Maven build.
  2. Fixed a random test failure due to a temporal issue between a text editor opened in background and a test responsible of asserting that the editor was properly opened: the test was sometimes run before the editor actually opened.
  3. Found out the reason why tests sometimes hang until Travis CI kill the job: an Errors in Workspace dialog sometimes opens unexpectly to warn about compilation issues. I don't know yet what causes it though.
  4. Discovered another random test failure. Test asserting that the number of mutations displayed in the PIT Mutations view is correct may fail because for a still undetermined reason PIT sometimes fail to find mutations even though it should: 
    7:20:14 AM PIT >> WARNING : No mutations found. This probably means there is an issue with either the supplied classpath or filters.

    That being said such errors occur like 1 time out of ten and each build takes around 30 min. I already spent way too much time on this so I'll take until the end of the week to attempt to fix the hanging issue which happens often enough to be annoying then I'll move on more important topics.

LorenzoBettini commented 3 years ago

Following https://github.com/pitest/pitclipse/pull/125#issuecomment-830055573 here are my findings:

Having said that, trying to understand what's going on and debugging is a real nightmare due to Cucumber, that makes it really hand to understand what's going wrong and in particular, it's impossible to run from Eclipse single scenarios (at least, I did not find a way). Please keep in mind that switching to a new version of the target platform (I've tried that locally) breaks SWTBot tests even at the very beginning, and, again, due to Cucumber, I don't understand what's going on.

I'd then suggest eliminating Cucumber and writing standard SWTBot tests by reusing most "steps" already implemented. I love Cucumber, but in this context, it makes UI tests impossible to maintain, especially when switching to a new version of Eclipse TP, which, I guess, should be done very soon.

If there are no objections, I can start working on that.

By the way, many tested scenarios could be eliminated completely.

echebbi commented 3 years ago

I fully agree. Cucumber here makes testing and debugging specific scenarios a bit clunky.

Just for the record I remember having faced issues with tests on Windows as well, even if it was unusual.