search

javaee-samples / javaee8-samples

Java EE 8 Samples
Other
376 stars 238 forks source link

Ignore failing tests #39

Closed pzygielo closed 5 years ago

pzygielo commented 5 years ago

I propose to convert failing travis/default build of master, to new issue and @Ignore failing tests. Tests were broken in #34, with explanation provided there. IMO new issue that would remind to remove this exclusion once [not sure about condition] is met, would be better than failing build. (Or perhaps warning for skipped tests is better than new issue, I do not know.)

Although #34 declares

changes that now make tests failing in Payara

I checked that for -P glassfish-embedded,!payara-ci-managed (i.e. for glassfish) the same happens.

arjantijms commented 5 years ago

@pzygielo In this specific project, tests are actually allowed to fail, as they indicate the project which they test against is faulty and the vendor should fix the issue. The test is not broken, but for that test Payara is.

It agree that is does make it difficult to see if new tests fail or not. We've all been trained (me too) to strive for passing builds, but this project is thus a little different. @Ignore is unfortunately not correct here.

pzygielo commented 5 years ago

I think it would be better to have such test in vendor(s) suite, and not in samples. From samples I'd expect to pass rather than show faults in used components (regardless of ci-job disclaimer).

With failing test, one always has to check the reason for failure. And having two failing just on start (that are not announced to be known to fail by default in any way in code), although somehow educational, isn't quite nice experience to begin with.

@Ignore is not marker of broken test (Probably I should have written Build was broken instead of Tests were broken).