[releng] Failing tests do not fail

[eclipse-modisco-bot]

| --- | --- | | Bugzilla Link | 582717 | | Status | RESOLVED WONTFIX | | Importance | P3 normal | | Reported | Dec 07, 2023 16:44 EDT | | Modified | Dec 13, 2023 13:10 EDT | | See also | 582716, 582735 | | Reporter | Ed Willink |

Description

The Bug 582716 problem with ClassNotFoundException for org.apache.maven.surefire.junit4.JUnit4Provider gives an fatal failure when the Tycho build is run locally, but a success albeit with diminished test count on Jenkins.

Ensure that Jenkins fails properly.

[eclipse-modisco-bot]

By Ed Willink on Dec 07, 2023 16:50

Perhaps time to review. The MoDisco tests seems to run both maven and tycho surefire variants even though only tycho can work (for now). The copy across from OCL's pom.xml was probably not properly debugged.

[eclipse-modisco-bot]

By Ed Willink on Dec 08, 2023 01:27

The only obvious difference between Jenkins and local is -B on the maven command line. It is not clear why batch mode should have the side effect of suppressing failures. THe modisco-branch-tests has no -B, but has the same problem.

The log file for a particular test concludes

[ERROR] There are test failures.

rather than the usual

[INFO] Tests run: X, Failures: |Y, Errors: Z, Skipped: 0

The inadequate development perhaps enables this through the need to avoid the many

[INFO] No tests to run.

failing the build. But why the Jenkins/interactive discrepancy.

Perhaps time to review.

[eclipse-modisco-bot]

By Steve Brown on Dec 13, 2023 10:30

Hi Ed,

I don't get this behaviour after the changes I made for Bug 582716.

[INFO] --- tycho-surefire:4.0.4:test (AllUITests) @ org.eclipse.modisco.xml.discoverer.tests ---\ [INFO] Executing test runtime with timeout (seconds): 1800, logs, if any, will be placed at: C:\Tools\eclipse-committers-2023-12\git\org.eclipse.modisco\org.eclipse.modisco.xml.discoverer.tests\target\work\data.metadata.log\ [INFO] Command line:\ [C:\Tools\jdk-17\bin\java.exe, -Dosgi.noShutdown=false, -Dosgi.os=win32, -Dosgi.ws=win32, -Dosgi.arch=x86_64, -DoutputEncoding=UTF-8, -Dosgi.clean=true, -jar, C:\Work.m2\repository\p2\osgi\bundle\org.eclipse.equinox.launcher\1.6.600.v20231106-1826\org.eclipse.equinox.launcher-1.6.600.v20231106-1826.jar, -consolelog, -data, C:\Tools\eclipse-committers-2023-12\git\org.eclipse.modisco\org.eclipse.modisco.xml.discoverer.tests\target\work\data, -install, C:\Tools\eclipse-committers-2023-12\git\org.eclipse.modisco\org.eclipse.modisco.xml.discoverer.tests\target\work, -configuration, C:\Tools\eclipse-committers-2023-12\git\org.eclipse.modisco\org.eclipse.modisco.xml.discoverer.tests\target\work\configuration, -application, org.eclipse.tycho.surefire.osgibooter.uitest, -testproperties, C:\Tools\eclipse-committers-2023-12\git\org.eclipse.modisco\org.eclipse.modisco.xml.discoverer.tests\target\surefire.properties, -testApplication, org.eclipse.ui.ide.workbench, -timeout, 1800000]\ Launching application org.eclipse.ui.ide.workbench\ !SESSION 2023-12-13 14:46:09.327 -----------------------------------------------\ eclipse.buildId=unknown\ java.version=17.0.9\ java.vendor=Eclipse Adoptium\ BootLoader constants: OS=win32, ARCH=x86_64, WS=win32, NL=en_GB\ Framework arguments: -application org.eclipse.tycho.surefire.osgibooter.uitest -testproperties C:\Tools\eclipse-committers-2023-12\git\org.eclipse.modisco\org.eclipse.modisco.xml.discoverer.tests\target\surefire.properties -testApplication org.eclipse.ui.ide.workbench -timeout 1800000\ Command-line arguments: -consolelog -data C:\Tools\eclipse-committers-2023-12\git\org.eclipse.modisco\org.eclipse.modisco.xml.discoverer.tests\target\work\data -application org.eclipse.tycho.surefire.osgibooter.uitest -testproperties C:\Tools\eclipse-committers-2023-12\git\org.eclipse.modisco\org.eclipse.modisco.xml.discoverer.tests\target\surefire.properties -testApplication org.eclipse.ui.ide.workbench -timeout 1800000

!ENTRY org.eclipse.tycho.surefire.osgibooter 1 0 2023-12-13 14:46:19.726\ !MESSAGE starting DumpStackTracesTimer with timeout=1680000 at 2023-12-13 14:46:19 +0000\ Running org.eclipse.modisco.xml.discoverer.tests.AllTestsInUIThread\ Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.547 s -- in org.eclipse.modisco.xml.discoverer.tests.AllTestsInUIThread\ org.eclipse.modisco.xml.discoverer.tests.Test001.test001 -- Time elapsed: 0.412 s\ org.eclipse.modisco.xml.discoverer.tests.Test002.test001 -- Time elapsed: 0.087 s

Results:

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

[eclipse-modisco-bot]

By Steve Brown on Dec 13, 2023 10:32

After the changes I made for Bug 582716, I don't get the behaviour you describe:

[INFO] --- tycho-surefire:4.0.4:test (AllUITests) @ org.eclipse.modisco.xml.discoverer.tests ---\ [INFO] Executing test runtime with timeout (seconds): 1800, logs, if any, will be placed at: C:\Tools\eclipse-committers-2023-12\git\org.eclipse.modisco\org.eclipse.modisco.xml.discoverer.tests\target\work\data.metadata.log\ [INFO] Command line:\ [C:\Tools\jdk-17\bin\java.exe, -Dosgi.noShutdown=false, -Dosgi.os=win32, -Dosgi.ws=win32, -Dosgi.arch=x86_64, -DoutputEncoding=UTF-8, -Dosgi.clean=true, -jar, C:\Work.m2\repository\p2\osgi\bundle\org.eclipse.equinox.launcher\1.6.600.v20231106-1826\org.eclipse.equinox.launcher-1.6.600.v20231106-1826.jar, -consolelog, -data, C:\Tools\eclipse-committers-2023-12\git\org.eclipse.modisco\org.eclipse.modisco.xml.discoverer.tests\target\work\data, -install, C:\Tools\eclipse-committers-2023-12\git\org.eclipse.modisco\org.eclipse.modisco.xml.discoverer.tests\target\work, -configuration, C:\Tools\eclipse-committers-2023-12\git\org.eclipse.modisco\org.eclipse.modisco.xml.discoverer.tests\target\work\configuration, -application, org.eclipse.tycho.surefire.osgibooter.uitest, -testproperties, C:\Tools\eclipse-committers-2023-12\git\org.eclipse.modisco\org.eclipse.modisco.xml.discoverer.tests\target\surefire.properties, -testApplication, org.eclipse.ui.ide.workbench, -timeout, 1800000]\ Launching application org.eclipse.ui.ide.workbench\ !SESSION 2023-12-13 14:46:09.327 -----------------------------------------------\ eclipse.buildId=unknown\ java.version=17.0.9\ java.vendor=Eclipse Adoptium\ BootLoader constants: OS=win32, ARCH=x86_64, WS=win32, NL=en_GB\ Framework arguments: -application org.eclipse.tycho.surefire.osgibooter.uitest -testproperties C:\Tools\eclipse-committers-2023-12\git\org.eclipse.modisco\org.eclipse.modisco.xml.discoverer.tests\target\surefire.properties -testApplication org.eclipse.ui.ide.workbench -timeout 1800000\ Command-line arguments: -consolelog -data C:\Tools\eclipse-committers-2023-12\git\org.eclipse.modisco\org.eclipse.modisco.xml.discoverer.tests\target\work\data -application org.eclipse.tycho.surefire.osgibooter.uitest -testproperties C:\Tools\eclipse-committers-2023-12\git\org.eclipse.modisco\org.eclipse.modisco.xml.discoverer.tests\target\surefire.properties -testApplication org.eclipse.ui.ide.workbench -timeout 1800000

!ENTRY org.eclipse.tycho.surefire.osgibooter 1 0 2023-12-13 14:46:19.726\ !MESSAGE starting DumpStackTracesTimer with timeout=1680000 at 2023-12-13 14:46:19 +0000\ Running org.eclipse.modisco.xml.discoverer.tests.AllTestsInUIThread\ Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.547 s -- in org.eclipse.modisco.xml.discoverer.tests.AllTestsInUIThread\ org.eclipse.modisco.xml.discoverer.tests.Test001.test001 -- Time elapsed: 0.412 s\ org.eclipse.modisco.xml.discoverer.tests.Test002.test001 -- Time elapsed: 0.087 s

Results:

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

[eclipse-modisco-bot]

By Ed Willink on Dec 13, 2023 13:10

(In reply to Steve Brown from comment #3)

I don't get this behaviour after the changes I made for Bug 582716.

No. The Bug 582716 fixes avoid the bad dependency problems with mismatching Tycho/platform versions.

This bug is that the dependency problem crashes the test in ways that report just 29 rather than 175 test successes.

---

Maybe this bug is just too hard. WONTFIX

Workaround:

• manually verify that the MoDisco master build has a credible number of passes
• manually run the MoDisco nightly build locally occasionally.

Perhaps time to review. The MoDisco tests seems to run both maven and tycho surefire variants even though only tycho can work (for now).

Not quite bug Bug 582735 raised.