Reactivate tests: disabled tests collection

HeikoKlare commented 1 year ago

Current Situation

Several tests have been deactivated over the years for different reasons, such as temporary incompatibilities with a specific platform/OS, bugs in the code, indeterministic behavior of the test etc.

These disablements are done by adding some prefix to test... methods in JUnit 3, by removing or commenting out @Test in JUnit 4 and 5, or by using @Ignore or @Disabled. In addition, several tests are only deactivated for specific platforms using early returns or assume statements within the test methods.

Goal

Since in most cases there is no additional, unique documentation or tagging of these disablements, they are not easy to find. This issue is supposed to be an umbrella to document disabled tests (existing ones and potentially further ones).

The purpose of this issue is to collect the disabled tests to have a central overview and provide the chance to reactivate and, if necessary, fix them. It also serves as a "good first issue" (see https://github.com/eclipse-platform/eclipse.platform.text/pull/212#issuecomment-1594576748) for interested new contributors, which can take any of the collected tests and try to fix them to learn about the Eclipse platform code while, at the same time, already having the chance to make valuable contributions to the project.

Process

I see the following ideas as good starting points for identifying disabled tests:

First concentrate on a single project, such as org.eclipse.core.tests.resources
Use different techniques to find disabled tests:
1. Completely disabled tests:
  - Search for missing or commented out @Test annotation (JUnit 4 and 5)
  - Search for tests not starting with test (JUnit 3), in particular those starting with _test
2. Tests disabled for specific platforms:
  - Search for accesses to OS constants in Platform, such as OS_WIN32 and validate if they are used to disable tests for specific platforms
  - Search for accesses to methods isLinux() etc. in ResourceTest subclasses and validate if they are used to disable tests for specific platforms
  - Search for assumeFalse or assumeTrue statements that make tests only run on specific platforms

Examples

The tests modified in https://github.com/eclipse-platform/eclipse.platform.text/pull/212
Tests in org.eclipse.core.tests.resources.regression.Bug_032076
The skipTest helper in org.eclipse.core.tests.resources.usecase.SnapshotTest

Disabled in Maven

Some tests are disabled in Maven builds, such that integration builds do not reveal whether they fail or not.

[ ] The bundle org.eclipse.core.tests.net is completely disabled. Tests run fine withing Eclipse on Windows, but Tycho execution fails, probably because of missing platform-dependent fragments (maybe have a look at how SWT deals with that).
[ ] org.eclipse.team.tests.core.AllTeamUITests and org.eclipse.team.tests.ui.synchronize.AllTeamSynchronizeTests are not executed as the test suite only executes org.eclipse.team.tests.core.AllTeamTests. Tests run fine within Eclipse on WIndows, but Tycho fails to execute some of the ScopeTests and the AllTeamSynchronizeTest.

Wittmaxi commented 1 year ago

Performing a naive search for _test, @Ignore, @Disabled, I found quite some artifacts which are worth adding to this issue

org.eclipse.core.tests.resources

[ ] SymLinkTest#testSymlinkExtendedChars(), (issue) - Enabling this test works in my native environment, I will test it in an headless-environment next. Notably, the very recently introduced assumeCanCreateSymLinks()(commit) requires administrator rights on windows to enable this test.
[ ] Bug_032076#testFileBugOnLinux(), issue
[ ] Bug_032076#testFolderBugOnLinux(), issue
[ ] Bug_032076#testProjectBugOnLinux(), issue
[ ] CharsetTest#testBug67606, issue
[ ] ~~LinkedResourceSyncMoveAndCopyTest#testMoveFolderWithLinksToNonExisitngLocations_withShallow(), issue~~ disabled but works fine, fixed via #603
[ ] LinkedResourceSyncMoveAndCopyTest#testCopyFolderWithLinksToNonExisitngLocations_withShallow(), issue
[x] ~~MarkerTest#testPerformanceManyResources~~ - removed via #674
[x] ~~MarkerTest#testPerformanceOneResource~~ - removed via #674
[ ] ResourceAttributeTest#testRefreshExxecutableOnFolder(), issue

org.eclipse.core.tests.runtime

[ ] ServiceSupplierTestCase#testUselessUpdates(), issue - this is a relatively new issue
[ ] EclipsePreferencesTest#testRemoveDeletesFile(), issue - Status CLOSED WONTFIX
[ ] BeginEndRuleTest#testEndNullRule(), issue - RESOLVED WONTFIX
[ ] DeadlockDetectionTest#testRuleHierarchyLockInteraction(), no issue
[ ] DeadlockDetectionTest#testComplexRuleLockInteraction(), no issue
[ ] DeadlockDetectionTest#testJobRuleCancellation, no issue
[ ] DeadlockDetectionTest#testLockMultipleAcquireThenSuspend, no issue
[ ] JobTest#testDone(), issue - RESOLVED WONTFIX - For this test, the documented behaviour of Job.done() needs to be understood and checked against the test. If the test does not check for documented behaviour, the test itself needs to be rewritten.

Michael5601 commented 1 year ago

Addition to https://github.com/eclipse-platform/eclipse.platform/issues/525#issuecomment-1613014550. Checked tests have pending or accepted PR:

org.eclipse.core.tests.resources

[ ] RefreshJobTest#XtestSmallRecursionRefresh() - test is disabled because it needs 400 seconds on fast SSD on Linux
[ ] RefreshJobTest#XtestBigRecursionDeepRefresh() - test is disabled because it needs 250 seconds on fast SSD on Linux
[ ] RefreshJobTest#XtestUnrelatedRule() - test is disabled because it is unstable
[ ] Bug_079398#testBug79398(), issue - test is not hooked up because bug is not fixed yet
[ ] IFileTest#testBug25658(), issue - bug seems to be fixed but test is still disabled. Can only be tested on Linux.
[ ] VirtualFolderTest#testIsLocal() - test is commented out
[ ] VirtualFolderTest#testIsSynchronizedforVirtualFolder() - test is commented out
[ ] LinkedResourceWithPathVariableTest#testGetPathVariable() - test is commented out
[ ] ISynchronizerTest#testSnap() - test is commented out
[ ] IWorkspaceTest#testValidateLinkLocation() - test is commented out
[ ] RefreshLocalTest#testDiscoverLinkedResource() - test is commented out
[ ] Bug552185PerformanceTest#testBug552185Performance(), issue - not included in actual tests, since performance is only printed and not asserted.
[x] ~~Bug_029851#test(), issue~~ disabled but works fine, fixed via #603

org.eclipse.debug.tests

[x] ~~SerialExecutorTest#testHeavyScheduling(), issue~~ - test is disabled but working because bug is already fixed, enabled via #617
[x] ~~ConsoleDocumentAdapterTests#_testInvalidInvocations()~~ - test is disabled but working, enabled via #617
[ ] RuntimeProcessTests#testOutputAfterDestroy(), issue - NEW BUG
[x] ~~CheckTests#testCheckReceiver()~~ - test is commented out, reenabled with #681.
[ ] ContentTests#_x_testChildrenUpdatesCompletedOutOfSequence(), issue - test works locally if reactivated but fails in CI builds, so it should remain disabled
[ ] DeltaTests#_x_testAddUnexpandedElement(), issue - RESOLVED WORKSFORME is bug status, but the test fails on windows
[ ] DeltaTests#_x_testRefreshUnexpandedElementsChildren() - test fails
[ ] JFaceViewerDeltaTests#testBug292322(), issue - comment states that test should be deleted after bug is fixed. But after reactivation the test does not terminate. It is not executed during CI builds (they run the VirtualViewerContentTests rather than the JFaceViewerDeltaTests), so that CI builds do not reveal that failure. Test should remain disabled.
[ ] VirtualViewerLazyModeTests#test() - test is not yet implemented
[x] ~~FilterTest#testVerify()~~ - test is commented out, enabled via #679

org.eclipse.core.tests.net.NetTest

[ ] NetTest#_testSimpleHost(), issue - enabled in #618 but failed in CI builds, thus disabled again via #633

org.eclipse.team.tests.core

[x] ~~ScopeTests#testScopeContraction()~~ - disabled but working, enabled via #619
[ ] ResourceContentTests#testFileChange() - test is commented out

org.eclipse.tips.tests

[x] TipImageBas64Test#testSetExtension() - test is commented out
[x] TipImageBas64Test#testSetExtension2() - test is commented out
[x] TipImageURLTest#testSetExtension() - test is commented out
[x] TipImageURLTest#testSetExtension2() - test is commented out
[x] ~~Good#test()~~ - useless test, removed via #620

org.eclipse.ua.tests

[ ] ContextTest#testCopyContextWithAttribute() - test is commented out
[x] ~~IndexEntryTest#testUserEntryChildEnablement(), issue~~ - test is commented out, enbaled via #621
[ ] TopicTest#testCopyTopicWithChildRemoveChild(), issue - test is commented out
[ ] TopicTest#testCopyTopicWithChildCheckParents(), issue - test is commented out
[ ] LocaleTest#testSearchQualifiedNames_de(), issue - test is commented out
[ ] TocModelSerializerTest#testRunSerializer() - test is commented out
[ ] AdvancedSearchServiceTest#testRemoteSearchJSONSchema() - test is not yet implemented
[ ] ContextServiceTest#testContextServiceJSONSchema() - test is not yet implemented
[ ] ExtensionServiceTest#testExtensionFragmentServiceJSONSchema() - test is not yet implemented
[ ] IndexFragmentServiceTest#testIndexFragmentServiceJSONSchema() - test is not yet implemented
[ ] SchemaValidator#testJSONSchema() - test is not yet implemented
[ ] TocFragmentServiceTest#testTocFragmentServiceJSONSchema() - test is not yet implemented
[ ] TocServiceTest#testTocServiceJSONSchema() - test is commented out
[x] ~~TocServiceTest#testTocServiceXMLSchema(), issue~~ - test is commented out but working, enabled via #621

eclipse-platform / eclipse.platform