Test failure due to missing API baseline

[eclipse-qvt-oml-bot]

| --- | --- | | Bugzilla Link | 582831 | | Status | NEW | | Importance | P3 normal | | Reported | Jan 05, 2024 07:57 EDT | | Modified | Aug 19, 2024 12:11 EDT | | See also | 583248 | | Reporter | Christopher Gerking |

Description

Changing from Eclipse 2022.12 to 2023.12 makes the JUnit plugin test for bug 579914 fail with the PDE error "An API baseline has not been set for the current workspace". The "current workspace" is the JUnit runtime workspace. To reproduce bug 579914, the test case creates a copy of the org.eclipse.m2m.qvt.oml project, for which API analysis is enabled.

The problem goes away when disabling the API analysis for the project by invoking NatureUtils.removeNature(desc, "org.eclipse.pde.api.tools.apiAnalysisNature") during the test setup. But why is it necessary to do this?

The error message rather indicates that an API baseline should be set, how would that be possible programmatically during a JUnit test?

[eclipse-qvt-oml-bot]

By Christopher Gerking on Jan 05, 2024 08:13

(In reply to Ed Willink from bug 579914 comment #11)

Oh joy. Another infrastructure change. But if it was pushed to master for M3, how did the release builds not fail?

Well, the build does not fail at all. The problem only occurs with a local JUnit run triggered through the Eclipse UI. My bad that I changed directly from 2022-12 to 2023-12, so the problem might actually be present since 2023-03 or 2023-06.

Behind the scenes I think this test autogenerates a new project

Exactly. But its no new project, but a copy of the org.eclipse.m2m.qvt.oml project to reproduce bug 579914.

and so it may be an autogenerator test code bug that the 'use API baseline' preference has not defaulted appropriately; a trivial fix that you have perhaps already found.

In fact, disabling the use of the API baseline is easy for the copied project locally (by removing the respective nature from it after creating the copy). But I was looking for a more global preference to disable the API analysis completely during the tests.

[eclipse-qvt-oml-bot]

By Christopher Gerking on Jan 05, 2024 10:28

(In reply to Christopher Gerking from comment #1)

But I was looking for a more global preference to disable the API analysis completely during the tests.

PDECore.getDefault().getPreferencesManager().setValue(ICoreConstants.DISABLE_API_ANALYSIS_BUILDER, true);

However the local solution is perhaps preferrable, explicitly assigning only the required natures (Java, Plugin) to the respective project.

[eclipse-qvt-oml-bot]

By Ed Willink on Jan 05, 2024 11:11

The local fix is clearly easy. When the sensitivity crept in is a bit academic.

However from an Eclipse user perspective, I see no reason to mandate API tooling; newbies are not ready for this complexity. Your test usage seems pretty trivial and so arguably newbie-ish. As a good Eclipse committer you have an opportunity (?? responsibility ??) to provide a simple repro for a PDE bug report to avoid the regression affecting newbies.

[eclipse-qvt-oml-bot]

By Christopher Gerking on Jan 10, 2024 06:55

(In reply to Ed Willink from comment #3)

The local fix is clearly easy.

Fixed in commit 3214ad5ee96e806c426fd62407e9115c9f807ce7 by restricting the natures to Java and Plugin.

As a good Eclipse committer you have an opportunity (?? responsibility ??) to provide a simple repro for a PDE bug report to avoid the regression affecting newbies.

Well, the repro is so simple that I wonder if it is a bug or intended: in a fresh Eclipse, just create a new plug-in project. The wizard has "Enable API analysis" checked by default, so that the missing API basline error is reported immediately.

[eclipse-qvt-oml-bot]

By Ed Willink on Jul 28, 2024 08:11

Reviewing the cgerking/583248 re-commit, I'm a bit suspicious. You open a project then improve it, which requires the platform to update correctly. I found that the platform fails to use a good classpath when natures change. For synthesized QVTd test projects, I found it necessary to fully synthesize .project, .classpath, plugin.xml and MANIFEST.MF before letting Eclipse open the project.

[eclipse-qvt-oml-bot]

By Christopher Gerking on Aug 03, 2024 04:43

(In reply to Ed Willink from comment #5)

Reviewing the cgerking/583248 re-commit, I'm a bit suspicious. You open a project then improve it, which requires the platform to update correctly.

I agree in principle. But of course Project.setDescription(...) requires that the project has been opened beforehand.

I found that the platform fails to use a good classpath when natures change.

Is it related to bug 544975?

For synthesized QVTd test projects, I found it necessary to fully synthesize .project, .classpath, plugin.xml and MANIFEST.MF before letting Eclipse open the project.

Could it help to invoke setDescription(...) with the appropriate updateFlags before invoking TestUtil.buildProject(...)?

[eclipse-qvt-oml-bot]

By Ed Willink on Aug 03, 2024 05:02

(In reply to Christopher Gerking from comment #6)

(In reply to Ed Willink from comment #5)

Reviewing the cgerking/583248 re-commit, I'm a bit suspicious. You open a project then improve it, which requires the platform to update correctly. I agree in principle. But of course Project.setDescription(...) requires that the project has been opened beforehand.

When you look at synthesizing .project as a text file the attraction of a standard API is outweighed by its inability to work satisfactorily across open.\ \ IIRC the transition from not-Java-nature to Java-nature is particularly troublesome as evidenced by the warning in the Project->Add Nature dialog that users may need to do non-trivial post add nature fixups. The insonsistency benween Add Java Nature and Configure Plugin Projects does not inspire confidence.\

I found that the platform fails to use a good classpath when natures change. Is it related to bug 544975?

Not really. Failure to refresh on open was just an unfriendly behaviour. Failure to accommodate all the incremental changes of a Java / plug-in nature / MANIFEST.MF content is a related but different bug.

For synthesized QVTd test projects, I found it necessary to fully synthesize .project, .classpath, plugin.xml and MANIFEST.MF before letting Eclipse open the project. Could it help to invoke setDescription(...) with the appropriate updateFlags before invoking TestUtil.buildProject(...)?

It is certainly possible that six years later things have got better (or worse or different). I just recall much pain trying to open my synthesized projects in Eclipse until I fully synthesized the 'text' files then opened.

[eclipse-qvt-oml-bot]

By Christopher Gerking on Aug 03, 2024 05:15

(In reply to Christopher Gerking from comment #6)

Could it help to invoke setDescription(...) with the appropriate updateFlags before invoking TestUtil.buildProject(...)?

Definitely not. Even an explicit invocation of refreshLocal(IResource.DEPTH_INFINITE, ...) leaves room for a project description that was not actually updated with the new natures, so that building the project may still cause the PDE error "An API baseline has not been set for the current workspace". I can reproduce the problem when debugging the JUnit test for bug 579914 in Eclipse. All tests pass when running the test, so it seems to be a continuous problem that our test resources are not properly synchronized.

[eclipse-qvt-oml-bot]

By Ed Willink on Aug 03, 2024 10:06

IIRC correctly one of the gotchas is the Java version, which for Java nature is defined in the .classpath, but gets superseded in the plugin nature by the BREE in the MANIFEST.MF but only if the Manifest action to update classpath is manually invoked. So you get stuch with a bad build that isn't cleanable.

The Manifest editor now has dubious nannyish warnings about BREE wrt dependencies. A more useful warning would be stale .classpath.

[eclipse-qvt-oml-bot]

By Christopher Gerking on Aug 04, 2024 06:31

The build of the test project can either fail due to the missing API baseline (reported in this bug), or can fail completely with tons of errors.

At the end of the day, both failures are probably not caused by resource refreshing, but due to concurrent building. Part of the problem is that the build errors reported in TestUtil.getBuildErrors(...) might not be the result of the manual build just triggered by our test, but result from earlier/concurrent auto builds. Whereas ParserTest turns off the auto-building in its setUp(), the bug is that we do not do so in our Bug577992 setUp(), which precedes the super.setUp() invocation. We already applied the same fix to bug 525852, including a join of all running build jobs (which seems to be an essential part of the fix).

Follow-up problems are the missing null checks during tearDown() in both Bug577992 and ParserTest, which prevent a thorough cleanup in case of a failing setup. This can lead to problems in subsequent test cases, which is probably the reason for the 9 failures reported in bug 583248.

Fixes in preparation.

[eclipse-qvt-oml-bot]

By Christopher Gerking on Aug 04, 2024 06:36

My experiments also revealed some rare random failures of the UI completion tests (presumably not related to this bug, or prevented by a proper cleanup for this bug). Anyway, this shows once again that bug 525852 is probably not completely fixed.

[eclipse-qvt-oml-bot]

By Ed Willink on Aug 04, 2024 06:37

(In reply to Christopher Gerking from comment #10)

Fixes in preparation.

Great. So long as regular 'good' builds have a 50/50 chance of failing it will be difficult to judge whether a development code change is good or bad.

[eclipse-qvt-oml-bot]

By Ed Willink on Aug 18, 2024 05:38

(In reply to Ed Willink from comment #12)

Great. So long as regular 'good' builds have a 50/50 chance of failing it will be difficult to judge whether a development code change is good or bad.

The much improved success rate on branch-tests suggests that

commit 74fdef72e2df1fd3e20697ac67a4900e7248e62f\ Author: Christopher Gerking christopher.gerking@kit.edu 2024-08-03 21:04:51\ Committer: Christopher Gerking christopher.gerking@kit.edu 2024-08-07 09:55:02\ Branches: origin/cgerking/582831

[582831] Prevent build failures on test project

needs pushing to master. (Today's master build failed again.)

[eclipse-qvt-oml-bot]

By Ed Willink on Aug 19, 2024 12:11

(In reply to Ed Willink from comment #13)

needs pushing to master. (Today's master build failed again.)

Pushed to master for 3.11.0M3. Master build ok again.