Improve the release engineering

[merks]

Tasks

• [x] Make Eclipse Marketplace work with the new location
• [X] Update website download locations
• [X] Update project download locations

Description

The setup should produce a usable workspace amenable to contribution. The target platform should be modernized to use newer dependencies; ideally it should be easier to maintain it and to update it to newer dependencies in the future. It should be easier (possible) to do local Tycho builds. It should be easier to produce nightly, milestone, and release builds. There appear to be too many products (*.product) and too many update sites (category.xml). It's not clear why those are all needed and who all consumes those. Simplification and reduction of these would help streamline the releng process.

Perhaps the EPP Eclipse IDE for Java and Report Developers could be revived, but the build results are of a quality not consumable by SimRel:

https://download.eclipse.org/oomph/archive/reports-extra/birt-snapshot/download.eclipse.org/birt/update-site/snapshot/index.html

[merks]

Well, it's not really an icon, it's a unicode character to be precise. But indeed, especially non-native speakers are more likely to understand icon than glyph and certainly no one will be confused that it's an emoji rather than an literal icon. 😕

[wimjongman]

why we shouldn't have on the lefthand side also a chapter "archive"?

Because we don't want people to go there and download old stuff.

You can now respond with: "But it is on the homepage as well" (I play chess ;). , to which I respond that I would rather also remove it from the homepage because we don't want to encourage people to get old stuff.

[wimjongman]

Another thing to consider is that the website is generated. @merks do you have facility to generate additional links?

[merks]

@speckyspooky

One thing to note is that the links in the left column also correspond to the breadcrumb links across the top. I would need to extend JustJ p2 publishing framework to accommodate extra links. That's not a huge challenge I guess, but I also note that there's a whole bunch of "old stuff" in archives and just pointing at the root also seems somewhat unhelpful:

https://archive.eclipse.org/justj/?file=birt

Do we really need folks to go there? I would hope not so much...

[speckyspooky]

OK, I agree with the argument to have only the latest version on the building page.

@wimjongman The experience of our support of the project and my experiences avid to delete the archive. There are many infrastructure dependencies over all versions. For example you cannot force to change by default based on the BIRD project to move to latest Java version. I remember on our unsolved issue with Rhino and Java 11 vs 17. So yes, it make sense to have the archive.

[wimjongman]

Ok, I have updated marketplace and the main website.

I installed latest from the marketplace in the latest 2023-06 but I ran into this issue:

https://github.com/eclipse-birt/birt/issues/1131

We should release 4.14 ASAP.

[speckyspooky]

Please wait with 4.14 I would add my small viewer changes immediately.

But it is a good idea to speed up 4.14.

[merks]

@wimjongman

FYI, I fixed all the licenses and am looking at unsigned content, some of which I've tracked down and will hopefully have fixed:

I also noticed this is open:

https://github.com/eclipse-birt/birt/pull/995/files

I see this version is available as an OSGi artifact so I will determine if we can't just reuse that...

https://repo1.maven.org/maven2/org/apache/derby/derby/10.16.1.1/

[wimjongman]

FYI, I fixed all the licenses and am looking at unsigned content,

I also noticed this is open:

https://github.com/eclipse-birt/birt/pull/995/files

I see this version is available as an OSGi artifact so I will determine if we can't just reuse that...

Thanks, Ed! Great progress for this project.

We are currently on 4.13 which will not run with 23-06 because of the GEF update. Can we hop on the train with 4.14 or will that be the next release?

[merks]

Given yesterday was the deadline for the latest contribution for 2023-06, absolutely not!

In the ideal world (at this point) we'd have a release ready for June 14th that works well with the 2023-06 packages.

The release seems planned for October:

https://projects.eclipse.org/projects/technology.birt/releases/4.14.0

It's been many years since a review!

[wimjongman]

Given yesterday was the deadline for the latest contribution for 2023-06, absolutely not!

😂

[merks]

@wimjongman

I'm not sure how the website works. I think it would be good to include it in the project setup and to include a .project file so one can import it. Is the index.html editable or is it somehow generated?

There's no favicon:

So I'd like to add one as in the diff above...

[merks]

I'm not sure which features in the repository are intended to be installed in an IDE, but if I try to install them all, it fails like this:

Cannot complete the request. Generating details. ERROR: org.eclipse.equinox.p2.director code=10053 Cannot complete the install because one or more required items could not be found. at org.eclipse.oomph.util.OomphPlugin.coreException(OomphPlugin.java:296) at org.eclipse.oomph.p2.internal.core.ProfileTransactionImpl.resolve(ProfileTransactionImpl.java:446) at org.eclipse.oomph.p2.internal.core.ProfileTransactionImpl.commit(ProfileTransactionImpl.java:356) at org.eclipse.oomph.setup.p2.impl.P2TaskImpl.perform(P2TaskImpl.java:904) at org.eclipse.oomph.setup.internal.core.SetupTaskPerformer.doPerformNeededSetupTasks(SetupTaskPerformer.java:3864) at org.eclipse.oomph.setup.internal.core.SetupTaskPerformer.performNeededSetupTasks(SetupTaskPerformer.java:3792) at org.eclipse.oomph.setup.internal.core.SetupTaskPerformer.performTriggeredSetupTasks(SetupTaskPerformer.java:3773) at org.eclipse.oomph.setup.internal.core.SetupTaskPerformer.perform(SetupTaskPerformer.java:3651) at org.eclipse.oomph.setup.ui.wizards.ProgressPage$9.run(ProgressPage.java:592) at org.eclipse.oomph.setup.ui.wizards.ProgressPage$11$1.run(ProgressPage.java:721) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63) ERROR: org.eclipse.equinox.p2.director code=0 Software being installed: artificial_root 1.0.0.v1686307550093 ERROR: org.eclipse.equinox.p2.director code=0 Missing requirement: Apache POI 4.1.1.v20221112-0806 (org.apache.poi 4.1.1.v20221112-0806) requires 'java.package; org.apache.commons.collections4 [4.4.0,5.0.0)' but it could not be found ERROR: org.eclipse.equinox.p2.director code=1 Cannot satisfy dependency: ERROR: org.eclipse.equinox.p2.director code=0 From: artificial_root 1.0.0.v1686307550093 ERROR: org.eclipse.equinox.p2.director code=0 To: org.eclipse.equinox.p2.iu; org.eclipse.birt.engine.runtime.feature.group 0.0.0 ERROR: org.eclipse.equinox.p2.director code=1 Cannot satisfy dependency: ERROR: org.eclipse.equinox.p2.director code=0 From: BIRT Engine Runtime 4.14.0.v202306081942 (org.eclipse.birt.engine.runtime.feature.group 4.14.0.v202306081942) ERROR: org.eclipse.equinox.p2.director code=0 To: org.eclipse.equinox.p2.iu; org.apache.poi [4.1.1.v20221112-0806,4.1.1.v20221112-0806]

I guess I should restrict it to the ones listed here:

https://www.eclipse.org/setups/marketplace/?id=5586359

[wimjongman]

Yes that installs, but that one has too little. It also requires the datatools bundles because, strangely enough, some icons are used from there.

[merks]

@wimjongman

The problem is there is no org.apache.commons.collections4 in the BIRT repo, in the release train repo, nor in the data tools repo. So I will try to include it in the features that include poi and test again...

BTW, did you see my comment https://github.com/eclipse-birt/birt/issues/1286#issuecomment-1582491056 ? I'd like to understand how the website works and how to edit/generate it...

[merks]

FYI, this is the only unsigned content left:

and these are because they use old weak signatures not recognized as signed by recent JREs. Only PGP signatures could address that... Some of these very old dependencies really need a refresh; another thing to discuss with respect to Orbit...

[wimjongman]

👍Looks good. Thanks!

[merks]

FYI, testing repository completeness is a pretty painful process. I must do a local build, which takes about 20 minutes. Then I can test that all the features can install with only the Platform, SimRel and BIRT repos.

Whatever is missing, I must include in some feature. It's a very long cycle time. 😱

[wimjongman]

Ah, you got 20-minute Yoga breaks? Excellent.

[merks]

@wimjongman

FYI, the latest nightly build as no unsigned content:

[wimjongman]

Wow that is amazing. How did you do that? Could you upgrade those old bundles?

[merks]

I mirrored the last Orbit release with old CVS content (the same one the Platform uses) PGP signed all bundles (so it works forever even if the jar signatures go stale) and added new checksums so no md5 warnings all just using cool Tycho features.

https://ci.eclipse.org/oomph/job/simrel/job/simrel-orbit-legacy/

Where there’s a will there’s a way.

[speckyspooky]

@merks I figured out one file which isn't included in oru automated version update. It is the index.jsp of the birt viewer.

Ther the version numbers are added like JSP--variable-declarations. I will change the file because I will renew the old index page in style of the new built-side. And I will also pimp a little bit the example reports.

[merks]

You deleted these two test*.rptdesign files:

The test1.rptdesign file is mentioned in the build.properties so should be removed from there if you delete it.

The test.rptdesign file, as the error message suggests, is mentioned in an ant script to that copies it, apparently as a hello world sample

Probably you shouldn't delete that one but rather update it with the new styling... Other things seem to expect it to exist:

Probably the file names are poor; they make searching find many unrelated matches...

[speckyspooky]

I will to 2 changes like a try. I will change the build.properties & the build.xml to the new name. I doesn't change the name "hello_world_rptdesign". But on this try I would emain the new names.

If the new commit won't work, I will rename to "test.rptdesign" and "test1.rptdesign".

Where you have found the copy issue, there are special log files so that I can see it?

[merks]

The GitHub action log shows this:

The ci job log also shows this:

Each can be opened from the link in the PR.

I suspect the build.properties mention of test1.rptdesign was just careless and isn't needed. But it also doesn't hurt to mention new new name just in case (though the test.rptdesign wasn't mentioned).

A meaningful new name is a good idea!

[speckyspooky]

Thanks, so I know now where I can check the next build.

The new "build is running" (with the new file names - crossed fingers).

[merks]

BTW, this issue prompted me to look at whether we could enable this test of the two runtimes:

But it fails for the OSGi runtime. I'm concerned that this could be an actual problem of something missing. In particular the org.eclipse.core.runtime is missing from the computed URLs...

[speckyspooky]

This testcase would be nice.

The new build finished successful on the PR. So if you could checkit for approval, I can go ahead with the merge.

[merks]

I see org.eclipse.emf.ecore in these places where I would also expect org.eclipse.core.runtime in those places:

merks@CORE MINGW64 /d/Users/merks/birt-master/git/birt/build/birt-packages/birt-runtime-osgi (master) $find . -name org.eclipse.emf.ecore_* ./target/package/ReportEngine/lib/org.eclipse.emf.ecore_2.34.0.v20230406-1203.jar ./target/package/ReportEngine/platform/plugins/org.eclipse.emf.ecore_2.34.0.v20230406-1203.jar ./target/package/WebViewerExample/WEB-INF/lib/org.eclipse.emf.ecore_2.34.0.v20230406-1203.jar ./target/package/WebViewerExample/WEB-INF/platform/plugins/org.eclipse.emf.ecore_2.34.0.v20230406-1203.jar ./target/products/org.eclipse.birt.engine.runtime/win32/win32/x86_64/ReportEngine/plugins/org.eclipse.emf.ecore_2.34.0.v20230406-1203.jar

merks@CORE MINGW64 /d/Users/merks/birt-master/git/birt/build/birt-packages/birt-runtime-osgi (master) $find . -name org.eclipse.core.runtime_* ./target/package/ReportEngine/platform/plugins/org.eclipse.core.runtime_3.27.0.v20230515-1719.jar ./target/package/WebViewerExample/WEB-INF/platform/plugins/org.eclipse.core.runtime_3.27.0.v20230515-1719.jar ./target/products/org.eclipse.birt.engine.runtime/win32/win32/x86_64/ReportEngine/plugins/org.eclipse.core.runtime_3.27.0.v20230515-1719.jar

I see this in the log when building so something is not quite right missing at least org.eclipse.core.runtime:

build-report-engine: [copy] Copying 360 files to D:\Users\merks\birt-master\git\birt\build\birt-packages\birt-runtime-osgi\target\package\ReportEngine\platform\plugins [copy] Copying 37 files to D:\Users\merks\birt-master\git\birt\build\birt-packages\birt-runtime-osgi\target\package\ReportEngine\lib [copy] Copying 3 files to D:\Users\merks\birt-master\git\birt\build\birt-packages\birt-runtime-osgi\target\package\ReportEngine\platform\configuration [copy] Copying 1 file to D:\Users\merks\birt-master\git\birt\build\birt-packages\birt-runtime-osgi\target\package\ReportEngine\platform\configuration [copy] Copying 1 file to D:\Users\merks\birt-master\git\birt\build\birt-packages\birt-runtime-osgi\target\package\ReportEngine\samples [copy] Copying 1 file to D:\Users\merks\birt-master\git\birt\build\birt-packages\birt-runtime-osgi\target\package\ReportEngine [copy] Copying 1 file to D:\Users\merks\birt-master\git\birt\build\birt-packages\birt-runtime-osgi\target\package\ReportEngine

build-webviewer: [echo] D:\Users\merks\birt-master\git\birt\viewer\org.eclipse.birt.report.viewer, D:\Users\merks\birt-master\git\birt\build\birt-packages\birt-runtime-osgi\target\package\WebViewerExample [copy] Copying 160 files to D:\Users\merks\birt-master\git\birt\build\birt-packages\birt-runtime-osgi\target\package\WebViewerExample [copy] Copying 37 files to D:\Users\merks\birt-master\git\birt\build\birt-packages\birt-runtime-osgi\target\package\WebViewerExample\WEB-INF\lib [copy] Copying 363 files to D:\Users\merks\birt-master\git\birt\build\birt-packages\birt-runtime-osgi\target\package\WebViewerExample\WEB-INF [copy] Copying 2 files to D:\Users\merks\birt-master\git\birt\build\birt-packages\birt-runtime-osgi\target\package\WebViewerExample\WEB-INF\lib [copy] Copying 3 files to D:\Users\merks\birt-master\git\birt\build\birt-packages\birt-runtime-osgi\target\package [copy] Copying 1 file to D:\Users\merks\birt-master\git\birt\build\birt-packages\birt-runtime-osgi\target\package\about_files [copy] Copying 1 file to D:\Users\merks\birt-master\git\birt\build\birt-packages\birt-runtime-osgi\target\package

build-webviewer-war: [copy] Copying 1 file to D:\Users\merks\birt-master\git\birt\build\birt-packages\birt-runtime-osgi\target\package\WebViewerExample\WEB-INF [jar] Building jar: D:\Users\merks\birt-master\git\birt\build\birt-packages\birt-runtime-osgi\target\package\birt.war

[speckyspooky]

Please take a look into my changes, I have only enhanced the 2 points from your note above which causes the osgi-issue of the test.rptdesign. But I have seen that the special instruction of file copy is only added on the osgi-product.

So I didn't know exactly what I can do on your new findings - yes, I can rollback all my changes...if you want.

[merks]

I don't expect you to do anything about my findings. I think this is maybe broken and I think maybe it has been for a long time already. Definitely go ahead with your changes!

[speckyspooky]

Ok, I understand and go ahead.

If you have an idea how I can support you let me know.

[merks]

My sense is that this test of the OSGi runtime never really worked and is maybe not actually very sensible. Certainly the URL classloader it's creating is very incomplete and is missing everything in the birt-runtime-osgi/ReportEngine/platform/plugins folder. When that's added, some things still fail and that seems related to nested jars in the Bundle-ClassPath not being found ,which is not surprising when only the containing bundle is on the classpath.

E.g., this fails to load:

Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: %Bundle-Name
Bundle-SymbolicName: org.eclipse.birt.report.engine; singleton:=true
Bundle-Version: 4.14.0.qualifier
Bundle-ClassPath: enginetesthelper.jar,
 .,
 lib/Tidy.jar,
 lib/commons-cli-1.0.jar

I don't want to bang my head against the wall on something that doesn't make sense...

[speckyspooky]

You are right, we need your head and brain in future. So please don't bang your head!

[merks]

The other runtime passes the test so I'm somewhat tempted to disable the OSGi part, but better would be to actually test it. Anyway, I've had to put that on the back burner with too much else to do...

I have looked at how to get a newer version of poi, which is a bit of tangles nightmare of dependencies, so that is a work in progress. Also, I thought getting a new derby would be easy, but I don't know if their bundle is actually functional. It's certainly poorly named and is problematic to get through the IP review process:

https://gitlab.eclipse.org/eclipsefdn/emo-team/iplab/-/issues/8773

[wimjongman]

Yeah, I don't really know the history of the OSGi runtime, nor if it is used by anybody. It is sure that the OSGi runtime is embedded in the IDE but I'm not sure what its advantage is vs the non-osgi runtime.

Maybe @thelaserninja knows?

[chloetz]

Just for information: all our users (approx. 350 000) use the OSGi runtime.

Von: Wim Jongman @.*** Gesendet: Mittwoch, 14. Juni 2023 08:54 An: eclipse-birt/birt Cc: Loetz, Christophe; Team mention Betreff: Re: [eclipse-birt/birt] Improve the release engineering (Issue #1286)

Yeah, I don't really know the history of the OSGi runtime, nor if it is used by anybody. It is sure that the OSGi runtime is embedded in the IDE but I'm not sure what its advantage is vs the non-osgi runtime.

Maybe @TheLaserNinjahttps://github.com/TheLaserNinja knows?

— Reply to this email directly, view it on GitHubhttps://github.com/eclipse-birt/birt/issues/1286#issuecomment-1590578929, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ATFLNSFOYRHPPGXTCCAQMH3XLFNXZANCNFSM6AAAAAAYJLVV5E. You are receiving this because you are on a team that was mentioned.Message ID: @.***>

[speckyspooky]

I have looked at how to get a newer version of poi, which is a bit of tangles nightmare of dependencies, so that is a work in progress...

Thanks for your investigation on it, yes the apache poi has many dependencies. So it could be if we would have a newer version on BIRT side we have to change/optimize some project parts e.g. the excel emitters of (own & spudsoft)

[wimjongman]

I think we are done here. Thanks, Ed, and all others involved.