Improve releng

[merks]

I wanted to establish (agree on) the goals for improving the releng for Nebula.

I assume one goals is to improve the update site structure as was done for other projects such as GEF recently:

https://download.eclipse.org/justj/?file=nebula

Do we want to keep maintaining the distinction between the non-incubating versus non-incubating widgets? If we are to maintain two updates sites, I would prefer each to be rooted in a different child folder of the nebula folder, but with what names?

Another goal is to have the build driven by a multi-branch Jenkinsfile. Probably we can do that with a single job even if we want to produce two separate update sites; I think it would still make sense to produce releases for the incubating widget...

@lcaron @wimjongman

Please share your thoughts about what you would like to see as an end-result.

[laeubi]

As a (random) user of nebula I never cared if something is "incubating" or not ... just put everything in one site, if there is a bug it must be fixed, at best by the reporting user if its really critical.

[merks]

It's certainly easier to maintain one site and it's an interesting and useful data point that as a consumer, a single site is also more convenient. Not only that, a bunch of incubating components have 1.x versions that suggest they are not incubating:

The builds don't appear to be using jgit stamping for the qualifier. I wonder if another goal would be to change that?

[wimjongman]

The incubation widgets are not stable. They should be separate or source only. In terms of root naming, we use incubation and release.

[laeubi]

@wimjongman does it really matter? I even use the SNPASHOTs an never get any "instability" given that there are not much contributions to nebula one should decide how much LTS support is available for a project.

e.g. for Tycho we always do MAJOR release now, so we don't promise 100% compatibility even though its often only a matter of increase the version number for 99% of the builds.

[merks]

The incubation widgets are not stable. They should be separate or source only. In terms of root naming, we use incubation and release.

Keep in mind the structure of an update site:

https://download.eclipse.org/justj/?file=tools/gef/classic

The red things are legacy things before before JustJ was used for publishing.

So names like mature/stable/ versus incubation at the root. If we don't publish/promote the incubation update site at all (and it's only available via the archive in the build job) that's fine too and then we can maintain a single site in the root directly. I'd just like to keep this simple to reduce the amount of work...

[laeubi]

It might be a different category but as said please still provide them (under whatever name)!

[wimjongman]

I see. Ok, whatever is most simple and user-friendly. If we have the incubation widgets in the same repo but in a different update site category, that is also fine by me.

[jantje]

As a (random) user of nebula I never cared if something is "incubating" or not ...

As a random user I care if something is "incubating" or not.

a bug it must be fixed

As a random user I am very happy that people spend their time developing and maintaining open source. I think it would be very arrogant of me to demand "bugs to be fixed". Who will fix it? Why would they? Even in many commercial projects there are many bugs that will never be fixed.

If we have the incubation widgets in the same repo but in a different update site category, that is also fine by me.

That would be fine for me to. Note however that any change in the repo's is very likely to cause incompatibilities in projects using nebula.

[merks]

@jantje

FYI, the Eclipse infrastructure has some convenient capabilities that will help with modernizing the update sites as we did for GEF in a way that's minimally disruptive to the existing consumers. You can read the details of that effort here if you are interested.

https://github.com/eclipse/gef-classic/issues/313

In particular if one moves something from download.eclipse.org to archive.eclipse.org, the 404 handler for download.eclipse.org will redirect the former to the latter (like a mirror). For example this link redirects in that way:

https://download.eclipse.org/tools/gef/classic/releases/3.18.0

The original URL can be used directly by p2:

This works well for simple update sites.

For composites, we can change the URL that it composes.

In this way we can ensure that all the old URLs continue to work while establishing an easier-to-maintain structure. This will allow the developers to spend less time on releng, accommodating more time for actually making code fixes and improvements, and makes it easier to deliver those results.

[jantje]

@merks Thanks for the input. Thanks for caring about existing users. Thanks for Nebula. 👍

[merks]

This isn’t done or even started yet..,

[lcaron]

My bad, sorry (I need to sleep)

[merks]

FYI, I have been making progress locally:

There are lots of inconsistencies and even typos in the names of bundles and features, so quite a bit to do to make this all clean...

[merks]

FYI, I plan to ensure that all incubating categories, features, and bundles are clearly marked as incubating in the unified update sites:

[lcaron]

Great ! Thank you your involvement.

[merks]

I've cleaned up and "normalized" the feature/bundle names:

So I'm almost ready to start real builds...

[merks]

@lcaron @wimjongman

I was making changes so that the two old Maven builds would still work, but it would be cleaner and simpler if I didn't concern myself with that. Are guys okay if I do a deeper cleaning to focus on a single Maven build of the union of all the things?

[wimjongman]

That is fine with me. I think now we have a stable and incubation feature that respectively contains all stable and incubation features, we also present the individual features. Each widget has its own feature. Can you work with that?

[laeubi]

Just in case, if each feature has only one bundle you can use the bundle as a feature so no need to duplicate all bundles as features.

[merks]

FYI, I have the first promoted builds available here:

https://download.eclipse.org/nebula/updates/nightly/latest

It's being built by this job from the "main" branch of my fork:

https://ci.eclipse.org/nebula/job/nebula-build-test/

The Jenkinsfile will be usable as a multi-branch pipeline...

Are there any outstanding concerns before I merge this into master?

---

This build is building against https://download.eclipse.org/releases/latest but I have tested that https://download.eclipse.org/releases/photon also works. Do we want the build to verify that it can build and run tests against photon? A build with testing and without signing takes about 4.5 minutes...

[wimjongman]

Maybe rename Nebula Widgets SDK to Nebula All Stable Widgets SDK so that it sits on top together with the incubation widgets.

Now the stable widgets all feature is in the bottom.

---

I don't think that photon is a concern. People can still use the old builds. "Wow" for the build speed Do we have active tests in nebula. I remember this not working in the past due to some VNC problems

[merks]

@wimjongman @lcaron

The multi-branch pipeline is working:

https://ci.eclipse.org/nebula/job/nebula-build/job/master/

I've copied over all the existing releases to the archive:

https://archive.eclipse.org/justj/?file=nebula/releases

I've tested locally that I can copy the most recent release into the new update structure and that I can redirect (via composites) the other "moving target" repositories into the new structure:

As such, I can delete these folders and replace them with the content of the above screen capture:

Via the composites and via the download.eclipse.org server's 404 handler, no existing clients using any of the legacy URLs should see any errors.

I've made a temporary copy of everything in case something goes horribly wrong:

https://download.eclipse.org/justj/?file=nebula/archive

If you have concerns, scream quickly because I'm ready to pull the plugin...

[merks]

I've tested that the following legacy URLs work:

• https://download.eclipse.org/nebula/incubation/latest/
• https://download.eclipse.org/nebula/incubation/snapshot/
• https://download.eclipse.org/nebula/snapshot/

All are redirections to the following:

https://download.eclipse.org/nebula/updates/nightly/latest

I've also tested that the following legacy URL works:

https://download.eclipse.org/nebula/releases/latest/

It is a redirection to the following:

https://download.eclipse.org/nebula/updates/release/latest

Which in turn currently redirects to:

https://download.eclipse.org/nebula/updates/release/3.1.1

The new update site location https://download.eclipse.org/nebula/updates is complete in that it includes even a latest release, allowing downstream consumers to migrate to the new structure at their leisure:

Note that I assumed the next release would be 3.1.2 which is determined by the version of this "branding feature":

• https://github.com/eclipse/nebula/blob/b4465920c2dbbca0d0ea965f37e9d49c1ccabff1/releng/org.eclipse.nebula.feature/pom.xml#L27
• https://github.com/eclipse/nebula/blob/b4465920c2dbbca0d0ea965f37e9d49c1ccabff1/releng/org.eclipse.nebula.feature/pom.xml#L27

But then I noticed that the next planned version is 3.2.0:

https://projects.eclipse.org/projects/technology.nebula/releases/3.2.0

So we'll need to change that version if that's the plan...

Please don't be afraid to ask questions!!

---

Almost all the jobs for https://ci.eclipse.org/nebula/ can be deleted now. Any concerns about deleting these?

I ran a report and things look to be in very good shape; lack of branding images is no big deal:

https://download.eclipse.org/oomph/archive/reports-extra/nebula-nightly/download.eclipse.org/nebula/updates/nightly/latest/index.html

[wimjongman]

Wow, Ed, this is just great. I can't believe how professional all this looks. Thank you!! 🙏

I will make an issue for creating the branding images with the Nebula logo.

So we'll need to change that version if that's the plan...

Yes, the next version is 3.2.0. We reserve the last digit for maintenance releases.

Please go ahead and delete the builds.

[merks]

It's suggest not worry about the branding images because the overhead for these things is stupid. Somehow the feature then gets a bunch of information from the branding plugin so unless each feature has its own branding plugin, it's kind of a confusing mess. Also, I don't think downstream consumers generally redistribute the feature making the time and overhead really not a good return on investment...

Look here

https://download.eclipse.org/staging/2024-03/buildInfo/archive/download.eclipse.org/staging/2024-03/index.html

The only visible impact to the user is no icon on the Help -> About dialog...

[wimjongman]

Ok, good to know.

[merks]

FYI, I have added a .project file to the repository root. To avoid having the same files appear more than once in the workspace (which I find super annoying, especially for searches), I have filtered that project's contents so that nested folders corresponding to projects that are themselves imported into the workspace are displayed without children:

I thought it best to explain this in case you wondered and were confused by it...

---

There is also a launcher for replicating the build locally:

[merks]

The build server https://ci.eclipse.org/nebula/ is cleaned up:

And each job has documentation:

PR builds will kick in manually. Such builds do not sign and do not publish results, though they do archive the unsigned repository contents. Branch builds are kicked of like this:

There are three choices of build type:

Give thought to the branding version

https://github.com/eclipse/nebula/blob/b4465920c2dbbca0d0ea965f37e9d49c1ccabff1/releng/org.eclipse.nebula.feature/pom.xml#L27

As this is what you'll see in the update sites:

And this will determine the release version number when you finally do a release build.

Note that a release build does a full build, but ignores the result repository. It will promote, byte-for-byte, the most recent milestone build. Do not do a release build without ensuring that the most recent commit is in the most recent milestone build and that the version number is the intended version of the release

[merks]

@ewillink

FYI, here is a recent example of using JustJ's p2.manager with detailed descriptions. Here is the meaty part of the pom.xml

https://github.com/eclipse/nebula/blob/b4465920c2dbbca0d0ea965f37e9d49c1ccabff1/releng/org.eclipse.nebula.site/promotion/pom.xml#L69-L101

It is just like the provided example:

https://eclipse.dev/justj/?page=tools#p2-manager-maven

[wimjongman]

Great stuff, Ed. I can't thank you enough.

[merks]

@wimjongman @lcaron

This action fails:

https://github.com/eclipse/nebula/actions/workflows/github-code-scanning/codeql

I appears that it needs to use a newer version of Java. I have no clue how this is configured.

[wimjongman]

I'm not sure either. I have not seen this before. Should it not come from https://github.com/eclipse/nebula/blob/master/.github/workflows/maven.yaml ?

[merks]

I would have thought so too, but that's only for this as far as I can tell:

https://github.com/eclipse/nebula/actions/workflows/maven.yaml

Maybe the foundation was asked to set this up and we are not able to configure it ourselves? Maybe we we should moving to an eclipse-nebula organization?

[wimjongman]

Yes. I have filed the request here: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/4275

[lcaron]

@merks Your work is just awesome. Thank you very very much :)

[merks]

I'll close this and we can do other stuff if/when we have our own org.