[Build] Perform native binaries build as part of the Jenkins pipeline

[HannesWell]

This changes the Jenkins pipeline of the equinox repository to build the native binaries automatically if the native sources have been changed since the last natives-build. It does the same as the following Jenkins free-style jobs managed only in the Jenkins-UI and triggered manually and consequently makes them obsolete:

• https://ci.eclipse.org/releng/view/Launcher/job/Build-eclipse-launcher/
• https://ci.eclipse.org/releng/view/Launcher/job/cocoa-launcher-macosx-aarch64/
• https://ci.eclipse.org/releng/view/Launcher/job/cocoa-launcher-macosx-x86_64/
• https://ci.eclipse.org/releng/view/Launcher/job/gtk-launcher-linux-aarch64/
• https://ci.eclipse.org/releng/view/Launcher/job/gtk-launcher-linux-ppc64le/
• https://ci.eclipse.org/releng/view/Launcher/job/gtk-launcher-linux-x86_64_pipeline/
• https://ci.eclipse.org/releng/view/Launcher/job/win32-launcher-windows-x86_64/

The task done in the Launcher-new-release job can also be done manually as part of the release procedure if desired at all (it last run over 1.5 years ago). So effectivly the entire Launcher group becomes obsolete.

With this change the native-binaries build also automatically chooses the right maintenance branch to push the built binaries to when changes on maintenance branches are necessary.

The following features of the existing launcher-jobs mentioned above are not supported anymore:

• ~Partial builds only for one specific platform~ (only the binaries of the platform for which the native sources changed and/or a re-build was enforced are build).
• Builds without incrementing binaries-versions. In each native build the version in make_version.mak is incremented (but if the build is not for the mater branch that change is eventually discarded).

This is inspired by the SWT Jenkins pipeline: https://github.com/eclipse-platform/eclipse.platform.swt/blob/aab3bfaeaa2d0823aecdff210d1b8f3c542bc435/Jenkinsfile

Fixes https://github.com/eclipse-equinox/equinox/issues/575

@tjwatson or @sravanlakkimsetti are you interested in reviewing this or are you fine with it as long as it works?

[HannesWell]

I also created a EF GitLab Help Desk issue to ask the Infra-Team to make the Jenkins nodes prepared to build the natives available for the equinox-job:

• https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/4609

[github-actions[bot]]

Test Results

   87 files  ±0     87 suites  ±0   36m 29s :stopwatch: - 1m 8s 2 199 tests ±0  2 152 :white_check_mark: ±0   47 :zzz: ±0  0 :x: ±0  6 729 runs  ±0  6 586 :white_check_mark: ±0  143 :zzz: ±0  0 :x: ±0 

Results for commit e2ba338b. ± Comparison against base commit 0b0fb950.

:recycle: This comment has been updated with latest results.

[HannesWell]

In order to have a reference for the detection of changes in the native-sources and to test this I also pushed the tag https://github.com/eclipse-equinox/equinox/releases/tag/v20240129-1338 to this repository at the commit which the native binaries have been pushed the last time. On the long run I think it would be good to change the tag schema to be a bit more specific respectively more meaningful, e.g. lv-<version> for launcher version and to use its version instead of a timestamp.

[HannesWell]

It looks like everything is working well so far. Although I'm relatively confident that the launcher binaries are correctly built and signed it would be good to have them tested from the branch with the current results: https://github.com/eclipse-equinox/equinox.binaries/tree/test-natives-build-pipeline

@laeubi could you try the binaries for Linux? @HeikoKlare or @Phillipus could you try them out for Mac x86_64/aarch64? For Windows I have already tested them by myself.

[Phillipus]

@HeikoKlare or @Phillipus could you try them out for Mac x86_64/aarch64?

@HannesWell How do I do that?

[HannesWell]

@HeikoKlare or @Phillipus could you try them out for Mac x86_64/aarch64?

@HannesWell How do I do that?

You can download the latest Eclipse SDK from https://download.eclipse.org/eclipse/downloads/drops4/I20240504-1800/ extract it and replace the executable (on mac just eclipse) and the launcher library eclipse_*.so in plugins\org.eclipse.equinox.launcher.cocoa.macosx.* by the one onces found in the linked branch. In https://github.com/eclipse-equinox/equinox.binaries/tree/test-natives-build-pipeline you find the executable under org.eclipse.equinox.executable/bin/cocoa/macosx and the launcher library in the project org.eclipse.equinox.launcher.cocoa.macosx.*. The name mismatch in the launcher library is not a problem, the executable will automatically select the one with the highest version.

[Phillipus]

@HannesWell Tested with both Mac versions x86_64 and aarch64. All working OK! 👍

[HeikoKlare]

Tested with both Mac versions x86_64 and aarch64. All working OK! 👍

Just as an additional confirmation, since @Phillipus already tested: works for me on aarch64 as well. Eclipse configuration log confirms that replaced shared library (11901) is loaded.

[HannesWell]

@HannesWell Tested with both Mac versions x86_64 and aarch64. All working OK! 👍

Tested with both Mac versions x86_64 and aarch64. All working OK! 👍

Just as an additional confirmation, since @Phillipus already tested: works for me on aarch64 as well. Eclipse configuration log confirms that replaced shared library (11901) is loaded.

Awesome. Thank you both for testing!

On the long run I think it would be good to change the tag schema to be a bit more specific respectively more meaningful, e.g. lv-<version> for launcher version and to use its version instead of a timestamp.

I have now implemented an adjusted tagging schema. In the equinox.binaries and equinox repository the automated commit that contains the re-build binaries respectively the changed versions is now tagged as follows: LBv<major>-<minor>

where LBv stands for Launcher Binaries version (upper case since lower case L is easy to confuse with an upper-case i). Major and minor are the versions defined in https://github.com/eclipse-equinox/equinox/blob/0b0fb95097f014b9a3c317b276dd9a5a324ae3f7/features/org.eclipse.equinox.executable.feature/library/make_version.mak#L15-L16

These versions are also used to name the shared library eclipse_<major><minor>.so/dll.

The following features of the existing launcher-jobs mentioned above are not supported anymore:

* Partial builds only for one specific platform

* Builds without incrementing binaries-versions.
  In each native build the version in `make_version.mak` is incremented (but if the build is not for the mater branch that change is eventually discarded).

Furthermore I have now also implemented the possibility for partial re-building the natives of specific platforms. This means if only the launcher sources for one specific platform change (e.g. in the win32 folder) only that platform's launcher binaries are rebuild and committed to the repository. The version in make_version.mak is always incremented and the not build binaries simply keep the old version and will make a larger jump the next time they are build. This discrepancy is not a problem since the executable searches for the latest version on the fly. It is also done this way in the existing, manually triggered Jenkins jobs.

In the Jenkins web-interface re-builds can now be enforced for specific platforms (only):

With that I think this change is in general ready. I'm currently conducting some more testing and plan to submit after I squashed all changed and completed the testing.

[HannesWell]

All testing went well and I'm about to submit this change, which will trigger an initial rebuild of the native launcher binaries. Subsequently only changes to the launcher native sources will (automatically) trigger a rebuild of the binaries.

Thanks to everyone that helped testing.