Convert Jenkinsfile to declarative pipeline

cdietrich commented 11 months ago

we should convert the current scripted pipeline Jenkinsfile to a declarative one (analogous to Xtext and MWE) cc @tjeske

merks commented 10 months ago

I am working on this as well, or plan too do that in the coming days.

Locally I have the build producing "proper update sites" like this:

This is like what I've done for a bunch of other projects...

Hopefully you guys are into that. I don't want to waste my time...

cdietrich commented 10 months ago

Help is welcome

merks commented 10 months ago

Very good!

I believe I can prototype everything from a fork, including prototyping the build from my fork, because I can use one of my other ci instance to build my fork directly. That way I'll have a fully functional proof of concept when I create a pull request later on...

merks commented 10 months ago

I'm pleased to report good progress.

This prototype build:

https://ci.eclipse.org/oomph/job/internal/job/xpect/

produces this update site:

https://download.eclipse.org/oomph/archive/xpect/updates/nightly/latest

The changes are on the master branch of my fork:

https://github.com/merks/Xpect

This is the declarative pipeline:

https://github.com/merks/Xpect/blob/master/Jenkinsfile

merks commented 10 months ago

@cdietrich

One huge request I have here is that Xpect produces a release compatible with Xtext's 2023-09 release before you start doing the migration to the new bundle symbolic names. Please please!

cdietrich commented 10 months ago

@merks i dont understand what you are referring to.

Xpect does not produce releases (i also have no cappa at all to work on that. i will even struggle a lot to produce something for Xtext)
which bundle symbolic name changes are you referring to?

merks commented 10 months ago

I am working on that and I will help with that. It's pretty much a trivial activity with the promotion infrastructure I'm adding right now...

I just noticed that building against the nightly build target doesn't actually work because of the classgraph bundle name change so given I have a bunch of changes in the works, I'm hoping that you are not making changes to xpect master while my changes are pending...

cdietrich commented 10 months ago

We can hold changes back until you are finished with your contribution

merks commented 10 months ago

Thank you. 👍

Will you be at EclipseCon next week so we can chat in person?

merks commented 10 months ago

The prototype is pretty much complete. This multi-branch pipeline does the builds:

https://ci.eclipse.org/oomph/job/internal/job/xpect/

The builds are parameterized with these as the default parameter values:

I fixed the nightly build target platform to work as well:

Doing milestone and release builds is just a matter of selecting that in the drop-down:

Here is the resulting update site after doing a milestone and a final release build:

https://download.eclipse.org/oomph/archive/xpect/updates/release/latest/

Note that the URLs displayed are for where the actual update sites will be when Xpect produces them.

@cdietrich

I don't have much time left to create a PR and tomorrow I travel to the conference...

All the changes are in this commit:

https://github.com/merks/Xpect/commit/master

If I were a committer, I could do all the final tweaking myself...

cdietrich commented 10 months ago

@merks many thanks. the pr looks very nice. unfortunately i wont make it to eclipsecon

merks commented 10 months ago

😭 I was really really hoping to see in person again!

cdietrich commented 10 months ago

new job still keeps me very busy.

merks commented 10 months ago

Well, that was one of the things I wanted to hear about!!

cdietrich commented 10 months ago

done

merks commented 10 months ago

@cdietrich

Look, it published the repo!

https://download.eclipse.org/xpect/updates/nightly/latest

I tested the real setup and it worked too.

https://github.com/eclipse/Xpect/blob/master/CONTRIBUTING.md

Could I ask you to kick off the build manually and choose "milestone"? So that we have our first milestone?

Do you want me to change the pipeline so that there is no target platform choice but rather both are built? The nightly one would be used only for test verification and promotion would only apply to the other one. Okay?

cdietrich commented 10 months ago

nice. @merks please do so, but take your time.

trancexpress commented 10 months ago

@merks , how do I build Xpect locally, with your change? I get asked for a password:

[INFO] --- tycho-eclipserun:3.0.0:eclipse-run (promote) @ org.eclipse.xpect.repository.promotion ---
[INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/tycho/extras/tycho-p2-extras-plugin/3.0.0/tycho-p2-extras-plugin-3.0.0.pom
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/eclipse/tycho/extras/tycho-p2-extras-plugin/3.0.0/tycho-p2-extras-plugin-3.0.0.pom (5.5 kB at 500 kB/s)
[INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/tycho/extras/tycho-p2-extras-plugin/3.0.0/tycho-p2-extras-plugin-3.0.0.jar
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/eclipse/tycho/extras/tycho-p2-extras-plugin/3.0.0/tycho-p2-extras-plugin-3.0.0.jar (31 kB at 3.5 MB/s)
[INFO] ### Using TychoRepositoryTransport for remote P2 access ###
[INFO]     Cache location:         /data/git/Xpect/xpect-local-maven-repository/.cache/tycho
[INFO]     Transport mode:         online
[INFO]     Update mode:            forced
[INFO]     Minimum cache duration: 60 minutes
[INFO]       (you can configure this with -Dtycho.p2.transport.min-cache-minutes=<desired minimum cache duration>)
[INFO] Adding repository https://download.eclipse.org/releases/2023-09
[INFO] Adding repository https://download.eclipse.org/justj/tools/updates/nightly/latest
[INFO] Expected eclipse log file: /data/git/Xpect/org.eclipse.xpect.releng/promotion/target/eclipserun-work/data/.metadata/.log
[INFO] Command line:
        [/usr/lib/jvm/jdk-17_adoptium/bin/java, -jar, /data/git/Xpect/xpect-local-maven-repository/p2/osgi/bundle/org.eclipse.equinox.launcher/1.6.500.v20230717-2134/org.eclipse.equinox.launcher-1.6.500.v20230717-2134.jar, -install, /data/git/Xpect/org.eclipse.xpect.releng/promotion/target/eclipserun-work, -configuration, /data/git/Xpect/org.eclipse.xpect.releng/promotion/target/eclipserun-work/configuration, -data, /data/git/Xpect/org.eclipse.xpect.releng/promotion/target/eclipserun-work/data, -consoleLog, -application, org.eclipse.justj.p2.manager, -data, @None, -nosplash, -remote, genie.xpect@projects-storage.eclipse.org:/home/data/httpd/download.eclipse.org/xpect, -retain, 5, -label, Xpect, -build-url, http://www.example.com/, -root, /data/git/Xpect/org.eclipse.xpect.releng/promotion/target/xpect-sync, -relative, updates, -excluded-categories-pattern, .*.Default, -iu-filter-pattern, org.eclipse.xpect.*, -version-iu, org.eclipse.xpect.sdk., -commit, https://github.com/eclipse/Xpect/commit/${git.commit}, -target-url, https://download.eclipse.org/xpect, -promote, /data/git/Xpect/org.eclipse.xpect.releng/promotion/../p2-repository/target/repository, -timestamp, 2023-10-20T15:24:34Z, -type, nightly, -breadcrumb, Xpect https://projects.eclipse.org/projects/modeling.xpect, -favicon, https://eclipseide.org/favicon.ico]
!SESSION 2023-10-20 17:29:09.826 -----------------------------------------------
eclipse.buildId=unknown
java.version=17.0.6
java.vendor=Eclipse Adoptium
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Framework arguments:  -application org.eclipse.justj.p2.manager -remote genie.xpect@projects-storage.eclipse.org:/home/data/httpd/download.eclipse.org/xpect -retain 5 -label Xpect -build-url http://www.example.com/ -root /data/git/Xpect/org.eclipse.xpect.releng/promotion/target/xpect-sync -relative updates -excluded-categories-pattern .*.Default -iu-filter-pattern org.eclipse.xpect.* -version-iu org.eclipse.xpect.sdk. -commit https://github.com/eclipse/Xpect/commit/${git.commit} -target-url https://download.eclipse.org/xpect -promote /data/git/Xpect/org.eclipse.xpect.releng/promotion/../p2-repository/target/repository -timestamp 2023-10-20T15:24:34Z -type nightly -breadcrumb Xpect https://projects.eclipse.org/projects/modeling.xpect -favicon https://eclipseide.org/favicon.ico
Command-line arguments:  -data /data/git/Xpect/org.eclipse.xpect.releng/promotion/target/eclipserun-work/data -consoleLog -application org.eclipse.justj.p2.manager -data @None -remote genie.xpect@projects-storage.eclipse.org:/home/data/httpd/download.eclipse.org/xpect -retain 5 -label Xpect -build-url http://www.example.com/ -root /data/git/Xpect/org.eclipse.xpect.releng/promotion/target/xpect-sync -relative updates -excluded-categories-pattern .*.Default -iu-filter-pattern org.eclipse.xpect.* -version-iu org.eclipse.xpect.sdk. -commit https://github.com/eclipse/Xpect/commit/${git.commit} -target-url https://download.eclipse.org/xpect -promote /data/git/Xpect/org.eclipse.xpect.releng/promotion/../p2-repository/target/repository -timestamp 2023-10-20T15:24:34Z -type nightly -breadcrumb Xpect https://projects.eclipse.org/projects/modeling.xpect -favicon https://eclipseide.org/favicon.ico

!ENTRY org.eclipse.justj.p2 2 0 2023-10-20 17:29:11.933
!MESSAGE Starting the p2 manager
/usr/bin/ssh genie.xpect@projects-storage.eclipse.org mkdir -p /home/data/httpd/download.eclipse.org/xpect/updates
genie.xpect@projects-storage.eclipse.org's password:

cdietrich commented 10 months ago

Simply disable the profile does not work ? -P!promote

merks commented 10 months ago

Note too that there are these run configurations:

They should all work locally. Even promotion can be tested locally:

trancexpress commented 10 months ago

@merks I can't build Xpect locally anymore, I get asked for a password:

[INFO] --- tycho-eclipserun:4.0.3:eclipse-run (promote) @ org.eclipse.xpect.repository.promotion ---
[INFO] ### Using TychoRepositoryTransport for remote P2 access ###
[INFO]     Cache location:         /data/git/Xpect/xpect-local-maven-repository/.cache/tycho
[INFO]     Transport mode:         online
[INFO]     Http Transport type:    Java11Client
[INFO]     Update mode:            forced
[INFO]     Minimum cache duration: 60 minutes
[INFO]       (you can configure this with -Dtycho.p2.transport.min-cache-minutes=<desired minimum cache duration>)
[INFO] Adding repository https://download.eclipse.org/releases/2023-09
[INFO] Adding repository https://download.eclipse.org/justj/tools/updates/nightly/latest
[INFO] Expected Eclipse log file: /data/git/Xpect/org.eclipse.xpect.releng/promotion/target/eclipserun-work/data/.metadata/.log
[INFO] Command line:
        [/usr/lib/jvm/jdk-17_adoptium/bin/java, -jar, /data/git/Xpect/xpect-local-maven-repository/p2/osgi/bundle/org.eclipse.equinox.launcher/1.6.500.v20230717-2134/org.eclipse.equinox.launcher-1.6.500.v20230717-2134.jar, -install, /data/git/Xpect/org.eclipse.xpect.releng/promotion/target/eclipserun-work, -configuration, /data/git/Xpect/org.eclipse.xpect.releng/promotion/target/eclipserun-work/configuration, -data, /data/git/Xpect/org.eclipse.xpect.releng/promotion/target/eclipserun-work/data, -consoleLog, -application, org.eclipse.justj.p2.manager, -data, @None, -nosplash, -remote, genie.xpect@projects-storage.eclipse.org:/home/data/httpd/download.eclipse.org/xpect, -retain, 5, -label, Xpect, -build-url, http://www.example.com/, -root, /data/git/Xpect/org.eclipse.xpect.releng/promotion/target/xpect-sync, -relative, updates, -excluded-categories-pattern, .*.Default, -iu-filter-pattern, org.eclipse.xpect.*, -version-iu, org.eclipse.xpect.sdk., -commit, https://github.com/eclipse/Xpect/commit/${git.commit}, -target-url, https://download.eclipse.org/xpect, -promote, /data/git/Xpect/org.eclipse.xpect.releng/promotion/../p2-repository/target/repository, -timestamp, 2023-10-20T14:54:38Z, -type, nightly, -breadcrumb, Xpect https://projects.eclipse.org/projects/modeling.xpect, -favicon, https://eclipseide.org/favicon.ico]
!SESSION 2023-10-20 16:57:45.584 -----------------------------------------------
eclipse.buildId=unknown
java.version=17.0.6
java.vendor=Eclipse Adoptium
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Framework arguments:  -application org.eclipse.justj.p2.manager -remote genie.xpect@projects-storage.eclipse.org:/home/data/httpd/download.eclipse.org/xpect -retain 5 -label Xpect -build-url http://www.example.com/ -root /data/git/Xpect/org.eclipse.xpect.releng/promotion/target/xpect-sync -relative updates -excluded-categories-pattern .*.Default -iu-filter-pattern org.eclipse.xpect.* -version-iu org.eclipse.xpect.sdk. -commit https://github.com/eclipse/Xpect/commit/${git.commit} -target-url https://download.eclipse.org/xpect -promote /data/git/Xpect/org.eclipse.xpect.releng/promotion/../p2-repository/target/repository -timestamp 2023-10-20T14:54:38Z -type nightly -breadcrumb Xpect https://projects.eclipse.org/projects/modeling.xpect -favicon https://eclipseide.org/favicon.ico
Command-line arguments:  -data /data/git/Xpect/org.eclipse.xpect.releng/promotion/target/eclipserun-work/data -consoleLog -application org.eclipse.justj.p2.manager -data @None -remote genie.xpect@projects-storage.eclipse.org:/home/data/httpd/download.eclipse.org/xpect -retain 5 -label Xpect -build-url http://www.example.com/ -root /data/git/Xpect/org.eclipse.xpect.releng/promotion/target/xpect-sync -relative updates -excluded-categories-pattern .*.Default -iu-filter-pattern org.eclipse.xpect.* -version-iu org.eclipse.xpect.sdk. -commit https://github.com/eclipse/Xpect/commit/${git.commit} -target-url https://download.eclipse.org/xpect -promote /data/git/Xpect/org.eclipse.xpect.releng/promotion/../p2-repository/target/repository -timestamp 2023-10-20T14:54:38Z -type nightly -breadcrumb Xpect https://projects.eclipse.org/projects/modeling.xpect -favicon https://eclipseide.org/favicon.ico

!ENTRY org.eclipse.justj.p2 2 0 2023-10-20 16:57:47.564
!MESSAGE Starting the p2 manager
/usr/bin/ssh genie.xpect@projects-storage.eclipse.org mkdir -p /home/data/httpd/download.eclipse.org/xpect/updates
genie.xpect@projects-storage.eclipse.org's password: 

a
a
genie.xpect@projects-storage.eclipse.org's password: 
genie.xpect@projects-storage.eclipse.org's password: 
Warning: Permanently added 'projects-storage.eclipse.org' (ECDSA) to the list of known hosts.
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,password).
exitValue=255
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for org.eclipse.xpect.root 0.3.0-SNAPSHOT:
[INFO] 
[INFO] org.eclipse.xpect.parent ........................... SUCCESS [  0.512 s]
[INFO] org.eclipse.xpect.mwe2 ............................. SUCCESS [01:02 min]
[INFO] org.eclipse.xpect .................................. SUCCESS [ 28.250 s]
[INFO] org.eclipse.xpect.ide .............................. SUCCESS [  2.201 s]
[INFO] org.eclipse.xpect.ui ............................... SUCCESS [  9.095 s]
[INFO] org.eclipse.xpect.ui.junit ......................... SUCCESS [  5.303 s]
[INFO] org.eclipse.xpect.xtext.lib ........................ SUCCESS [  2.002 s]
[INFO] org.eclipse.xpect.xtext.xbase.lib .................. SUCCESS [  2.279 s]
[INFO] org.eclipse.xpect.xtext.lib.feature ................ SUCCESS [  1.323 s]
[INFO] org.eclipse.xpect.xtext.xbase.lib.feature .......... SUCCESS [  0.632 s]
[INFO] org.eclipse.xpect.sdk .............................. SUCCESS [  0.858 s]
[INFO] org.eclipse.xpect.repository ....................... SUCCESS [  4.014 s]
[INFO] org.eclipse.xpect.target.eclipse_2023_09-xtext_nightly SUCCESS [  0.009 s]
[INFO] org.eclipse.xpect.root ............................. SUCCESS [  0.017 s]
[INFO] org.eclipse.xpect.repository.promotion ............. FAILURE [01:41 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  04:34 min
[INFO] Finished at: 2023-10-20T16:59:12+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.eclipse.tycho.extras:tycho-eclipserun-plugin:4.0.3:eclipse-run (promote) on project org.eclipse.xpect.repository.promotion: Error while executing platform: Error while executing platform: return code=255, see content of /data/git/Xpect/org.eclipse.xpect.releng/promotion/target/eclipserun-work/data/.metadata/.logfor more details. -> [Help 1]

Should this promotion really be enabled by default?

merks commented 10 months ago

I played with that and it somehow seems as if the profiles could not be independently configured. Not specifying the target explicitly also didn’t work (for me). So there are checked in launchers that do the correct thing and they’re easy to use.

trancexpress commented 10 months ago

Sorry for the double post, I didn't notice the 2nd time I was writing a comment, that I already wrote one.

Simply disable the profile does not work ? -P!promote

Yes, this works. But why do I have to add an extra argument just to make the build work?

merks commented 10 months ago

I think I tried to answer that. When I made that one not active by default it disabled them all by default. Not what I expected…. Otherwise I would have done it like that.

trancexpress commented 10 months ago

I think I tried to answer that. When I made that one not active by default it disabled them all by default. Not what I expected…. Otherwise I would have done it like that.

Ah, sorry then, I didn't understand your comment.

Maybe https://github.com/eclipse/Xpect#readme can be updated to include -P!promote then? That'd work for me.

merks commented 10 months ago

Yes the readme needs updating. Also to document the new update sites. I plan to follow up on that next week.

eclipse / Xpect

Convert Jenkinsfile to declarative pipeline #328