Move to Maven/Tycho for projects and build infrastructure

[LorenzoBettini]

Replaces https://github.com/eclipse/xtext-lib/pull/500

[LorenzoBettini]

@cdietrich After removing the features the xtext-eclipse still fails. I just realized that .xbase.lib (without .feature) is what we actually publish: https://download.eclipse.org/modeling/tmf/xtext/updates/releases/2.29.0/features/

That's a small disgrace because changing names now would be disruptive.

I'll see whether with the suggestion https://github.com/eclipse/xtext-lib/pull/500#issuecomment-1387173352 but I don't like it right from the start...

If also that solution does not work, I can still fix things: it's just a matter of building the features (because we do the same "mistake" in most of our features) and the p2 repository on a separate Maven reactor (i.e., on another Maven build step).

Changing the eclipse project's name, on the other hand, can be done without problems.

[cdietrich]

i dont know if we want to change the feature name. does it work to add the feature.group to pom and project but keep the id? i hoped @szarnekow has an opinion on that

[cdietrich]

also the category.xml does not contain it anymore

[cdietrich]

of course we could also rename the two features and adopt all downstream stuff. and add it to the release notes

@szarnekow what do you think?

[cdietrich]

https://download.eclipse.org/eclipse/updates/4.27-I-builds/I20230118-0200/plugins/ and https://download.eclipse.org/eclipse/updates/4.27-I-builds/I20230118-0200/features/ has both a org.eclipse.jdt_ wonder how they do that. they have a different group-id https://github.com/eclipse-jdt/eclipse.jdt/blob/701de3cca1ea57ea701a937fd27426757c6a5dd2/org.eclipse.jdt-feature/pom.xml#L19

[LorenzoBettini]

also the category.xml does not contain it anymore

it's because in my latest commit I removed it

[LorenzoBettini]

https://download.eclipse.org/eclipse/updates/4.27-I-builds/I20230118-0200/plugins/ and https://download.eclipse.org/eclipse/updates/4.27-I-builds/I20230118-0200/features/ has both a org.eclipse.jdt_ wonder how they do that. they have a different group-id https://github.com/eclipse-jdt/eclipse.jdt/blob/701de3cca1ea57ea701a937fd27426757c6a5dd2/org.eclipse.jdt-feature/pom.xml#L19

uh! I haven't thought about a different groupId!!! I'll try that ASAP!

[szarnekow]

of course we could also rename the two features and adopt all downstream stuff. and add it to the release notes

If it keeps things simple, I'd be in favor of fixing the names, making them more consistent, and announcing the change.

Most clients will consume the artifacts as plugins anyhow.

[LorenzoBettini]

@cdietrich it works! How can it be I haven't thought about that :) When the Jenkins build has finished we can retrigger the job for xtext-eclipse

[LorenzoBettini]

@cdietrich I retriggered the xtext-eclipse job and the TP is being resolved...

[LorenzoBettini]

@cdietrich I retriggered the xtext-eclipse job and the TP is being resolved...

https://ci.eclipse.org/xtext/job/xtext-eclipse/job/lb_2052_maven_tycho/ builds again!

[LorenzoBettini]

of course we could also rename the two features and adopt all downstream stuff. and add it to the release notes

If it keeps things simple, I'd be in favor of fixing the names, making them more consistent, and announcing the change.

Most clients will consume the artifacts as plugins anyhow.

So it looks like many Eclipse projects do have features with the same id of bundles (JDT and, IIRC, EMF), so we could also live with that. From the Maven point of view, it's just a matter of having a different groupId, which is a clean solution.

What I don't like of the current shape of Xtext features, like .xbase.lib feature, is that they include bundles XXX and XXX.source. This has also always sounded odd. For sources there are source features. I have already performed such a change in this PR. So now we have features and source features:

Are you OK with that?

The thing that might sounds strange (though again you find in several Eclipse projects) is that by default Tycho adds the string "Developer Resources" to the generated feature, e.g.,

Xtend Library Developer Resources

We can customize that. Do you prefer simply "Sources"? E.g.,

Xtend Library Sources

[cdietrich]

i somewhere read in my platform, tycho, cross-projects whatever mails that source features should be abandoned

[LorenzoBettini]

i somewhere read in my platform, tycho, cross-projects whatever mails that source features should be abandoned

@cdietrich you're referring to this maybe? https://www.eclipse.org/lists/cross-project-issues-dev/msg19553.html

If I understand it correctly Mikael means that we won't have to care about them because everything should be automatic. Another good motivation NOT to include source bundles in the main features.

[LorenzoBettini]

Concerning this previous note:

• xbase.lib.tests is not even an Eclipse plug-in project transformed in a plug-in project and added this to the TP:

      <location includeDependencyDepth="none" includeDependencyScopes="compile" includeSource="true" missingManifest="generate" type="Maven">
          <dependencies>
              <dependency>
                  <groupId>com.google.guava</groupId>
                  <artifactId>guava-testlib</artifactId>
                  <version>30.1-jre</version>
                  <type>jar</type>
              </dependency>
          </dependencies>
          <instructions><![CDATA[
Bundle-Name:           Bundle derived from maven artifact ${mvnGroupId}:${mvnArtifactId}:${mvnVersion}
version:               ${version_cleanup;${mvnVersion}}
Bundle-SymbolicName:   ${mvnGroupId}.${mvnArtifactId}
Bundle-Version:        ${version}
Import-Package:        *;resolution:=optional
Export-Package:        *;version="${version}";-noimport:=true
DynamicImport-Package: *
]]></instructions>
      </location>

because guava-testlib is not in Orbit. I don't know if Oomph will be able to handle that. The alternative is to turn it into a plain Maven project and take the dependency directly from Maven central. I can investigate.

That is not needed anymore: .xbase.lib.tests is now a pure Maven project so we take everything from Maven central! :)

[cdietrich]

@LorenzoBettini yes this is the message i meant i understand it: dont provide source features, just include the sources in the p2 repo and tycho will help with that

[cdietrich]

@LorenzoBettini please note:

we use https://oss.sonatype.org/content/repositories/snapshots/ for snapshots and https://s01.oss.sonatype.org/service/local/staging/deploy/maven2/ for milestones and releases

see also https://github.com/xtext/publishing/blob/aa97f2b592e9b64debde904ce262c35514a296a2/buildSrc/src/main/java/io/typefox/publishing/MavenUploadRepository.java#L13

[LorenzoBettini]

@LorenzoBettini please note:

we use https://oss.sonatype.org/content/repositories/snapshots/ for snapshots and https://s01.oss.sonatype.org/service/local/staging/deploy/maven2/ for milestones and releases

see also https://github.com/xtext/publishing/blob/aa97f2b592e9b64debde904ce262c35514a296a2/buildSrc/src/main/java/io/typefox/publishing/MavenUploadRepository.java#L13

Thanks! I forgot to ask. I'll update the URL for release.

However, the snapshot repository is the same I've used, and sonatype is happy :)

https://oss.sonatype.org/content/repositories/snapshots/org/eclipse/xtext/xtext-dev-bom/2.30.0-SNAPSHOT/

For the moment, I created a TEMP build job on JIRO, using a dedicated Jenkinsfile jenkins/Jenkinsfile-maven-deploy

[cdietrich]

we use 2 different repos currently. i dont know if and how they also sync snapshots among them

[LorenzoBettini]

we use 2 different repos currently. i dont know if and how they also sync snapshots among them

This is the standard section for deployment of Maven artifacts:

    <distributionManagement>
        <snapshotRepository>
            <id>ossrh</id>
            <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
        </snapshotRepository>
        <repository>
            <id>ossrh</id>
            <url>https://s01.oss.sonatype.org/service/local/staging/deploy/maven2/</url>
        </repository>
    </distributionManagement>

did you mean something else? Maven knows by itself if it's a snapshot (-SNAPSHOT in the version) or something that goes into the release repository (no -SNAPHSOT in the version).

[cdietrich]

yes this is what i meant

[LorenzoBettini]

Looks like signing when also running tests is likely to break tests:

14:24:40  [ERROR] class "org.eclipse.xtext.parsetree.reconstr.SerializationBug269362Test"'s signer information does not match signer information of other classes in the same package
14:24:40  [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked process
14:24:40  [ERROR] class "org.eclipse.xtext.parsetree.reconstr.SerializationBug269362Test"'s signer 

Since I'd avoid creating other Jenkins jobs, I'll execute tests in the first run, and then execute another run skipping tests doing the actual signing. After all, these are just experiments.

[LorenzoBettini]

mh... no, that doesn't work:

• if I run the build with signing and then the build with tests, I lose the archived p2 repo with signed contents
• if I do it the other way around, I lose the test results to archive

For these experiments, I have to create separate jobs just for archiving the p2 repository with signed contents. I'll use these latter p2 repositories to create later an aggregated p2 repository with signed contents in the umbrella.

This will take me some more time...

[cdietrich]

is the jar first built and then tested? or is target/classes used in tests?

[LorenzoBettini]

That depends on whether the tests are run with maven-surefire (before package) or with tycho-surefire (after package). In any case, it's not worthwhile to sign in the same build where we test. I thought I could do the two things together but that doesn't work.

[cdietrich]

so we have to run all builds twice. or can we maven sign separately? by calling deploy jar in a loop or so

[LorenzoBettini]

so we have to run all builds twice. or can we maven sign separately? by calling deploy jar in a loop or so

I'm creating build jobs unrelated from the multi-branch pipeline, with a dedicated Jenkinsfile used only for Maven deploy and p2 deploy. One for Maven deploy and one for p2 deploy. A pair for each Git repository.

This is just to test that sonatype is happy with our deployed Maven artifacts and to create p2 repositories with signed contents that I can aggregate into the main p2 repository.

These are all temporary workarounds due to the split Git repositories.

However, in the end, it's still better to have an on-demand job for Maven deploy and another one for p2 release.

[LorenzoBettini]

xtext-core Maven artifacts are accepted by sonatype, e.g., https://oss.sonatype.org/content/repositories/snapshots/org/eclipse/xtext/org.eclipse.xtext/2.30.0-SNAPSHOT/

temporary p2 repository for xtext-core with signed contents is here https://ci.eclipse.org/xtext/view/Xtext-release/job/TEMP-xtext-core-p2-deploy/lastSuccessfulBuild/artifact/build/p2-repository/

[cdietrich]

are the criteria for sonatype to accept snapshots and the releases the same. i vaguely remember there was some differences

[LorenzoBettini]

are the criteria for sonatype to accept snapshots and the releases the same. i vaguely remember there was some differences

Not that I know.

However, remember that I created the POMs starting from the one you used to generate from gradle, so the important information needed by sonatype are there.

[LorenzoBettini]

mh... no, that doesn't work:

* if I run the build with signing and then the build with tests, I lose the archived p2 repo with signed contents

* if I do it the other way around, I lose the test results to archive

For these experiments, I have to create separate jobs just for archiving the p2 repository with signed contents. I'll use these latter p2 repositories to create later an aggregated p2 repository with signed contents in the umbrella.

This will take me some more time...

I forgot to mention that xtext-lib (the first one I started from with these experiments) does not have such problems: we can sign and run tests in the same build, because the tests are in the same project containing main source folders.

[szarnekow]

So the problem is caused by split packages between production and test projects?

[LorenzoBettini]

So the problem is caused by split packages between production and test projects?

That's my understanding as well. However, as I said, from what I know, you sign only when releasing (snapshots, milestones, etc.), where you typically don't run tests.

[LorenzoBettini]

xtext-extras Maven artifacts are accepted by sonatype

temporary p2 repository https://ci.eclipse.org/xtext/view/Xtext-release/job/TEMP-xtext-extras-p2-deploy/lastSuccessfulBuild/artifact/build/p2-repository/

[cdietrich]

However, as I said, from what I know, you sign only when releasing (snapshots, milestones, etc.), where you typically don't run tests. we currently do

there is another "something that rings my mind"

• maven surefire vs tycho tests using maven surefire. but i cannot relate right now
• maybe phases test vs integ test

[LorenzoBettini]

However, as I said, from what I know, you sign only when releasing (snapshots, milestones, etc.), where you typically don't run tests. we currently do

there is another "something that rings my mind"

* maven surefire vs tycho tests using maven surefire. but i cannot relate right now

* maybe phases test vs integ test

Yes, I have already answered above :) https://github.com/eclipse/xtext-lib/pull/501#issuecomment-1406557481

what do you mean by "we currently do"?

[cdietrich]

https://github.com/eclipse/xtext/blob/master/Builds.md#preparing-milestones-and-releases we run tests during the release process. we run a normal build

[LorenzoBettini]

https://github.com/eclipse/xtext/blob/master/Builds.md#preparing-milestones-and-releases we run tests during the release process. we run a normal build

It's a waste of time running tests during release ;)

[cdietrich]

we have some utils that behave differently based on how version numbers look like .... and thus test fails/bugs in the past

https://github.com/eclipse/xtext-core/issues/1351

[LorenzoBettini]

we have some utils that behave differently based on how version numbers look like .... and thus test fails/bugs in the past

eclipse/xtext-core#1351

nothing prevents you from running the tests with what you want to release before you release, and then release skipping tests.

[cdietrich]

so we are back at running the build twice? so release process Jeninks File would be

mvn doStuff -PskipSign
mvn deploy -Psign -DskipTests

[szarnekow]

Not that a mvn deploy without clean does not necessarily run the build twice. If things go well, the produced JARs would just be picked up, signed and deployed, won't they?

[LorenzoBettini]

Not that a mvn deploy without clean does not necessarily run the build twice. If things go well, the produced JARs would just be picked up, signed and deployed, won't they?

yes, though we should skip also xtend-maven-plugin in that case.

In any case, I don't see a huge problem here: if you want to make sure the current version works you run the tests, when the tests are green, you run the build skipping the tests: it's not a big deal ;)

[cdietrich]

In any case, I don't see a huge problem here: if you want to make sure the current version works you run the tests, when the tests are green, you run the build skipping the tests: it's not a big deal ;)

my goal would be to have a one click solution (as we have it right now) and not starting 4+ jenkins jobs manually

[LorenzoBettini]

I haven't said that: when I talk about a build I mean a maven command. Nothing prevents us to run several maven commands one after the other in the same job.

The experiments I'm doing now are different because I first need to make sure each single repo does the right thing.

In general, though, it's better to have a clean, maintainable and simple configuration that might require two clicks (two jobs) than a complex over-enginereed one requiring one click ;)

For sure, it's better to have different jobs for everyday development builds and for releasing.

[LorenzoBettini]

so we have to run all builds twice. or can we maven sign separately? by calling deploy jar in a loop or so

I'm creating build jobs unrelated from the multi-branch pipeline, with a dedicated Jenkinsfile used only for Maven deploy and p2 deploy. One for Maven deploy and one for p2 deploy. A pair for each Git repository.

This is just to test that sonatype is happy with our deployed Maven artifacts and to create p2 repositories with signed contents that I can aggregate into the main p2 repository.

These are all temporary workarounds due to the split Git repositories.

However, in the end, it's still better to have an on-demand job for Maven deploy and another one for p2 release.

I think I found a solution that allows me to run the tests and later to sign in the same Jenkins job. The trick is

• don't build the p2 repository during the first run when we run the tests
• on the second run, don't run the tests, don't clean, but sing and create the p2 repo

This way, in the same Jenkins job I can record test results and archive the p2 repository with signed contents.

I'm not completely happy with this solution, but at least, it's the cleanest one that saves time, and it's temporary anyway. I'll apply that also to the other git repositories.

However, in the meantime, @merks , when you have time, could you please check that also this p2 repo is correct? https://ci.eclipse.org/xtext/job/xtext-core/job/lb_2052_maven_tycho/lastSuccessfulBuild/artifact/build/p2-repository/

[merks]

It still looks fine:

https://download.eclipse.org/oomph/archive/reports-extra/xtext-tycho/ci.eclipse.org/xtext/job/xtext-core/job/lb_2052_maven_tycho/lastSuccessfulBuild/artifact/build/p2-repository/index.html

[LorenzoBettini]

Looks like JIRO is overloaded (due to the new milestone release?).

@cdietrich, concerning the new milestone, I'll try to update my branches, but first, I'd like to conclude the "harmonization" of all the Git repositories to produce a complete p2 repository with signed contents.

[cdietrich]

@LorenzoBettini @szarnekow we need to discuss how we bring that live with 2.31.0.M1 anyway i wonder if we should PR the manifest/build.properties + rename/refactor changes to master independently and then with the real stuff PR the pom/nature etc changes

[LorenzoBettini]

Some updates:

• Xtend p2 repository with signed contents is now available from here https://ci.eclipse.org/xtext/job/xtext-xtend/job/lb_2052_maven_tycho/lastSuccessfulBuild/artifact/build/p2-repository/
• Xtend Maven artifacts (snapshots) are accepted by sonatype
• xtext-eclipse p2 repository with signed contents is now available from here: https://ci.eclipse.org/xtext/job/xtext-eclipse/job/lb_2052_maven_tycho/lastSuccessfulBuild/artifact/build/p2-repository/

@cdietrich just as a confirmation: excluding xtend-maven-plugin, which is taken with the current latest released versions, all the other Maven artifacts dependencies concerning our bundles, even across Git repositories, should be taken with -SNAPSHOT version, right? By default they would be taken from sonatype snapshots. In the build are taken from latest Jenkins Maven repositories, is that correct?

That is, <xtext-current-release-version>2.30.0.M1</xtext-current-release-version> should be in all Git repositories <xtext-current-release-version>${project.version}</xtext-current-release-version>. Am I correct?