Create a build to promote a release

ruspl-afed commented 1 year ago

We need a release candidate build that upload binaries in a form of p2 repository that can be installed to an Eclipse product (let's say EPP for Committers) without troubles.

ruspl-afed commented 1 year ago

I've created a 3.26.0 RC1 build https://download.eclipse.org/mylyn/stage/milestone/3.26.0-RC1/ Any comments are welcome @BeckerFrank @wimjongman

BeckerFrank commented 1 year ago

I did a first install in a fresh 2023-03 Commiter Eclipse installation and a first test shows no problems. One thing is that the bundles are not signed so we get the trust dialog. Can we add signatures for the bundles? Will continue with some more tests.

ruspl-afed commented 1 year ago

One thing is that the bundles are not signed so we get the trust dialog.

Thank you for reporting @BeckerFrank I'll have a look

ruspl-afed commented 1 year ago

Build was fully recreated, here is the result https://download.eclipse.org/mylyn/stage/latest/

BeckerFrank commented 1 year ago

Maybe we can find what we need to change so that the last 5 org.eclipse.mylyn.* bundles are trusted

ruspl-afed commented 1 year ago

I've downloaded org.eclipse.mylyn_feature_3.26.0.N20230416-1636.jar to examine its content and I can see ECLIPSE_.RSA and ECLIPSE_.SF files inside META-INF directory.

@merks what else should be done to make installation process happy?

merks commented 1 year ago

This issue of generating OSGi artifacts from Maven artifacts is of great concern to me:

https://github.com/eclipse-mylyn/org.eclipse.mylyn/issues/103

According to this report there are only 4 unsigned artifacts:

https://download.eclipse.org/oomph/archive/reports-extra/mylyn/download.eclipse.org/mylyn/stage/latest/index.html

If you're going to pull dependencies directly from Maven you will need to enable PGP signing in your builds. This has been discussed repeatedly on cross projects for years...

Probably the above issue with org.eclipse.mylyn.* artifacts is the result of having older unsigned versions in the bundle pool that have the exact same artifact key as the newer signed versions. This is why it's never good to allow unsigned artifacts to escape into the wild, i.e., available on download.eclipse.org, because they are impossible to stamp them out except to provide at least a new qualifier version.

merks commented 1 year ago

Which target are you using for your builds? It's easier to help if I know which target file needs attention...

ruspl-afed commented 1 year ago

This is what we use: https://github.com/eclipse-mylyn/org.eclipse.mylyn/blob/main/org.eclipse.mylyn-target/mylyn-e4.26.target

I can try to redirect com.sun/javax/jakarta back to orbit, but we do not have enough Lucene libs there

merks commented 1 year ago

Yes, I see these all fail:

And I don't see Orbit versions of all of these...

ruspl-afed commented 1 year ago

Yes, we don't have them in Orbit. And even if I'll push Mylyn team to degrade back to Lucene 9.4.2 we still have no Lucene Query Parser 9.4.2 in Orbit. What should I do to propose Lucene (and Mockito) for the "centralized" location?

merks commented 1 year ago

I've already been prototyping producing generated direct-from-maven from m2e targets so I have something working locally already:

I've been using this recipe where the imports are not optional:

          <instructions><![CDATA[
Bundle-Name:           Bundle derived from maven artifact ${mvnGroupId}:${mvnArtifactId}:${mvnVersion}
version:               ${version_cleanup;${mvnVersion}}
Bundle-SymbolicName:   ${mvnGroupId}.${mvnArtifactId}
Bundle-Version:        ${version}
Import-Package:        *
Export-Package:        *;version="${version}";-noimport:=true
DynamicImport-Package: *
]]></instructions>

I'm very concerned though that you are generating a release, not just a milestone so any mistakes now will be etched in stone. Also, once this IU is published and available, I cannot change the recipe without some version change, and I'm not 100% sure this is the exact best recipe. See this comment about that concern:

https://github.com/merks/simrel-maven/issues/4#issuecomment-1510740227

In Orbit, each bundle had its own specialized BND recipe and qualifier versions were created so if the recipe changed, there would be a new version. Also, it's not entirely clear if we should jar sign these things or PGP sign them. The point of PGP signing was that we didn't want to modify the original artifact as provided by Maven, but that argument doesn't apply when we create a modified artifact...

We'd certainly all be better off if there was a bit more runway here for takeoff...

merks commented 1 year ago

BTW, I am assuming all the other m2e-location dependencies you can get from here:

https://download.eclipse.org/oomph/simrel-orbit/milestone/S202304261239/index.html

merks commented 1 year ago

I'm also investigating the direct-from-maven dependencies that are OSGi bundles. These versions are not the latest:

Minor Updates

com.sun.xml.bind
- jaxb-impl 2.3.3 < 2.3.8
org.mockito
- mockito-inline 4.8.1 < 4.11.0

I don't have a good sense of whether all these requires are actually needed

Major Updates

com.sun.xml.bind
- jaxb-impl 2.3.3 < 4.0.2
jakarta.xml.bind
- jakarta.xml.bind-api 2.3.3 < 4.0.0
org.mockito
- mockito-inline 4.8.1 < 5.2.0

Content

com.sun.activation
- jakarta.activation 2.0.1
com.sun.xml.bind
- jaxb-impl 2.3.3
jakarta.xml.bind
- jakarta.xml.bind-api 2.3.3
org.apache.lucene
- lucene-analysis-common 9.5.0
- lucene-core 9.5.0
- lucene-queryparser 9.5.0
org.eclipse.mylyn.github
- org.eclipse.egit.github.core 6.1.0.202203080745-r
org.glassfish.hk2
- osgi-resource-locator 2.4.0
org.mockito
- mockito-inline 4.8.1

merks commented 1 year ago

I updated the available minor updates and now I look at what is transitively pulled in:

If I change to just includeDependencyScopes="compile" there are fewer things:

If I change includeDependencyDepth="none" there are of course only the direct things:

But even in this case, there are no errors in the workspace and the Tycho build gets almost to the end:

[INFO] org.eclipse.mylyn.tests 3.26.0-SNAPSHOT ............ FAILURE [ 1.512 s] [INFO] Mylyn for Eclipse 4.x 3.26.0-SNAPSHOT .............. SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:32 min [INFO] Finished at: 2023-05-01T13:13:46+02:00 [INFO] ------------------------------------------------------------------------ [ERROR]

I skipped the tests because they abort probably because I'm on Windows, but who knows...

Even commenting some of them out does not result in errors in the workspace and the build gets to the same point:

I'm really uncomfortable not understanding what's here that's strictly required and even with a recipe that does this where it's valid for all packages to simply be missing even though it's highly unlikely to be true:

Import-Package:        *;resolution:=optional

ruspl-afed commented 1 year ago

jaxb-impl is used on runtime, it will not fail during compilation. perhaps some tests could fail, but I do not know for sure.

I do not insist on recipe directives, it could have got there from another "already working" eclipse project without too much understanding what exactly it does.

could you please formulate your changes as a PR, the most interesting parts are 1) lucene 2) mockito

for jaxb/activation I hope to use either Orbit or what Platform is used

For EGit we already have work in progress here

merks commented 1 year ago

That sounds like a good plan.

ruspl-afed commented 1 year ago

I'm very concerned though that you are generating a release, not just a milestone so any mistakes now will be etched in stone.

What is published on https://download.eclipse.org/mylyn/stage/latest/ is 3.26.0 RC1, but I don't think that exact milestone is very important for external consumers during Mylyn release cycle. Actually, I started from some deep structure of integration/milestone/release builds, but then I quickly found that the thing we need now is one stable build to be accepted by consumers and promoted to release grade. And stage/latest looks good enough for it.

I plan to update (technically, it is not "update" but "replace") the content at stage/latest URL as many times as we need before promoting it to another release URL.

I'm open to learn your thoughts regarding how to organize the structure and management of update URLs in a better way @merks

merks commented 1 year ago

There seem to be many ways to do this process. Updating a repo in place is not really a best practice, but many people do that because it's easy. A release repository ideally has metadata about mirror locations, but most don't because it's not easy.

I won't tell you how you should do it, but I can tell you that when doing this for going on 10 projects that one thinks, how best to automate this? So when I developed JustJ I took EMF's p2 update site builder, generalized it, and used it for JustJ. Eventually I replaced EMF's update site builder with the resuable one from JustJ. Then I reused it for a bunch of other projects so you'll see they all look quite similar:

In all cases, one simply does a build and decides is it a nightly build, a milestone build, or a release build. The nightly builds are limited in number so older ones are deleted. Milestones are deleted when a new version is started so they too are limited in number. A release build simply mirrors the latest milestone build; no new artifacts are produced so you release what you tested and used... Mirror URLs are added by the framework. Version numbers are determined by a designated IUs' feature, e.g., a root "SDK" feature.

I've offered to set this up for another project, but it seems that folks like to do things themselves; there's no point in doing that work if people don't like it and would rather do their own thing.

So my comment here wasn't so much about the location of the repo but about the duration/lifetime of the repo; you should never delete or replace releases, so you can't fix them.

wimjongman commented 1 year ago

I saw your beautiful update sites, Ed, but did not realize that this was already generalized. Is this something that could go in Tycho? @laeubi

merks commented 1 year ago

Here’s an example of how it’s integrated into a Tycho build.

https://github.com/eclipse-cbi/targetplatform-dsl/blob/01c3a1c1cdbf511d804cf08caa76e2364a02e2ee/promotion/pom.xml#L30

Here’s how most of the builds are configured:

https://github.com/eclipse-cbi/targetplatform-dsl/blob/01c3a1c1cdbf511d804cf08caa76e2364a02e2ee/Jenkinsfile#LL1C1-L1C1

Lots of commented out stuff in that one…

laeubi commented 1 year ago

I saw your beautiful update sites, Ed, but did not realize that this was already generalized. Is this something that could go in Tycho?

@wimjongman if it is of general use it can be part of Tycho (assuming there are no IP issues), in general it helps:

have an issue describing the new feature](https://github.com/eclipse-tycho/tycho/issues) and what it is supposed to solve
provide a PR with an integration-test to demonstrate the issue/feature at best adding some documentation and have a release note entry.

As the tycho-eclipserun-plugin is used here, I'd like to point t a very cool new feature (for developers not users) that allows to fire up and eclipse-application inside a mojo and execute code inside that (embedded but still isolated) application: