search

eclipse / xtext

Eclipse Xtext™ is a language development framework
http://www.eclipse.org/Xtext
Eclipse Public License 2.0
767 stars 321 forks source link

Target Platform for 3rd parties: Orbit vs Maven #2133

Open cdietrich opened 1 year ago

cdietrich commented 1 year ago

Target Platform for 3rd parties: Orbit vs Maven what happens if orbit is discontinued

problems / knowledge gaps for maven dependencies in target

cdietrich commented 1 year ago

cc @merks @kthoms this collects some of the knowledge gaps and problems i see with "using maven artifacts in target"

merks commented 1 year ago

Note that Oomph now does support including a normal *.target into a Targlet container and that target can contain Maven dependencies. This work was funding by https://www.diamond.ac.uk/

The approach I described in the forum will allow Maven dependencies in a centrally managed .target file (along with distributed managed additional .target files from SimRel projects) to be used to generate a normal p2 repository that can be consumed in the normal way exactly as you are currently consuming from Orbit. Mind you, the artifacts are PGP-signed, not jar-signed.

Will that address all your concerns?

cdietrich commented 1 year ago

@merks with orbit i had that impression that projects are "encouraged" to repackage their dependencies into their own repo. now when i will repackage the pgp signed stuff from the "new pgp signed orbit", will these become pgp signed automatically (just copy over the original signing info) or will they be unsigned and thus we have to care about signing again (point 4 in my list)

will do some experiments on this

merks commented 1 year ago

The PGP key and signature are properties on the artifact, e.g., 

    <artifact classifier='osgi.bundle' id='com.ibm.icu' version='72.1.0'>
      <properties size='14'>
        <property name='maven-groupId' value='com.ibm.icu'/>
        <property name='maven-artifactId' value='icu4j'/>
        <property name='maven-version' value='72.1'/>
        <property name='maven-repository' value='eclipse.maven.central.mirror'/>
        <property name='maven-type' value='jar'/>
        <property name='download.checksum.sha-1' value='bc9057df4b5efddf7f6d1880bf7f3399f4ce5633'/>
        <property name='download.size' value='14386028'/>
        <property name='artifact.size' value='14386028'/>
        <property name='download.md5' value='b98735702f497dd482638bedc16a1c09'/>
        <property name='download.checksum.sha-512' value='48130eb346a5bb5bdaff867e5231808a3cd03d1876d8e6b7c501336df6992e912f7c53456dc72b673ad3272f3b54b414343eea8a1118d01d0e517403cab3e324'/>
        <property name='download.checksum.md5' value='b98735702f497dd482638bedc16a1c09'/>
        <property name='download.checksum.sha-256' value='3df572b240a68d13b5cd778ad2393e885d26411434cd8f098ac5987ea2e64ce3'/>
        <property name='pgp.signatures' value='-----BEGIN PGP SIGNATURE-----&#10;&#10;iQIzBAABCAAdFiEEnjBEBxt1jry35FZzcA5PObwFNksFAmO2D+QACgkQcA5PObwF&#10;Nkty6w/6A2ZeB18CSP7dVTSb0DFOmzA0Q3W/vhoAWm32unUzBDSUA7gloXp6KdqJ&#10;xDKn/28AYfGuFGcZBSN/FSVDWyXQcJD3/Idix4xjRPJ45HibzXej4Pfuomr3Fdi3&#10;TMeedRWfVJeXX1nTht5Wbx7FCU3TvG2f7jJcJnZc0x3CU8KEKrFNP/QAmcEvo2vZ&#10;6kuLSKBQ5093+UCNpu1YaGZA3nJm10j3AZ3pDzyB0bvb2tZ+ATAoNZ37NsgP98wm&#10;5f1l5mgwhG73yLjk1KLlujjUKvY552gRaCju3ceNKkfGtCSyLiZDjf2qoTn5l5mv&#10;IJJqvfHZVi3iUQa/8Jf/E64A1W+2wMKyLBHnOq4gKPrw2b3dLp+AXi+Ub3178q+a&#10;fwyULsqcny5RL6QpjynyoJJlB9q8PZmcajIY/rrvHi4peO3psm+wgqUEeaJGmf1Y&#10;BkIyIJ8R3xoBK1Hv7xVvl9wmKa12znC2CKHPda5jjGMMc3z6zQSngZhl73PQ/4P6&#10;KVjvhQEpJq+w0Zk5czZ9iJHXCqtjyCamGDvg5cLQt/Kw6Pkj1Pjgi01XQ3mJqD3B&#10;jormwEBrtsctSluJ02WGznbnZbE8r70UF/+T76qYHW303LUoDjTYDTyllxISZ01T&#10;+nYTmDArweB3Bb/whbkfMufMjHnH0oR+Pu8WAdEsAes0eT+tSEQ=&#10;=bbNj&#10;-----END PGP SIGNATURE-----'/>
        <property name='pgp.publicKeys' value='-----BEGIN PGP MESSAGE-----&#10;&#10;uQINBFhaXPsBEAC3bR7f5euHbpIDDTuFYHPI0+S5X0DhuqcGBUL2HSFhWMwIlfsA&#10;aO+pt7GyfXLUkTmzugwmwO+sOW2QmwEZQcK2z3BrcjytZophZ9AUajbAjnadSH6U&#10;XCMmfExVVnaYSfl/+Uub42szQE/r3gCRIz6M6clVVAjpFv4G/mumfQUV/XzLoUEY&#10;XTgwTokFJ97R+hDbHvBEBrUT8M6zHP5DhN3EBug3qb6wZVOa/+HEX3M+7k4jVT/p&#10;pNumw0acg0DDoSNQ13VsRV6sV0XE4zr3Zfs84f8xCgXpEMs4U6DZGqs3iJVVtbRf&#10;0oL0fgcxNgRrmbCrBfbXYfrS4u+fJ0vB+Wrflv9eNA3i6TtVL6uYpZy9uO2B1olK&#10;VzfEhsgB3QrULB4jVHZjIXGe4ILn45ndMtAeY4M91wyobgG99Xl+1vPHrxV0+2zR&#10;P66J3puyxiKE2B7gd7hib54CB3lYyrG1S+K1kZGCI1IFKCnqmTJXY0tKoLAASS3v&#10;tDcknXenzR5RVSpWTDuxtusekfL0Bw8pCBoz9L4Hex8Q1j//D5CZlqcg1NKFfmBZ&#10;7ta9PTuJcpOsz/LaPG/0VHYt/QAv5o4eeZESl7iZyM4/0NFh2s/rq0R8Z9yVSSkI&#10;vvO8d8XGZ65NTm3T4NFuEihn+AEm+zg4KiGdYBEZvs8QQoW9e1+MMN8xnwARAQAB&#10;iQREBBgBCAAPAhsCBQJhuzR9BQkSxtkCAinBXSAEGQEIAAYFAlhaXPsACgkQcA5P&#10;ObwFNkunSw//SRR1tGS1pDj2jonLpR0wPilCphS6ANv895yvlg6rHG4nKi4hQ0Jz&#10;ZxhGCwkgxEkRaKiyLfEiTihETkF161AqLPhyvE8LuQ1AG+A+tUnR8/T3gKE8t/m2&#10;/UtScZwN1QEQVc/uG7MTrbZ2ngXfH65k3fzhjy95AnJHAswu2vic1hzDi77HlQpN&#10;0O3adJuU/jfdu1RxNE0MRt8MFEjsTFwSBVm6lDxgcZV+qjRLGQznTyLF5/AyCI7Z&#10;4z9xHZPKFq1eHzqevifNiqfb8KX22sHKOSdnVBzBq/UxbT5jIbNSRhD91FjtZD7Z&#10;6wi3POsB/9RWZBldCov4ZEajmxFzxpx4RAqYOSIkEor9ZtRGbZuWvTie4vFIur7T&#10;f543mE6nxKcggExNp4MTyOd1scMc9oyczH561OTdHOCYEyoCwpG9N2Hb1/MDnWSi&#10;HKG451CvdrE5FHcPZKjp/nHUcRw/WQC3bgj6ScAay64EKC5S9tW+Wp85Oyyvj+M7&#10;lBzOxp19nESpfC++fzBAQPMxtD8EvrZTxqFSJxMOH9bhzB8+MFt08tmYb5SwoYi4&#10;C8JJ+wZgNetJKK+j07fvyMUChH/SbkCVszMiiSEjHA2Kk0LMVYKS/OLJU7i7tZXV&#10;aJ078QEeTDy5hSzsutd+orlFkR9+mgr1HUh0UgYlofTfEi7bLDeSr0cJELbTq5vM&#10;ZBKCicIP/irazYBVKw0SluhHtjzRcs5WIdH5bVPsEE87+iUc4daONWdVIhLdokxt&#10;OWlrEmZFLKqq9Z8fzvlf5LAQMOBkMAkl0z2ej4KG7zrjWyqDgysEI2WBlqTAFSeL&#10;+89Kc9BzJE9heYW8EfpXbNfOnKnAYWsbhcomSxVQ/jBIuyLBg/0gYKpBNx8HC6v9&#10;xNH0Ja+wM/7w3JC1aIwMYJn1yF2ykUYS+BoTCU7TA8r43pHg4I4Fz+Y2P5RLk+RJ&#10;I4kJezDNiJOpIcr/nKTPxMGUzMtWlGyAJ7LkyOZCtQXhtXwaT8grjtHzlwlGrpgD&#10;Rtf7wWjzEWeaQSegTFM9Mid+09kCp0PkJvveg8wJCuoVboNOto0O5rQsUczjXxiW&#10;kXYlHGeQL4rWc1zP7F1n4DEwDbVZC7jOn/80l3x4LcKuhc86gP4L5HKbdjn5GcQ0&#10;3RVLl1WVTQCdpr0+am28hl9XpyHdlWwSEmqqoUnjGv5B8RClocBRS4ECPPZCVSBl&#10;yK8eDgRww9Fu1EFq4xkq5fGj4YUOAIm756iW41NQ3VnPYbom/J27iFFN8+h92CSb&#10;KAqhmRwQh+GGo0eGCXmPHyQ/KCHTvnTZCFBUvabm3rVNFaDO+RvmwPwNCRz0DYzG&#10;paeMOGo4nMMGbzdhgfJ/X5Ed1/Mqz8egHhGIO94ebKEN5ZtJjAOK&#10;=xtSl&#10;-----END PGP MESSAGE-----'/>
      </properties>
    </artifact>

The underlying p2 infrastructure used by Tycho (a Tycho version that uses at least the 2022-03 release before which the PGP support was broken) and for installing or mirroring will transfer these properties from the source to the target. So yes, I do expect PGP signatures and keys to "follow" the artifact and that definitely generally works, e.g., the SimRel aggregator does nothing special to transfer the PGP key and signature properties, relying on p2 to do that....

merks commented 1 year ago

And yes, the goal here is to provide something such that downstream consumers can avoid needing to do PGP signing and such that it's easy to provide downstream consumers with additional dependencies upon request, with an overall goal to be easier/cheaper to maintain than is currently the case for Orbit.

cdietrich commented 1 year ago

is there an easy way to check if a p2 repo has valid pgp signature for a certain artifact?

merks commented 1 year ago

Open the artifacts.jar/artifacts.xml.xz and the artifacts.xml inside that. Find the <artifact classifier='osgi.bundle' id='*' version='*'> and check to see that it has pgp.signatures and pgp.publicKeys properties. If those properties are present, it's safe to assume they came from the originating p2 repository and are valid.

LorenzoBettini commented 1 year ago

@cdietrich, what I'm willing to try to do is, as I proposed in the past, switch entirely to Maven/Tycho. In my experience, that should simplify lots of our building parts, not to mention that the p2 repositories generated will not have the problems we have experienced lately. Also, my projects (both the official Eclipse projects and other projects I maintain) have an involved build structure, which I can easily manage with Maven/Tycho. I'm willing to do that as long as there's interest from the Xtext committers.

cdietrich commented 1 year ago

@LorenzoBettini i am not sticking to any technology here as long as we are still compatibly consumable from pure maven and gradle builds and thus wont break these clients.

@szarnekow i assume you would be fine with pure tycho builds as well.

merks commented 1 year ago

I can also help with pure Tycho builds having setup and maintained a large number of them over the past years.

LorenzoBettini commented 1 year ago

@LorenzoBettini i am not sticking to any technology here as long as we are still compatibly consumable from pure maven and gradle builds and thus wont break these clients.

Yes, my artifacts are available from p2 repositories and Maven central. (everything signed, of course). I typically deploy to Maven central artifacts without the build qualifier (e.g., 2.12.1), while on p2 repositories, I leave the full build qualifier (e.g., 2.12.1.v202212...blah blah). But that's just my design choice.

cdietrich commented 1 year ago

regardning repackaging looks good with tycho 2.7.5 and tycho 3.0.1: https://github.com/cdietrich/repackage-pgp-signed-stuff-experiments

szarnekow commented 1 year ago

@szarnekow i assume you would be fine with pure tycho builds as well.

Absolutely.

I'd go even a step further:: If things get simpler by merging everything into a single repo, I'm willing to do that, too.

LorenzoBettini commented 1 year ago

I'd go even a step further:: If things get simpler by merging everything into a single repo, I'm willing to do that, too.

I didn't dare to say that, but I'd love to try that one as well :)

merks commented 1 year ago

FYI, I merged the EMF and XSD repositories, preserving the history. It wasn't so hard...

cdietrich commented 1 year ago

the question is, how to stack the stuff on each other, first monorepo, then adapt existing builds then tycho build?

LorenzoBettini commented 1 year ago

Yes, that could be a possible workflow. Alternatively, I'd start porting to Maven/Tycho the "core" and then go on. However, having everything in the same repo will likely make the port to Maven/Tycho easier due to some inter-dependencies. On the other hand, the merge to a single repo might be more disruptive... just a few random thoughts.

@merks, how have you implemented merging? My idea was something like that:

merks commented 1 year ago

I needed some help at the end from the webmaster:

https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/2444

As mentioned there, I followed these simple instructions which sounds like what you are describing:

https://jeffkreeftmeijer.com/git-combine/

It was way simpler than I though it would be...

Also this helped simplify the build so I am able to use a multi-branch pipeline:

https://ci.eclipse.org/emf/job/build/

I also got rid of EMF's specialized site promoter/generator and reused the JustJ-provided one instead.

cdietrich commented 1 year ago

i guess we can start experimenting with core, then we might see what makes problems and what works

LorenzoBettini commented 1 year ago

i guess we can start experimenting with core, then we might see what makes problems and what works

do you mean to experiment with Maven/Tycho with "core" or to try to merge "core" with another Git repository?

cdietrich commented 1 year ago

experiment with tycho

LorenzoBettini commented 1 year ago

Note that Oomph now does support including a normal *.target into a Targlet container and that target can contain Maven dependencies. This work was funding by https://www.diamond.ac.uk/

@merks this means however that Oomph cannot yet support Maven coordinates (as we do in the .target files) directly, is that right?

The approach I described in the forum will allow Maven dependencies in a centrally managed .target file (along with distributed managed additional .target files from SimRel projects) to be used to generate a normal p2 repository that can be consumed in the normal way exactly as you are currently consuming from Orbit. Mind you, the artifacts are PGP-signed, not jar-signed.

Could you please point me to the forum post you're mentioning?

In general, would you suggest switching to target files in Oomph as well?

merks commented 1 year ago

Sorry I thought I had added a link to the Bugzilla that has a PDF attachment with documentation

https://bugs.eclipse.org/bugs/show_bug.cgi?id=580054 https://bugs.eclipse.org/bugs/attachment.cgi?id=288551

WindowBuilder uses it:

https://github.com/eclipse/windowbuilder/blob/master/setups/WindowBuilder.setup

This target

https://github.com/eclipse/windowbuilder/blob/master/target-platform/wb.target

Includes this target:

https://github.com/eclipse/windowbuilder/blob/master/target-platform/mvn/wb-mvn.target

And that one is also used by Oomph's targlets.

I had to add this task

https://github.com/eclipse/windowbuilder/blob/06ff485102ce83f83082dc2d9ef0e6720286afc5/setups/WindowBuilder.setup#L117-L121

So that the wb-mvn.target is available in the workspace for use here:

https://github.com/eclipse/windowbuilder/blob/06ff485102ce83f83082dc2d9ef0e6720286afc5/setups/WindowBuilder.setup#L141

I'm hoping get what I described on cross-projects working over the weekend, in which case I could just add what you need so that it's in a regular p2 repository. Is there something specifically that you needed that's not also used directly by the platform?

LorenzoBettini commented 1 year ago

@merks thanks for the answer and sorry for my delay. While porting Xtext projects to Maven/Tycho there were a few projects that needed dependencies not available from Orbit and as a first attempt, I used Maven dependencies in the .target file. Later, I realized that for those specific (test) projects, I could go completely Maven (no PDE), so dependencies are taken directly from Maven Central. So for the moment the .target only contains p2 units.

merks commented 1 year ago

This this available, I think all is prepared:

https://download.eclipse.org/oomph/simrel-orbit/

I'm not sure what's left to do for this issue....