eclipse-repository pulls dependencies from maven instead of p2

jonahgraham commented 1 year ago

Discussed in https://github.com/eclipse-tycho/tycho/discussions/2732

I have converted this discussion to a bug as Tycho is requiring the artifact to be in an accessible Maven repository.

For example Orbit makes an Ant bundle like this. Essentially this is a bundle published to a private repo instead of Maven Central.

Now for Tycho to consume this bundle I need to have that version of the bundle available in both the p2 repo and included in a maven repository.

Without the duplicate published bundle I get this error (in a clean ~/.m2/repository):

19:33:57 [ERROR] Failed to execute goal on project epp.package.dsl: Could not resolve dependencies for project org.eclipse.epp:epp.package.dsl:eclipse-repository:4.30.0-SNAPSHOT: Could not find artifact org.eclipse.orbit:ant:jar:1.10.14 in eclipse.maven.central.mirror (https://repo.eclipse.org/content/repositories/maven_central/) -> [Help 1]
19:33:57 org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project epp.package.dsl: Could not resolve dependencies for project org.eclipse.epp:epp.package.dsl:eclipse-repository:4.30.0-SNAPSHOT: Could not find artifact org.eclipse.orbit:ant:jar:1.10.14 in eclipse.maven.central.mirror (https://repo.eclipse.org/content/repositories/maven_central/)
[...]
19:33:57 Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact org.eclipse.orbit:ant:jar:1.10.14 in eclipse.maven.central.mirror (https://repo.eclipse.org/content/repositories/maven_central/)
19:33:57     at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed (ArtifactTransportListener.java:48)
19:33:57     at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:369)
19:33:57     at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run (RunnableErrorForwarder.java:75)
19:33:57     at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
19:33:57     at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
19:33:57     at java.lang.Thread.run (Thread.java:833)

And this slightly more complete error message if I had previously downloaded the bundle:

[INFO] Artifact org.eclipse.orbit:ant:pom:1.10.14 is present in the local repository, but cached from a remote repository ID that is unavailable in current build context, verifying that is downloadable from [eclipse-simultaneous-release (https://download.eclipse.org/staging/2023-12/, p2, disabled), license-feature (https://download.eclipse.org/cbi/updates/license/, p2, disabled), justj (https://download.eclipse.org/justj/jres/17/updates/release/latest/, p2, disabled), justj-epp (https://download.eclipse.org/justj/epp/milestone/latest/, p2, disabled), central (https://repo.maven.apache.org/maven2, default, releases)]
[WARNING] The POM for org.eclipse.orbit:ant:jar:1.10.14 is missing, no dependency information available
[INFO] Artifact org.eclipse.orbit:ant:jar:1.10.14 is present in the local repository, but cached from a remote repository ID that is unavailable in current build context, verifying that is downloadable from [eclipse-simultaneous-release (https://download.eclipse.org/staging/2023-12/, p2, disabled), license-feature (https://download.eclipse.org/cbi/updates/license/, p2, disabled), justj (https://download.eclipse.org/justj/jres/17/updates/release/latest/, p2, disabled), justj-epp (https://download.eclipse.org/justj/epp/milestone/latest/, p2, disabled), central (https://repo.maven.apache.org/maven2, default, releases)]
[...]
[ERROR] Failed to execute goal on project epp.package.rcp: Could not resolve dependencies for project org.eclipse.epp:epp.package.rcp:eclipse-repository:4.30.0-SNAPSHOT: The following artifacts could not be resolved: org.eclipse.orbit:ant:jar:1.10.14 (present, but unavailable): org.eclipse.orbit:ant:jar:1.10.14 was not found in https://repo.maven.apache.org/maven2 during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]

Reported and worked around in https://github.com/eclipse-packaging/packages/issues/66

merks commented 1 year ago

Yes, I encountered this with EMF's build yesterday and was pulling my hair out about the cause...

I can (and should?) probably fix/change this in Orbit if I promoted to a Maven coordinate that also exists on Maven central

https://github.com/eclipse-orbit/orbit-simrel/blob/f6ea491f4ada5d6d6750d8118bf40d9802993376/maven-bnd/tp/MavenBND.target#L1193-L1210

I wasn't sure if maybe it's caused by the way the I-Build is published to snapshots, but that didn't appear to be the case...

laeubi commented 1 year ago

Please use latest Maven (3.9.x instead of 3.8.6) and latest Tycho (4.0.3 instead of 3.0.5)
If problem persist, provide an integration-test to demonstrate the issue

Essentially this is a bundle published to a private repo instead of Maven Central.

Then you need to add https://repo.eclipse.org/content/repositories/orbit-approved-artifacts as a repository to your build (or simply publish to maven central where most of the users would expect things)

Beside that, the title is wrong, Tycho is never pulling anything from maven instead of P2, as explained Tycho only injects the provided maven coordinates into the maven model and maven try to resolve them from the repositories you have configured (what are missing your custom repository).

merks commented 1 year ago

I assume Tycho is looking at these p2 metadata properties and is trying to use them to resolve dependencies:

  <properties size='4'>
    <property name='org.eclipse.equinox.p2.name' value='Apache Ant'/>
    <property name='maven-wrapped-groupId' value='org.eclipse.orbit'/>
    <property name='maven-wrapped-artifactId' value='ant'/>
    <property name='maven-wrapped-version' value='1.10.14'/>
  </properties>

Is that correct? It looks like the only place where it could have gotten the coordinate information, hence my assumption. It seems kind of pointless (wasted network bandwidth) to check these coordinates when all dependencies are fully specified in the target platform.

If I removed such properties, would Tycho/Maven no longer do such a check?

In any case I'm very pleased to report that Tycho 4.0.3 does not have this problem! It still complains about it:

[INFO] Resolving dependencies of MavenProject: org.eclipse.emf:org.eclipse.emf.test.core:2.29.0-SNAPSHOT @ D:\Users\merks\emf-2.36\git\org.eclipse.emf\tests\org.eclipse.emf.test.core\pom.xml
[WARNING] Mapping P2 > Maven Coordinates failed: The following artifacts could not be resolved: org.eclipse.orbit:ant:jar:1.10.14 (absent): Could not find artifact org.eclipse.orbit:ant:jar:1.10.14 in central-id (https://repo1.maven.org/maven2)

But the build finishes without errors. I still need to verify this on a build server.

merks commented 1 year ago

I can confirm that Tycho 4.0.3 works well on a ci instance. Furthermore, for EMF I do builds against ancient target platforms, e.g., kepler, and those builds work with Tycho 4.0.3. So all good news. Kudos to @laeubi for providing quality build infrastructure.

That just leaves me with the open question of what I might do to avoid this problem in general? In particular, is removing the maven-wrapped-* properties from the p2 metadata sufficient to avoid Tycho/Maven trying to resolve such a coordinate? Especially for all the cases where an OSGi bundle is created from a non-OSGi maven artifact, it really doesn't seem to make sense to try to resolve the original coordinate to an actual Maven artifact...

laeubi commented 1 year ago

I assume Tycho is looking at these p2 metadata properties and is trying to use them to resolve dependencies

Tycho does not resolve the dependencies from there, Tycho resolves dependencies from P2. What Tycho does is restoring the maven information and pass that to maven so other tools can use that (e.g the licence-tool, other maven mojos and so on...)

It seems kind of pointless (wasted network bandwidth) to check these coordinates when all dependencies are fully specified in the target platform

It seems kind of pointless (wasted network bandwidth and server storage) to publish dependencies to an additional P2 site all dependencies are fully available from maven already ;-)

If I removed such properties, would Tycho/Maven no longer do such a check?

If you remove these properties Tycho will not be able to perform the injection of maven coordinates and will fall back to a synthetic non portable form, you can see the difference when calling mvn dependency:tree on your project.

laeubi commented 1 year ago

I can confirm that Tycho 4.0.3 works well on a ci instance. Furthermore, for EMF I do builds against ancient target platforms, e.g., kepler, and those builds work with Tycho 4.0.3

Usually older releases work fine with never Tycho but for newer releases we can't guarantee this obviously because of new features not understood by old platforms (e.g. PGP signatures, changed bundle names, ...).

it really doesn't seem to make sense to try to resolve the original coordinate to an actual Maven artifact...

Tycho is a maven tool, so it tries to interact with maven as good as possible, if it makes sense in a very specific particular singular use-case is not that important on the other side if it helps on the broader range.

The initial idea of "using artifacts from maven" is and was that I add an artifact to my target from maven and then PDE can handle this, and for Tycho it looks like a native maven artifact (that is it is deployed somewhere and has a GAV that can be matched to the original).

Now that artifacts are not traveling from "usual" sources to the build but hopping from maven > P2 > private repo > P2 things become more complicated and working less "logical".

laeubi commented 1 year ago

Just for reference support for mapping wrapped artifacts was added here:

https://github.com/eclipse-tycho/tycho/pull/2052

A wrapped artifact is usually only wrapped for the sake of using it inside OSGi it does not add anything new to a dependency (from maven point of view) therefore it is still "that maven artifact".

merks commented 1 year ago

Thanks. This all makes perfectly good sense! Indeed this Ant :ant: 😱 thing is a horrible corner case. Though to split hairs, it's not going to a private repo, it's just not going to Maven Central. Regardless, it is arguably going into dark hole...

Ideally the Platform would just use the ant artifacts and wrap them as needed such that we don't have this problem. But there is too much in the way of dependencies around the older structure of the wrapped ant bundle that I personally do not want to revisit, and likely no one wants to revisit it because it's probably more pleasant to pull out a fingernail.

From your description, I think it is definitely generally useful to know the original coordinates of the wrapped artifacts even for these cases where they are repackaged. It's just ant that is an outlier because it doesn't really come from anything one thing directly available at Maven central; that was true even for the EBR-based build of it.

I get the sense that there is really nothing here for Tycho to do given that 4.0.3 works, at least for me... Given @jonahgraham opened this I'll leave this for him to comment later in the day.

laeubi commented 1 year ago

@merks if you created a special ant, its perfectly fine to deploy it to maven central under another group id, this is especially important if people are using a bundle from platform in their plain maven builds that relies on that. I think it would also needed if we later want to map this back when the platform release is pushed to maven later on.

merks commented 1 year ago

If anything, It would be better to publish the actual result OSGi wrapped bundle because the intermediate things are effectively a transformed dump of the two zips files from the following sites, maintained in a form that remains consumable via a *.target with BND instructions:

I don't think those would even conform to the nexus publishing rules. I.e., these are an intermediate result that is of no direct use to anyone because they are not really proper/well-formed maven jars...

The EBR wrapped versions were never published either. We just mapped that EBR-wrapped bundle to https://repo1.maven.org/maven2/org/apache/ant/ant and called it goodness...

But we digress...

laeubi commented 1 year ago

We just mapped that EBR-wrapped bundle to https://repo1.maven.org/maven2/org/apache/ant/ant and called it goodness...

But why this can't then be used as the wrapped item as well?

You wrote

maintained in a form that remains consumable via a *.target with BND instructions

it does not has a manifest so it should be wrappable as well...

But we digress...

I don't think so, the issue only arise because of this quite unusual setup, so I think it is important to understand why it has become an issue, if it is possible to maybe prevent it somehow or if something must/should be improved to support such use-cases.

merks commented 1 year ago

We just mapped that EBR-wrapped bundle to https://repo1.maven.org/maven2/org/apache/ant/ant and called it goodness...

But why this can't then be used as the wrapped item as well?

Because the actual wrapped bundle is a composition of a whole whack of jars from maven central all rooted at this folder:

https://repo1.maven.org/maven2/org/apache/ant/

Plus some "spice" copied into the Git repository. If you wish the historical gory details you can peruse the following, starting with the readme:

https://github.com/eclipse-orbit/orbit/tree/main/apache/ant

You wrote

maintained in a form that remains consumable via a *.target with BND instructions

it does not has a manifest so it should be wrappable as well...

I'm not sure what point you are trying to make. I think we are both saying it's in a form that is wrappable...

But we digress...

I don't think so, the issue only arise because of this quite unusual setup, so I think it is important to understand why it has become an issue, if it is possible to maybe prevent it somehow or if something must/should be improved to support such use-cases.

From a Tycho point of view, it's only relevant that the maven-wrapped-properties are handed off to Maven and that Maven tries to resolve them in Maven central or in the repositories specified in the poml.xml.

The issue of the ant abomination (that I inherited) is specific to Orbit, which is why I opened this issue about it just today:

https://github.com/eclipse-orbit/orbit-simrel/issues/17

It's also why the issue of how ant is handled was discussed previously in Obit:

https://github.com/eclipse-orbit/orbit-simrel/issues/13

There is much detail recorded there for those who wish to understand all the historical and current details. That to me seems like the best place to discuss ant...

laeubi commented 1 year ago

Because the actual wrapped bundle is a composition of a whole whack of jars from maven central

Just in case you can use one jar as the "host" and any additional stuff as fragment, thats how m2e is doing it do merge different items into one composite:

Main bundle: https://github.com/eclipse-m2e/m2e-core/blob/8dd14dbdea5a6731983fcece5c845a87ee3d584e/target-platform/target-platform.target#L113C3-L113C3

Fragments merged with the host: https://github.com/eclipse-m2e/m2e-core/blob/8dd14dbdea5a6731983fcece5c845a87ee3d584e/target-platform/target-platform.target#L136

eclipse-tycho / tycho

eclipse-repository pulls dependencies from maven instead of p2 #2881

Discussed in https://github.com/eclipse-tycho/tycho/discussions/2732