Fetching maven relocation issues is flaky, leading to missing relocations

[holly-cummins]

See, for example, https://quarkus.io/extensions/io.quarkus/quarkus-webjars-locator/ which should be a relocation but today is not. Similarly, https://quarkus.io/extensions/io.quarkus/quarkus-rest/.

We cache to reduce flakiness of the maven API but I think one side effect of this is that errors and missing data persist for longer.

[jmini]

For io.quarkus:quarkus-webjars-locator there is a relocation information in quarkus-webjars-locator-3.14.2.pom.

This is present since quarkus-webjars-locator-3.11.0.pom

The last non relocated version is quarkus-webjars-locator-3.10.2.pom

As indicated on the page:

Latest Version: 3.10.2

Each pom published since 3.10.2 contains the data with the <relocation> section and the <message>:

Update the artifactId in your project build file. Refer to https://github.com/quarkusio/quarkus/wiki/Migration-Guide-3.11 for more information.

For me the maven API way to check if there is a newer version since 3.10.2 is to fetch the maven-metadata.xml file.

---

For io.quarkus:quarkus-rest I do not see a relocated information in quarkus-rest-3.14.2.pom.

[holly-cummins]

For me the maven API way to check if there is a newer version since 3.10.2 is to fetch the maven-metadata.xml file.

Yes, that's the bit that fails intermittently in the CI. We just can't fetch the files from maven reliably. We cache to reduce our dependence on the flaky maven APIs, but that also means once we decide a relocation doesn't exist, we cache that incorrect answer. (Fixing that so we only cache correct answers, not incorrect answers, is the intention of this work item.)

For io.quarkus:quarkus-rest I do not see a relocated information in quarkus-rest-3.14.2.pom.

The quarkus-rest artifact is the newer one, so for that pair of artifacts, the relocation information is in https://central.sonatype.com/artifact/io.quarkus/quarkus-resteasy-reactive.

As an example of it working/not-working intermittently, last Wednesday, https://quarkus.io/extensions/io.quarkiverse.reactivemessaging.nats-jetstream/quarkus-reactive-messsaging-nats-jetstream/ gave this:

On Thursday and Friday the relocation information wasn't visible on the live site. I went to take a screencap showing the missing of relocation information today, but the relocation information is back, so I can't screencap the failure case. There's been no code changes in that time, just flaky download endpoints for the information we need, and cache expiries making everything even more non-deterministic.

[jmini]

I see you are doing already a lot and mostly I can't follow what is going on.

We just can't fetch the files from maven reliably.

I don't know what you are doing, but I can't imagine that fetching the latest version of quarkus-webjars-locator:

<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-webjars-locator</artifactId>
  <version>RELEASE</version>
</dependency>

can return the 3.10.2 version…

Sometimes you can have some caching delay, but not more than 3 months (3.11.0 was released on May 22, 2024)

[jmini]

I did some more investigations that might be interesting…

EDIT: to provide some context on my investigations, I did not looked at how this project is doing the lookup of the artifacts in maven central, but I investigated different way how this can be done with other tools

I think the major issue is that maven already "resolves" the relocation information. I found online that a [warn] line might be in the build logs. I am really more used to the way gradle works (which corresponds in most the the cases to my expectations).

The idea is to use the tool to get the latest released version for without having to implement it.

1) getting the effective POM with maven

I do not know how to query the POM file with maven, I thought the effective POM is probably one of the closest artifact I can get

With the command:

mvn org.apache.maven.plugins:maven-help-plugin:3.5.0:effective-pom -Dartifact=io.quarkus:quarkus-webjars-locator:RELEASE 

in a project where you have a dummy pom.xml file:

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>g</groupId>
    <artifactId>a</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>pom</packaging>
</project>

returns a very long effective pom, but already pointing to the new location:

<?xml version="1.0" encoding="UTF-8"?>
<!-- ====================================================================== -->
<!--                                                                        -->
<!-- Generated by Maven Help Plugin                                         -->
<!-- See: https://maven.apache.org/plugins/maven-help-plugin/               -->
<!--                                                                        -->
<!-- ====================================================================== -->
<!-- ====================================================================== -->
<!--                                                                        -->
<!-- Effective POM for project                                              -->
<!-- 'io.quarkus:quarkus-web-dependency-locator:jar:3.14.2'                 -->
<!--                                                                        -->
<!-- ====================================================================== -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd" child.project.url.inherit.append.path="false">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-web-dependency-locator-parent</artifactId>
    <version>3.14.2</version>
  </parent>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-web-dependency-locator</artifactId>
  <version>3.14.2</version>
  <name>Quarkus - Web Dependency Locator - Runtime</name>
  <description>Simplify paths and importmap support for Web dependencies</description>
  <url>https://github.com/quarkusio/quarkus</url>
  <inceptionYear>2020</inceptionYear>
  ...

==> Basically when you ask for the effective POM of a relocated artifact, you get the effective relocated POM.

2) jbang info tools

I do not know how to get a POM type with jbang, (there is the @POM notation but this is for BOM and version enforcement) so I decided to load the jar and to abuse the jbang info tools to check what was downloaded

Mimimal jbang script:

Content of X.java

class X {}

Call:

jbang info tools --deps io.quarkus:quarkus-webjars-locator:RELEASE X.java

==> The generated JSON file contains in the first line of resolvedDependencies a reference to quarkus-web-dependency-locator-3.14.2.jar so also the relocated artifact.

complete output

```json { "originalResource": "X.java", "backingResource": "/___user-home___/tmp/tmp-pom/output/X.java", "applicationJar": "/___user-home___/.jbang/cache/jars/X.java.5ffc598c82aff782fe8bf6d2d11081c5cce8d44d6c77d020e95a9a23aa56d7ac/X.jar", "dependencies": [ "io.quarkus:quarkus-webjars-locator:RELEASE" ], "resolvedDependencies": [ "/___user-home___/.m2/repository/io/quarkus/quarkus-web-dependency-locator/3.14.2/quarkus-web-dependency-locator-3.14.2.jar", "/___user-home___/.m2/repository/io/quarkus/quarkus-core/3.14.2/quarkus-core-3.14.2.jar", "/___user-home___/.m2/repository/jakarta/annotation/jakarta.annotation-api/3.0.0/jakarta.annotation-api-3.0.0.jar", "/___user-home___/.m2/repository/jakarta/enterprise/jakarta.enterprise.cdi-api/4.1.0/jakarta.enterprise.cdi-api-4.1.0.jar", "/___user-home___/.m2/repository/jakarta/enterprise/jakarta.enterprise.lang-model/4.1.0/jakarta.enterprise.lang-model-4.1.0.jar", "/___user-home___/.m2/repository/jakarta/el/jakarta.el-api/6.0.0/jakarta.el-api-6.0.0.jar", "/___user-home___/.m2/repository/jakarta/interceptor/jakarta.interceptor-api/2.2.0/jakarta.interceptor-api-2.2.0.jar", "/___user-home___/.m2/repository/jakarta/inject/jakarta.inject-api/2.0.1/jakarta.inject-api-2.0.1.jar", "/___user-home___/.m2/repository/io/smallrye/common/smallrye-common-os/2.6.0/smallrye-common-os-2.6.0.jar", "/___user-home___/.m2/repository/io/quarkus/quarkus-ide-launcher/3.14.2/quarkus-ide-launcher-3.14.2.jar", "/___user-home___/.m2/repository/io/quarkus/quarkus-development-mode-spi/3.14.2/quarkus-development-mode-spi-3.14.2.jar", "/___user-home___/.m2/repository/io/smallrye/config/smallrye-config/3.9.1/smallrye-config-3.9.1.jar", "/___user-home___/.m2/repository/io/smallrye/config/smallrye-config-core/3.9.1/smallrye-config-core-3.9.1.jar", "/___user-home___/.m2/repository/org/eclipse/microprofile/config/microprofile-config-api/3.1/microprofile-config-api-3.1.jar", "/___user-home___/.m2/repository/io/smallrye/common/smallrye-common-classloader/2.4.0/smallrye-common-classloader-2.4.0.jar", "/___user-home___/.m2/repository/org/ow2/asm/asm/9.7/asm-9.7.jar", "/___user-home___/.m2/repository/io/smallrye/config/smallrye-config-common/3.9.1/smallrye-config-common-3.9.1.jar", "/___user-home___/.m2/repository/org/jboss/logging/jboss-logging/3.6.0.Final/jboss-logging-3.6.0.Final.jar", "/___user-home___/.m2/repository/org/jboss/logmanager/jboss-logmanager/3.0.6.Final/jboss-logmanager-3.0.6.Final.jar", "/___user-home___/.m2/repository/io/smallrye/common/smallrye-common-constraint/2.2.0/smallrye-common-constraint-2.2.0.jar", "/___user-home___/.m2/repository/io/smallrye/common/smallrye-common-cpu/2.2.0/smallrye-common-cpu-2.2.0.jar", "/___user-home___/.m2/repository/io/smallrye/common/smallrye-common-expression/2.2.0/smallrye-common-expression-2.2.0.jar", "/___user-home___/.m2/repository/io/smallrye/common/smallrye-common-function/2.2.0/smallrye-common-function-2.2.0.jar", "/___user-home___/.m2/repository/io/smallrye/common/smallrye-common-net/2.2.0/smallrye-common-net-2.2.0.jar", "/___user-home___/.m2/repository/io/smallrye/common/smallrye-common-ref/2.2.0/smallrye-common-ref-2.2.0.jar", "/___user-home___/.m2/repository/jakarta/json/jakarta.json-api/2.1.3/jakarta.json-api-2.1.3.jar", "/___user-home___/.m2/repository/org/eclipse/parsson/parsson/1.1.6/parsson-1.1.6.jar", "/___user-home___/.m2/repository/org/jboss/logging/jboss-logging-annotations/3.0.1.Final/jboss-logging-annotations-3.0.1.Final.jar", "/___user-home___/.m2/repository/org/jboss/threads/jboss-threads/3.6.1.Final/jboss-threads-3.6.1.Final.jar", "/___user-home___/.m2/repository/org/slf4j/slf4j-api/2.0.6/slf4j-api-2.0.6.jar", "/___user-home___/.m2/repository/org/jboss/slf4j/slf4j-jboss-logmanager/2.0.0.Final/slf4j-jboss-logmanager-2.0.0.Final.jar", "/___user-home___/.m2/repository/org/wildfly/common/wildfly-common/1.7.0.Final/wildfly-common-1.7.0.Final.jar", "/___user-home___/.m2/repository/io/quarkus/quarkus-bootstrap-runner/3.14.2/quarkus-bootstrap-runner-3.14.2.jar", "/___user-home___/.m2/repository/io/quarkus/quarkus-classloader-commons/3.14.2/quarkus-classloader-commons-3.14.2.jar", "/___user-home___/.m2/repository/io/smallrye/common/smallrye-common-io/2.6.0/smallrye-common-io-2.6.0.jar", "/___user-home___/.m2/repository/io/quarkus/quarkus-fs-util/0.0.10/quarkus-fs-util-0.0.10.jar", "/___user-home___/.m2/repository/io/quarkus/quarkus-vertx-http/3.14.2/quarkus-vertx-http-3.14.2.jar", "/___user-home___/.m2/repository/io/quarkus/quarkus-security-runtime-spi/3.14.2/quarkus-security-runtime-spi-3.14.2.jar", "/___user-home___/.m2/repository/io/quarkus/quarkus-tls-registry/3.14.2/quarkus-tls-registry-3.14.2.jar", "/___user-home___/.m2/repository/io/quarkus/quarkus-arc/3.14.2/quarkus-arc-3.14.2.jar", "/___user-home___/.m2/repository/org/eclipse/microprofile/context-propagation/microprofile-context-propagation-api/1.3/microprofile-context-propagation-api-1.3.jar", "/___user-home___/.m2/repository/io/quarkus/quarkus-credentials/3.14.2/quarkus-credentials-3.14.2.jar", "/___user-home___/.m2/repository/io/quarkus/arc/arc/3.14.2/arc-3.14.2.jar", "/___user-home___/.m2/repository/jakarta/transaction/jakarta.transaction-api/2.0.1/jakarta.transaction-api-2.0.1.jar", "/___user-home___/.m2/repository/io/quarkus/quarkus-mutiny/3.14.2/quarkus-mutiny-3.14.2.jar", "/___user-home___/.m2/repository/io/smallrye/reactive/mutiny/2.6.2/mutiny-2.6.2.jar", "/___user-home___/.m2/repository/org/jctools/jctools-core/4.0.5/jctools-core-4.0.5.jar", "/___user-home___/.m2/repository/io/quarkus/quarkus-smallrye-context-propagation/3.14.2/quarkus-smallrye-context-propagation-3.14.2.jar", "/___user-home___/.m2/repository/io/smallrye/smallrye-context-propagation/2.1.2/smallrye-context-propagation-2.1.2.jar", "/___user-home___/.m2/repository/io/smallrye/smallrye-context-propagation-api/2.1.2/smallrye-context-propagation-api-2.1.2.jar", "/___user-home___/.m2/repository/io/smallrye/smallrye-context-propagation-storage/2.1.2/smallrye-context-propagation-storage-2.1.2.jar", "/___user-home___/.m2/repository/io/smallrye/reactive/mutiny-smallrye-context-propagation/2.6.2/mutiny-smallrye-context-propagation-2.6.2.jar", "/___user-home___/.m2/repository/io/smallrye/common/smallrye-common-vertx-context/2.6.0/smallrye-common-vertx-context-2.6.0.jar", "/___user-home___/.m2/repository/io/vertx/vertx-core/4.5.9/vertx-core-4.5.9.jar", "/___user-home___/.m2/repository/io/netty/netty-common/4.1.111.Final/netty-common-4.1.111.Final.jar", "/___user-home___/.m2/repository/io/netty/netty-buffer/4.1.111.Final/netty-buffer-4.1.111.Final.jar", "/___user-home___/.m2/repository/io/netty/netty-transport/4.1.111.Final/netty-transport-4.1.111.Final.jar", "/___user-home___/.m2/repository/io/netty/netty-handler/4.1.111.Final/netty-handler-4.1.111.Final.jar", "/___user-home___/.m2/repository/io/netty/netty-transport-native-unix-common/4.1.111.Final/netty-transport-native-unix-common-4.1.111.Final.jar", "/___user-home___/.m2/repository/io/netty/netty-handler-proxy/4.1.111.Final/netty-handler-proxy-4.1.111.Final.jar", "/___user-home___/.m2/repository/io/netty/netty-codec-socks/4.1.111.Final/netty-codec-socks-4.1.111.Final.jar", "/___user-home___/.m2/repository/io/netty/netty-codec-http/4.1.111.Final/netty-codec-http-4.1.111.Final.jar", "/___user-home___/.m2/repository/io/netty/netty-codec-http2/4.1.111.Final/netty-codec-http2-4.1.111.Final.jar", "/___user-home___/.m2/repository/io/netty/netty-resolver/4.1.111.Final/netty-resolver-4.1.111.Final.jar", "/___user-home___/.m2/repository/io/netty/netty-resolver-dns/4.1.111.Final/netty-resolver-dns-4.1.111.Final.jar", "/___user-home___/.m2/repository/io/netty/netty-codec-dns/4.1.111.Final/netty-codec-dns-4.1.111.Final.jar", "/___user-home___/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.16.1/jackson-core-2.16.1.jar", "/___user-home___/.m2/repository/io/quarkus/security/quarkus-security/2.1.0/quarkus-security-2.1.0.jar", "/___user-home___/.m2/repository/io/quarkus/quarkus-vertx/3.14.2/quarkus-vertx-3.14.2.jar", "/___user-home___/.m2/repository/io/quarkus/quarkus-netty/3.14.2/quarkus-netty-3.14.2.jar", "/___user-home___/.m2/repository/io/netty/netty-codec/4.1.111.Final/netty-codec-4.1.111.Final.jar", "/___user-home___/.m2/repository/com/aayushatharva/brotli4j/brotli4j/1.16.0/brotli4j-1.16.0.jar", "/___user-home___/.m2/repository/com/aayushatharva/brotli4j/service/1.16.0/service-1.16.0.jar", "/___user-home___/.m2/repository/com/aayushatharva/brotli4j/native-osx-x86_64/1.16.0/native-osx-x86_64-1.16.0.jar", "/___user-home___/.m2/repository/io/netty/netty-codec-haproxy/4.1.111.Final/netty-codec-haproxy-4.1.111.Final.jar", "/___user-home___/.m2/repository/io/smallrye/common/smallrye-common-annotation/2.6.0/smallrye-common-annotation-2.6.0.jar", "/___user-home___/.m2/repository/io/quarkus/quarkus-virtual-threads/3.14.2/quarkus-virtual-threads-3.14.2.jar", "/___user-home___/.m2/repository/io/quarkus/quarkus-vertx-latebound-mdc-provider/3.14.2/quarkus-vertx-latebound-mdc-provider-3.14.2.jar", "/___user-home___/.m2/repository/io/smallrye/reactive/smallrye-mutiny-vertx-core/3.14.0/smallrye-mutiny-vertx-core-3.14.0.jar", "/___user-home___/.m2/repository/io/smallrye/reactive/smallrye-mutiny-vertx-runtime/3.14.0/smallrye-mutiny-vertx-runtime-3.14.0.jar", "/___user-home___/.m2/repository/io/smallrye/reactive/vertx-mutiny-generator/3.14.0/vertx-mutiny-generator-3.14.0.jar", "/___user-home___/.m2/repository/io/vertx/vertx-codegen/4.5.9/vertx-codegen-4.5.9.jar", "/___user-home___/.m2/repository/io/smallrye/smallrye-fault-tolerance-vertx/6.4.0/smallrye-fault-tolerance-vertx-6.4.0.jar", "/___user-home___/.m2/repository/io/smallrye/reactive/smallrye-mutiny-vertx-web/3.14.0/smallrye-mutiny-vertx-web-3.14.0.jar", "/___user-home___/.m2/repository/io/smallrye/reactive/smallrye-mutiny-vertx-web-common/3.14.0/smallrye-mutiny-vertx-web-common-3.14.0.jar", "/___user-home___/.m2/repository/io/smallrye/reactive/smallrye-mutiny-vertx-auth-common/3.14.0/smallrye-mutiny-vertx-auth-common-3.14.0.jar", "/___user-home___/.m2/repository/io/smallrye/reactive/smallrye-mutiny-vertx-bridge-common/3.14.0/smallrye-mutiny-vertx-bridge-common-3.14.0.jar", "/___user-home___/.m2/repository/io/smallrye/reactive/smallrye-mutiny-vertx-uri-template/3.14.0/smallrye-mutiny-vertx-uri-template-3.14.0.jar", "/___user-home___/.m2/repository/io/vertx/vertx-uri-template/4.5.9/vertx-uri-template-4.5.9.jar", "/___user-home___/.m2/repository/io/vertx/vertx-web/4.5.9/vertx-web-4.5.9.jar", "/___user-home___/.m2/repository/io/vertx/vertx-web-common/4.5.9/vertx-web-common-4.5.9.jar", "/___user-home___/.m2/repository/io/vertx/vertx-auth-common/4.5.9/vertx-auth-common-4.5.9.jar", "/___user-home___/.m2/repository/io/vertx/vertx-bridge-common/4.5.9/vertx-bridge-common-4.5.9.jar", "/___user-home___/.m2/repository/io/github/crac/org-crac/0.1.3/org-crac-0.1.3.jar" ], "availableJdkPath": "/___user-home___/.sdkman/candidates/java/21.0.2-tem", "compileOptions": [ "-g" ], "sources": [ { "originalResource": "X.java", "backingResource": "/___user-home___/tmp/tmp-pom/output/X.java" } ] } ```

3) Using Gradle to download the pom

At the end the way I managed to look at the pom of the latest version, without any modification (the same file as the one that is hosted on maven central) was to write a small gradle project allowing to download the specified dependencies:

build.gradle file:

configurations {
    conf
}

repositories {
    mavenCentral()
}

dependencies {
    conf("io.quarkus:quarkus-webjars-locator:latest.release@pom")
}

//simple task to download the content of the "conf" configuration into the "output" folder
task copyDependencies(type: Copy) {
   from configurations.conf
   into 'output'
}

And then calling gradle copyDependencies copies the expected POM file into the output folder:

output
└── quarkus-webjars-locator-3.14.2.pom

Here you would get the relocation information.

3.b) trying to download the relocated jar with gradle

This is off topic

I then tried to changed the line to indicate to gradle to fetch the jar:

    conf("io.quarkus:quarkus-webjars-locator:latest.release")

And it is failing with:

* What went wrong:
Execution failed for task ':copyDependencies'.
> Could not resolve all files for configuration ':conf'.
   > Could not find io.quarkus:quarkus-web-dependency-locator:${project.version}.
     Required by:
         project : > io.quarkus:quarkus-webjars-locator:3.14.2

This might bug in Gradle, because I think they do not fully honor the resolution of ${project.variable} (see "Project Interpolation and Variables") in this case.

Nobody is perfect 😉

I also saw this open issue about not telling that the relocated jar was loaded: https://github.com/gradle/gradle/issues/26436

[holly-cummins]

I see you are doing already a lot and mostly I can't follow what is going on.

We just can't fetch the files from maven reliably.

I don't know what you are doing, but I can't imagine that fetching the latest version of quarkus-webjars-locator:

<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-webjars-locator</artifactId>
  <version>RELEASE</version>
</dependency>

can return the 3.10.2 version…

Sometimes you can have some caching delay, but not more than 3 months (3.11.0 was released on May 22, 2024)

The longest we have our cache set to hold information for is about 10 days, and that's only for data that we expect not to change at all. The reason the version number is old is that you're looking at the pre-relocation version of the page (you can tell from the URL, which is io.quarkus/quarkus-webjars-locator/, the previous coordinates). The last version released with the old maven coordinates is 3.10.2, which is why you're seeing that version. If you want to see the more recent releases, you should use the url with the updated coordinates, https://quarkus.io/extensions/io.quarkus/quarkus-web-dependency-locator/.

The "old" page should have a little banner at the top saying 'there's a new version of this jar', with a link to the new ones, and the "new" version should say "this used to have a different name," with a link to the old one. We also would normally only show the "new" page on the front page listing. That's not happening, because the maven endpoint is flaky and was having a bad day when we populated the cache, so we weren't able to get the relevant xml out of maven central. At some point, in a non-deterministic way (or if I improve the caching logic to propagate errors around so it never caches 'error' results), both pages will show what they're supposed to.

[jmini]

The last version released with the old maven coordinates is 3.10.2, which is why you're seeing that version

This is where I think I disagree with this analysis.

3.10.2 is the latest version that has a jar extension.

If you would do a pom extension lookup you would find more recent information.

With gradle this is the artifact with coordinates io.quarkus:quarkus-webjars-locator:latest.release@pom. With built-in maven I am not sure if you can query the pom. But manually based on [maven-metadata.xml](https://repo1.maven.org/maven2/io/quarkus/quarkus-webjars-locator/maven-metadata.xml) and then for each entry in that XML file fetch to the corresponding file with a pom extension (see here for the string concatenation logic) you will find the data you need.

And because you are doing a good job in the quarkus project, you even have a message pointing to the release note or blog post indicating why the artifact was relocated. But you need the pom file to know this.