No real libyear information from toolbox:libyear

[lfvjimisola]

The output from mvn eu.maveniverse.maven.plugins:toolbox:libyear is:

[INFO] --- toolbox:0.1.25:libyear (default-cli) @ topsky-swim-feeder ---
Downloading from lfv-sysdev-jfrog: https://jfrog.somehost.com/artifactory/sysdev-maven2/com/coopans/topsky/adapters/topsky-consumer-adapter/maven-metadata.xml
[WARNING] Could not transfer metadata com.coopans.topsky.adapters:topsky-consumer-adapter/maven-metadata.xml from/to lfv-sysdev-jfrog (https://jfrog.somehost.com/artifactory/sysdev-maven2/): status code: 401, reason phrase: Unauthorized (401)
<...>
[INFO] Outdated versions
[INFO] io.micrometer:micrometer-registry-prometheus:jar 1.13.0 (?) => 1.13.1 (?)
[INFO] io.micrometer:micrometer-tracing-bridge-otel:jar 1.3.0 (?) => 1.3.1 (?)
[INFO] io.micrometer:micrometer-tracing-test:jar 1.3.0 (?) => 1.3.1 (?)
[INFO] io.micrometer:micrometer-tracing:jar 1.3.0 (?) => 1.3.1 (?)
[INFO] org.eclipse.jetty.http2:jetty-http2-server:jar 12.0.9 (?) => 12.0.10 (?)
[INFO] org.eclipse.jetty:jetty-alpn-conscrypt-server:jar 12.0.9 (?) => 12.0.10 (?)
[INFO] 
[INFO] Total of 0.00 years from 6 outdated dependencies
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.546 s
[INFO] Finished at: 2024-06-14T11:24:51+02:00
[INFO] ------------------------------------------------------------------------

There are two issues and I'm not sure if they are related:

1. Unauthorization fails to our JFrog instance (works for everything else we use)
2. There is question mark (?) everywhere instead of what I suspect should be libyear information (Total is 0.00 as well). This is valid also for other maven runs which has no authorization issues, i.e. Total of 0.00 years.

Regarding authorization, username and JFrog generate user access token are in settings.xml for relevant servers.
Authentication works for other maven runs/builds.

The file maven-metadata.xml  of the failed download above exists and is accessible when authorized correctly.

[WARNING] Toolbox 0.1.25 (MIMA Runtime 'embedded-maven' version 2.4.14) [WARNING] ======= [INFO] Maven version 3.9.6 [INFO] Managed false [INFO] Basedir [INFO] Offline false [INFO] [INFO] MAVEN_HOME /opt/dev/sdkman/candidates/maven/current [INFO] settings.xml /opt/dev/sdkman/candidates/maven/current/conf/settings.xml [INFO] toolchains.xml /opt/dev/sdkman/candidates/maven/current/conf/toolchains.xml [INFO] [INFO] USER_HOME /home/user1/.m2 [INFO] settings.xml /home/user1/.m2/settings.xml [INFO] settings-security.xml /home/user1/.m2/settings-security.xml [INFO] local repository /home/user1/.m2/repository [INFO] [INFO] PROFILES [INFO] Active [] [INFO] Inactive [] [INFO] [INFO] REMOTE REPOSITORIES [INFO] lfv-sysdev-jfrog (https://jfrog.somehost.com/artifactory/sysdev-maven2/, default, releases), mirror of [INFO] central (https://repo.maven.apache.org/maven2/, default, releases) [INFO] lfv-sysdev-jfrog-maven2-repo (https://jfrog.somehost.com/artifactory/sysdev-maven2/, default, releases+snapshots) [INFO] [INFO] HTTP PROXY [INFO] url http://proxy.somehost.com:8080 [INFO] nonProxyHosts localhost|127.0.0.0/8|::1|10.0.0.0/8|somehost.com|*.somehost.com [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1.185 s [INFO] Finished at: 2024-06-14T11:31:03+02:00 [INFO] ------------------------------------------------------------------------

[cstamas]

Howdy @lfvjimisola ! Thanks for the issue.

First, libYear in Toolbox relies on https://github.com/apache/maven-indexer/tree/master/search-backend-remoterepository

Hence, make sure that your artifact server (JFrog AF in this case, I guess) sends the proper (and expected) artifact date (== which is and should be roughly the "publish date").

The Indexer API "remoterepository" backend really just reuses this (as shown in case of Central, but also works with Nexus, as it preserves file timestamps when artifact proxied, just like Resolver does):

[cstamas@angeleyes ~]$ curl -I https://repo.maven.apache.org/maven2/io/micrometer/micrometer-tracing/1.3.0/micrometer-tracing-1.3.0.jar
HTTP/2 200 
etag: "5d4396205e5a6ed008d17fb82c014941"
content-type: application/java-archive
last-modified: Mon, 13 May 2024 17:48:38 GMT
x-checksum-md5: 5d4396205e5a6ed008d17fb82c014941
x-checksum-sha1: 05a15be17c11ad128afd419536c603b9f215b137
via: 1.1 varnish, 1.1 varnish
accept-ranges: bytes
date: Fri, 14 Jun 2024 09:42:08 GMT
age: 872348
x-served-by: cache-iad-kjyo7100058-IAD, cache-cph2320049-CPH
x-cache: HIT, HIT
x-cache-hits: 14701, 1
x-timer: S1718358128.008121,VS0,VE1
content-length: 109665

[cstamas@angeleyes ~]$ 

For start, make sure your artifact server sends the proper last-modified HTTP header in response.

[lfvjimisola]

Wow! That was a fast answer. Thank you. Noticed now that another libyear maven plugin has the same issue.

u30576@DCL0004:~$curl -I https://jfrog.somehost.com/artifactory/sysdev-maven2/io/micrometer/micrometer-tracing/1.3.0/micrometer-tracing-1.3.0.jar
HTTP/1.1 200 OK
accept-ranges: bytes
content-disposition: attachment; filename="micrometer-tracing-1.3.0.jar"; filename*=UTF-8''micrometer-tracing-1.3.0.jar
content-length: 109665
content-type: application/java-archive
date: Fri, 14 Jun 2024 09:57:49 GMT
etag: 05a15be17c11ad128afd419536c603b9f215b137
last-modified: Mon, 13 May 2024 17:48:38 GMT
x-artifactory-filename: micrometer-tracing-1.3.0.jar
x-artifactory-id: 469de5e46a1f5bfd:5b069de5:18fcf0bb1e0:-8000
x-artifactory-node-id: jfrog-prod-artifactory-0
x-checksum-md5: 5d4396205e5a6ed008d17fb82c014941
x-checksum-sha1: 05a15be17c11ad128afd419536c603b9f215b137
x-checksum-sha256: a3cde37db423c22fa45ebe6072f8de3b45855007158ec5e959a1d34fb83cb75e
x-jfrog-version: Artifactory/7.59.22 75922900
set-cookie: bd82700a40947b710eacf8eda6578dbb=ece79c60f8c76d8c4cbc4d36b183278c; path=/; HttpOnly; Secure; SameSite=None
cache-control: private

[cstamas]

Hm, for start verified that libyear CAN get the date okay, IF only central involved:

[cstamas@angeleyes ~]$ mvn eu.maveniverse.maven.plugins:toolbox:gav-libyear -Dgav=io.micrometer:micrometer-registry-prometheus:1.13.0
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] --------------------------------[ pom ]---------------------------------
[INFO] 
[INFO] --- toolbox:0.1.25:gav-libyear (default-cli) @ standalone-pom ---
[INFO] Outdated versions with known age
[INFO] 0.13 years from io.prometheus:prometheus-metrics-core:jar 1.2.1 (2024-04-04) => 1.3.1 (2024-05-23) [age 0.06 years]
[INFO] 0.13 years from io.prometheus:prometheus-metrics-tracer-common:jar 1.2.1 (2024-04-04) => 1.3.1 (2024-05-23) [age 0.06 years]
[INFO] 0.13 years from io.prometheus:prometheus-metrics-exposition-formats:jar 1.2.1 (2024-04-04) => 1.3.1 (2024-05-23) [age 0.06 years]
[INFO] 0.08 years from io.micrometer:micrometer-core:jar 1.13.0 (2024-05-13) => 1.13.1 (2024-06-10) [age 0.00 years]
[INFO] 
[INFO] 
[INFO] Total of 0.48 years from 4 outdated dependencies
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.645 s
[INFO] Finished at: 2024-06-14T12:06:17+02:00
[INFO] ------------------------------------------------------------------------
[cstamas@angeleyes ~]$ 

(hint: almost all Mojos are "doubled": w/ and w/o gav- prefix. Those without gav- prefix require a project, while those with gav- prefix does not, but require -Dgav=g:a:v param and can target any existing/reachable artifact.)

[cstamas]

Btw, and date is strange you got: that is today date, no? And libYear will eliminate dates if "newer" is not AFTER "older" date... Nope, sorry, mixed up date and last-modified :smile:

[lfvjimisola]

A bit of the root issue here as we are behind company proxy and JFrog that acts as a mirror for Maven Central (see toolbox:dump above). I removed everything but proxy settings from settings.xml,but goal execution fails with HttpConnectTimeoutException. Which seems strange because eu/maveniverse/maven/plugins/toolbox/maven-metadata.xml is downloaded.

Any ideas?

u30576@DCL0004:~/.m2$mvn -U eu.maveniverse.maven.plugins:toolbox:gav-libyear -Dgav=io.micrometer:micrometer-registry-prometheus:1.13.0
[INFO] Scanning for projects...
Downloading from central: https://repo.maven.apache.org/maven2/eu/maveniverse/maven/plugins/toolbox/maven-metadata.xml
Downloaded from central: https://repo.maven.apache.org/maven2/eu/maveniverse/maven/plugins/toolbox/maven-metadata.xml (1.1 kB at 4.5 kB/s)
[INFO] 
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] --------------------------------[ pom ]---------------------------------
[INFO] 
[INFO] --- toolbox:0.1.25:gav-libyear (default-cli) @ standalone-pom ---
[INFO] 
[INFO] 
[INFO] Total of 0.00 years from 0 outdated dependencies
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  10.621 s
[INFO] Finished at: 2024-06-14T12:45:21+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal eu.maveniverse.maven.plugins:toolbox:0.1.25:gav-libyear (default-cli) on project standalone-pom: Runtime exception: java.io.UncheckedIOException: java.net.http.HttpConnectTimeoutException: HTTP connect timed out -> [Help 1]

[cstamas]

Sadly, one of the issues are identified: https://github.com/maveniverse/toolbox/issues/51 Bummer. This means that in your env no released toolbox can work, until this is fixed.

[lfvjimisola]

Ok. Thank that the info. I'll keep an eye on #51 then.

[cstamas]

@lfvjimisola please try the version 0.1.26 of toolbox. Added Auth and Proxy support (not yet mirror, but I have a feeling it should be enough). Just now released, so "sync to Central pending" (10-20 minutes).

[lfvjimisola]

Great. Thanks. But sadly, still no dates.

Do you think that it's an issue with JFrog Artifactory?

[INFO] --- toolbox:0.1.26:gav-libyear (default-cli) @ topsky-swim-feeder ---
[INFO] Outdated versions
[INFO] io.micrometer:micrometer-core:jar 1.13.0 (?) => 1.13.1 (?)
[INFO] io.prometheus:prometheus-metrics-core:jar 1.2.1 (?) => 1.3.1 (?)
[INFO] io.prometheus:prometheus-metrics-exposition-formats:jar 1.2.1 (?) => 1.3.1 (?)
[INFO] io.prometheus:prometheus-metrics-tracer-common:jar 1.2.1 (?) => 1.3.1 (?)
[INFO] 
[INFO] Total of 0.00 years from 4 outdated dependencies
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS

[cstamas]

What combo did you try? As you have MRM but also a HTTP proxy in play... Could you maybe try different scenarios? Only proxy, only AF etc? Or is HTTP proxy required to reach AF?

[cstamas]

Doht, figured the problem... there are multiple ones:

• first, search-backend-rr have "extractors" only for these remotes: so Maven Central, and Nx2 (as ASF uses it). There is no AF extracor.
• second, remote repositories needs "hint" to select selector.
• if no selector, then all RR backends will end up here, hence this method will have empty list to iterate over.

Will try to figure out something.

[cstamas]

So, released 0.1.27 with improvement that should work for you. Here is what happens: despite there is no AF "extractor", the search use case libYear performs in fact does not need extractor (as it goes directly for artifact). Hence, -Dtoolbox.search.backend.type=central will "force" the central extractor for you, but again, it should work.

So command would look like this:

$ mvn eu.maveniverse.maven.plugins:toolbox:0.1.27:libyear -Dtoolbox.search.backend.type=central

For libYear use case it does not matter (while it does for Mojos like gav-list is).

[lfvjimisola]

It works! Thanks :pray:

From a user perspective; it's a bit unfortunate that the option -Dtoolbox.search.backend.type=central is needed since most users will "just run it" and get the impression that it does not work. Would it be possible for the tool to have some sort of auto-detect or fall-back making the use of the option unnecessary?

[lfvjimisola]

EDITED: coped in output from the non-working instead of the working one

[INFO] --- toolbox:0.1.27:libyear (default-cli) @ topsky-swim-feeder --- [...] [INFO] Outdated versions with known age [INFO] 0.08 years from org.springframework.boot:spring-boot-starter:jar 3.3.0 (2024-05-23) => 3.3.1 (2024-06-20) [age 0.00 years] [INFO] 0.08 years from org.springframework.boot:spring-boot-starter-log4j2:jar 3.3.0 (2024-05-23) => 3.3.1 (2024-06-20) [age 0.00 years] [INFO] 0.08 years from org.springframework.boot:spring-boot-starter-validation:jar 3.3.0 (2024-05-23) => 3.3.1 (2024-06-20) [age 0.00 years] [INFO] 0.08 years from org.springframework.boot:spring-boot-devtools:jar 3.3.0 (2024-05-23) => 3.3.1 (2024-06-20) [age 0.00 years] [INFO] 0.08 years from org.springframework.boot:spring-boot-configuration-processor:jar 3.3.0 (2024-05-23) => 3.3.1 (2024-06-20) [age 0.00 years] [INFO] 0.08 years from org.springframework.boot:spring-boot-starter-test:jar 3.3.0 (2024-05-23) => 3.3.1 (2024-06-20) [age 0.00 years] [INFO] 0.08 years from org.springframework.boot:spring-boot-starter-web:jar 3.3.0 (2024-05-23) => 3.3.1 (2024-06-20) [age 0.00 years] [INFO] 0.08 years from org.springframework.boot:spring-boot-starter-jetty:jar 3.3.0 (2024-05-23) => 3.3.1 (2024-06-20) [age 0.00 years] [INFO] 0.08 years from org.springframework.boot:spring-boot-starter-security:jar 3.3.0 (2024-05-23) => 3.3.1 (2024-06-20) [age 0.00 years] [INFO] 0.08 years from org.springframework.security:spring-security-test:jar 6.3.0 (2024-05-20) => 6.3.1 (2024-06-17) [age 0.00 years] [INFO] 0.08 years from org.springframework.boot:spring-boot-starter-hateoas:jar 3.3.0 (2024-05-23) => 3.3.1 (2024-06-20) [age 0.00 years] [INFO] 0.08 years from org.springframework.boot:spring-boot-starter-actuator:jar 3.3.0 (2024-05-23) => 3.3.1 (2024-06-20) [age 0.00 years] [INFO] 0.08 years from org.springframework.boot:spring-boot-starter-artemis:jar 3.3.0 (2024-05-23) => 3.3.1 (2024-06-20) [age 0.00 years] [INFO] 0.08 years from io.micrometer:micrometer-registry-prometheus:jar 1.13.0 (2024-05-13) => 1.13.1 (2024-06-10) [age 0.02 years] [INFO] 0.08 years from io.micrometer:micrometer-tracing:jar 1.3.0 (2024-05-13) => 1.3.1 (2024-06-11) [age 0.02 years] [INFO] 0.08 years from io.micrometer:micrometer-tracing-bridge-otel:jar 1.3.0 (2024-05-13) => 1.3.1 (2024-06-11) [age 0.02 years] [INFO] 0.08 years from io.micrometer:micrometer-tracing-test:jar 1.3.0 (2024-05-13) => 1.3.1 (2024-06-11) [age 0.02 years] [INFO] 0.06 years from org.eclipse.jetty.http2:jetty-http2-server:jar 12.0.9 (2024-05-03) => 12.0.10 (2024-05-30) [age 0.06 years] [INFO] 0.06 years from org.eclipse.jetty:jetty-alpn-conscrypt-server:jar 12.0.9 (2024-05-03) => 12.0.10 (2024-05-30) [age 0.06 years] [INFO] [INFO] [INFO] Total of 1.42 years from 19 outdated dependencies [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 4.699 s [INFO] Finished at: 2024-06-20T21:09:27+02:00 [INFO] ------------------------------------------------------------------------

[cstamas]

If this is used as part of project, then you can create file .mvn/maven.config that has this content:

-D
toolbox.search.backend.type=central

And check it into SCM. This way that project will have it "working" (as Maven will pick up conf from this file, see https://maven.apache.org/configure.html)

[cstamas]

What is the second reply? It is still ??

[lfvjimisola]

If this is used as part of project, then you can create file .mvn/maven.config that has this content:

-D
toolbox.search.backend.type=central

And check it into SCM. This way that project will have it "working" (as Maven will pick up conf from this file, see https://maven.apache.org/configure.html)

We have a team policy of really avoiding config files in our repos as it clutters the git repos and does not allow one to change the setting easy globally.

Can set it in the configuration section of the plugin, e.g.

<build>
    <plugins>
        <plugin>
            <groupId>eu.maveniverse.maven.plugins</groupId>
            <artifactId>toolbox</artifactId>
            <version>0.1.27</version>
            <executions>
                <execution>
                    <goals>
                        <goal>libyear</goal>
                    </goals>
                    <configuration>
                        <searchBackendType>central</searchBackendType>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Otherwise, we should just be able to set it in the pom (or tile in our case as we use maven-tiles) as a property correct?

<properties>
    <toolbox.search.backend.type>central</toolbox.search.backend.type>
</properties>

[cstamas]

Here is a PR that should solve it: https://github.com/maveniverse/toolbox/pull/57

Can set it in the configuration section of the plugin, e.g.

    <build>
        <plugins>
            <plugin>
                <groupId>eu.maveniverse.maven.plugins</groupId>
                <artifactId>toolbox</artifactId>
                <executions>
                    <execution>
                        <goals>
                            <goal>libyear</goal>
                        </goals>
                        <configuration>
                            <repositoryVendor>central</repositoryVendor>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

Or in project properties as:

    <properties>
        <toolbox.search.backend.type>central</toolbox.search.backend.type>
    </properties>

Any chance you can try the PR out?

[cstamas]

Tested locally and finally released as 0.1.28.

[lfvjimisola]

It was a (big) public holiday in Sweden yesterday (midsummer), but I just tested 0.1.28 with both plugin configuration and properties. It works! Thanks!

Will users find these settings (especially, the properties one)?

[cstamas]

Heh, well, Maveniverse is actually some "lab" or "skunworks" for Maven, so things done here may or may not end up in Maven proper project. Toolbox might not, as it would "compete" with m-dependency-p, but i want these functionalities available, and I like to have toolbox.

OTOH, version is intentionally 0.1.x, as all this, while functional, is pretty much hacked together, so not only documentation is needed to be done, but proper test coverage and so on... I plan to do it in a moment I have enough time to make it happen.

Btw, is this issue done then? Can you close it out?

[lfvjimisola]

Closed. Thanks. Now it's vacay time here. See you on the other side.

[lfvjimisola]

hi @cstamas I'm adding libyear to our CI/CD and noticed just now that it does not work due to 401 Unauth against JFrog Artifactory when trying to download two different maven-metadata.xml. I've confirmed that the two files exist. These are internal artifacts.

We have no auth issues with other plugins.

[cstamas]

I changed my mind: will close this issue and @lfvjimisola pls open new one... also add info which goal you exactly use, how is auth set up, etc Thanks!

[lfvjimisola]

Ok. My head is done for today. Will do it tomorrow. Thanks.

[cstamas]

In general (even in Maven) HTTP 401 is handled as "error" and will kill the build. So there is either some blank spot where remote repo is created (and not equipped with auth) or something AF specific maybe? Unsure yet

In fact, this handling is happening in maven-indexer (handles 401 as error), but I assume Toolbox misses some auth setup somewhere: https://github.com/apache/maven-indexer/blob/master/search-backend-remoterepository/src/main/java/org/apache/maven/search/backend/remoterepository/internal/RemoteRepositorySearchBackendImpl.java#L157-L166