Dependabot is unable to resolve a Maven property defined in a parent pom not located in Maven Central

[tmortagne]

Package ecosystem Maven Manifest location and content before the Dependabot update https://github.com/xwiki/xwiki-platform/blob/22029e026763ef50d45a3fbca3d3a20562ea6ec0/pom.xml#L23 dependabot.yml content The dependabot.yml file can be seen on https://github.com/xwiki/xwiki-platform/blob/master/.github/dependabot.yml What you expected to see, versus what you actually saw I expect Dependabot to use the repositories indicated in the dependabot.yml file to find parents and resolve the effective pom.

That being said, I'm actually surprised that Dependabot even cared about this property at all since it's only used for artifacts which are supposed to be ignored according to https://github.com/xwiki/xwiki-platform/blob/47779ad51f1b9bd5524fde52d6acb6e14ae3a1c3/.github/dependabot.yml#L44. I don't remember getting this error when we started using Dependabot on this repository (and we did not event indicated any repository in dependabot.yml back then), and we definitely used to get Dependabot pull requests on it (for example https://github.com/xwiki/xwiki-platform/pull/1884). Native package manager behavior Maven will search for parents in all the repositories indicated in ~/.m2/settings.xml Images of the diff or a link to the PR, issue, or logs 🕹 Bonus points: Smallest manifest that reproduces the issue I'm not really sure (since I'm not exactly sure for what it tried to resolve this property), but I guess it might be something like the following:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>org.xwiki.commons</groupId>
    <artifactId>xwiki-commons-pom</artifactId>
    <version>14.7-SNAPSHOT</version>
  </parent>
  <groupId>org.xwiki.platform</groupId>
  <artifactId>xwiki-platform</artifactId>
  <packaging>pom</packaging>
  <version>14.7-SNAPSHOT</version>
  <build>
    <extensions>
      <extension>
        <groupId>org.xwiki.commons</groupId>
        <artifactId>xwiki-commons-tool-webjar-handlers</artifactId>
        <version>${commons.version}</version>
      </extension>
    </extensions>
  </build>
</project>

[jakecoffman]

In the logs I see Dependabot is trying to use the credentials provided in dependabot.yml but is getting a 401:

updater | INFO <job_480391681> Starting update job for xwiki/xwiki-platform
  proxy | 2022/10/10 14:55:47 [134] GET https://nexus.xwiki.org:443/nexus/content/groups/public-snapshots/org/xwiki/commons/xwiki-commons-pom/14.9-SNAPSHOT/xwiki-commons-pom-14.9-SNAPSHOT.pom
  proxy | 2022/10/10 14:55:47 [134] * authenticating maven repository request (host: nexus.xwiki.org)
  proxy | 2022/10/10 14:55:47 [134] 401 https://nexus.xwiki.org:443/nexus/content/groups/public-

I suggest removing the registries from dependabot.yml since the registry is publicly accessible: https://nexus.xwiki.org/nexus/content/groups/public-snapshots/org/xwiki/commons/xwiki-commons-pom/14.9-SNAPSHOT/

I tried that locally and got a different result:

2022/10/12 02:25:55 [130] GET https://nexus.xwiki.org:443/nexus/content/groups/public-snapshots/org/xwiki/commons/xwiki-commons-pom/14.9-SNAPSHOT/xwiki-commons-pom-14.9-SNAPSHOT.pom
2022/10/12 02:25:55 [130] 404 https://nexus.xwiki.org:443/nexus/content/groups/public-snapshots/org/xwiki/commons/xwiki-commons-pom/14.9-SNAPSHOT/xwiki-commons-pom-14.9-SNAPSHOT.pom

It seems we've hardcoded the naming scheme here, so we'll have to update that code to pull the maven-metadata.xml to figure out which snapshot to download.

I'm currently doing some improvements to Maven so I'll try to take this work on soon.

[tmortagne]

I suggest removing the registries from dependabot.yml since the registry is publicly accessible

Well that was obvious to me too, and it was the first version, but Dependabot was complaining about the required username and password properties. Unless you mean something else than removing the credentials (I don't see how Dependabot is going to find those repositories if I completely remove them from the configuration, since indicating those was the only point of those registries).

[jakecoffman]

Dependabot will parse the pom.xml and gathers these registries and use them to find the parent pom.

Currently the dependabot.yml registries are only used for injecting credentials into the request which is why username and password are required. Even if it is provided with valid credentials it won't actually query the registry provided unless they exist in a pom.xml repository setting.

This does seem pretty confusing, I'll try to see if I can improve this. Thanks for the help in figuring this out!

[tmortagne]

Dependabot will parse the pom.xml and gathers these registries and use them to find the parent pom.

You sure it will ? Those are only defined when a specific profile is enabled.

The only reason why I added those registries in Dependabot configuration is that Dependabot was complaining about missing parent artifacts that were available on those repositories, so it made sense to me to try to find a configuration to give them to Dependabot. But I guess that's the resolution bug you were mentioning ?

[jakecoffman]

You sure it will ? Those are only defined when a specific profile is enabled.

Keep in mind Dependabot doesn't use the maven executable at the moment, it's all done in Ruby code. The RepositoriesFinder code doesn't appear to try to figure out the active profile.

The only reason why I added those registries in Dependabot configuration is that Dependabot was complaining about missing parent artifacts that were available on those repositories, so it made sense to me to try to find a configuration to give them to Dependabot. But I guess that's the resolution bug you were mentioning ?

Yes it gets a 404 since it got the parent pom name wrong and failed the job.

I completely understand the confusion here and why you added registries to the dependabot.yml. I'll see if I can make what you tried actually work since other Maven users probably will think the same thing, and it makes sense!

[jakecoffman]

I made a number of improvements to how Dependabot works with Maven. I ran the xwiki update locally with these changes and it seems to be working!

If you see any other issues with this feel free to reopen.

[tmortagne]

Thanks !

[tmortagne]

@jakecoffman it does not seems like I can reopen but I tried to run dependabot on https://github.com/xwiki/xwiki-platform and it's failing for something else now:

  proxy | time="2022-10-21T07:13:23Z" level=info msg="proxy starting" commit=4d808903b152d6796377f4c0f871735a93f27b90
  proxy | 2022/10/21 07:13:23 Listening (:1080)
updater | 2022-10-21T07:13:23.305552667 [anonymous-instance:main:WARN:src/firecracker/src/main.rs:370] You are using a deprecated parameter: --seccomp-level 2, that will be removed in a future version.
updater | 2022-10-21T07:13:23.367544890 [489066706:main:WARN:src/devices/src/legacy/serial.rs:432] Detached the serial input due to peer close/error.
updater | time="2022-10-21T07:13:26Z" level=info msg="guest starting" commit=d1d6583d3c5b9180b328be5dd89c5a6c65767abb
updater | time="2022-10-21T07:13:26Z" level=info msg="starting job..." fetcher_timeout=5m0s job_id=489066706 updater_timeout=45m0s updater_version=b9705ab5e242d289cee5066c6ff27490539a2ea8
updater | I, [2022-10-21T07:13:28.835524 #7]  INFO -- sentry: ** [Raven] Raven 3.1.2 ready to catch errors
updater | To use retry middleware with Faraday v2.0+, install `faraday-retry` gem
updater | INFO <job_489066706> Starting job processing
  proxy | 2022/10/21 07:13:32 [002] GET https://api.github.com:443/repos/xwiki/xwiki-platform
  proxy | 2022/10/21 07:13:32 [002] * authenticating github api request
  proxy | 2022/10/21 07:13:32 [002] 200 https://api.github.com:443/repos/xwiki/xwiki-platform
  proxy | 2022/10/21 07:13:32 [004] GET https://api.github.com:443/repos/xwiki/xwiki-platform/git/refs/heads/master
  proxy | 2022/10/21 07:13:32 [004] * authenticating github api request
  proxy | 2022/10/21 07:13:32 [004] 200 https://api.github.com:443/repos/xwiki/xwiki-platform/git/refs/heads/master
  proxy | 2022/10/21 07:13:32 [006] GET https://api.github.com:443/repos/xwiki/xwiki-platform/contents/pom.xml?ref=48caf7491595238af2b531026a614221d5d61f38
  proxy | 2022/10/21 07:13:32 [006] * authenticating github api request
  proxy | 2022/10/21 07:13:32 [006] 200 https://api.github.com:443/repos/xwiki/xwiki-platform/contents/pom.xml?ref=48caf7491595238af2b531026a614221d5d61f38

[...]

  proxy | 2022/10/21 07:15:01 [146] GET https://nexus.xwiki.org:443/nexus/content/groups/public-snapshots/org/xwiki/commons/xwiki-commons/14.10-SNAPSHOT/xwiki-commons-14.10-20221020.185047-5.pom
  proxy | 2022/10/21 07:15:01 [146] 200 https://nexus.xwiki.org:443/nexus/content/groups/public-snapshots/org/xwiki/commons/xwiki-commons/14.10-SNAPSHOT/xwiki-commons-14.10-20221020.185047-5.pom
updater | ERROR <job_489066706> {"errors":[{"status":500,"title":"Internal Server Error"}]}
updater | ERROR <job_489066706> /home/dependabot/dependabot-updater/lib/dependabot/api_client.rb:129:in `update_dependency_list'
updater | ERROR <job_489066706> /home/dependabot/dependabot-updater/lib/dependabot/service.rb:19:in `update_dependency_list'
updater | ERROR <job_489066706> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:992:in `update_dependency_list'
updater | ERROR <job_489066706> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:606:in `dependencies'
updater | ERROR <job_489066706> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:74:in `run'
updater | ERROR <job_489066706> /home/dependabot/dependabot-updater/lib/dependabot/update_files_job.rb:17:in `perform_job'
updater | ERROR <job_489066706> /home/dependabot/dependabot-updater/lib/dependabot/base_job.rb:50:in `run'
updater | ERROR <job_489066706> bin/update_files.rb:23:in `<main>'
updater | I, [2022-10-21T07:15:42.473433 #7]  INFO -- sentry: ** [Raven] Sending event 9b40d20e38b64096b1bfb015bd428066 to Sentry
  proxy | 2022/10/21 07:15:42 [150] POST https://sentry.io:443/api/1451818/store/
  proxy | 2022/10/21 07:15:42 [150] 200 https://sentry.io:443/api/1451818/store/
updater | INFO <job_489066706> Finished job processing
updater | INFO Results:
updater | Dependabot encountered '1' error(s) during execution, please check the logs for more details.
updater | time="2022-10-21T07:15:43Z" level=info msg="task complete" container_id=job-489066706-updater exit_code=0 job_id=489066706 step=updater

[jakecoffman]

That's a bummer. It seems Dependabot is failing to record the large amount of manifests in the XWiki project. Dependabot keeps track of the manifest paths to provide extra functionality like starting a run when a manifest changes to pick up added/removed dependencies.

We also are tracking this internally and figuring out how to solve it without degrading functionality.

[tmortagne]

OK, thanks for the debug @jakecoffman !

[tmortagne]

xwiki-platform is still failing, but the error is slightly different (I guess it's just that the code changed a bit it's still the same root cause)

  proxy | 2023/09/26 07:21:14 [488] 200 https://nexus.xwiki.org:443/nexus/content/groups/public-snapshots/org/xwiki/commons/xwiki-commons/15.9-SNAPSHOT/xwiki-commons-15.9-20230925.182935-7.pom
updater | 2023/09/26 07:21:48 ERROR <job_726482763> {"errors":[{"status":500,"title":"Internal Server Error"}]}
updater | 2023/09/26 07:21:48 ERROR <job_726482763> /home/dependabot/dependabot-updater/lib/dependabot/api_client.rb:132:in `update_dependency_list'
updater | 2023/09/26 07:21:48 ERROR <job_726482763> /home/dependabot/dependabot-updater/lib/dependabot/service.rb:67:in `update_dependency_list'
updater | 2023/09/26 07:21:48 ERROR <job_726482763> /home/dependabot/dependabot-updater/lib/dependabot/update_files_command.rb:29:in `perform_job'
updater | 2023/09/26 07:21:48 ERROR <job_726482763> /home/dependabot/dependabot-updater/lib/dependabot/base_command.rb:53:in `run'
updater | 2023/09/26 07:21:48 ERROR <job_726482763> bin/update_files.rb:24:in `<main>'
updater | 2023/09/26 07:21:48 INFO <job_726482763> Sending event d61037ebe77143cc871da59040aeba15 to Sentry
  proxy | 2023/09/26 07:21:48 [492] POST https://sentry.io:443/api/1451818/store/
  proxy | 2023/09/26 07:21:48 [492] 200 https://sentry.io:443/api/1451818/store/
updater | 2023/09/26 07:21:49 INFO Results:
updater | Dependabot encountered '1' error(s) during execution, please check the logs for more details.
updater | +---------------+
updater | |    Errors     |
updater | +---------------+
updater | | updater_error |
updater | +---------------+

Do you think it would be possible to tell dependabot to only check specific poms ? Most of our dependencies versions are actually defined in the root pom, and it won't find much in others, so maybe that would help (better than nothing at least :) ).