Support snapshot artifacts when fetching Maven parent POMs

[Malayke]

Hello,

I encountered an issue while performing a security scan on one of my Java projects using osv-scanner. The scan failed, and the error output is as follows:

❯ ./osv-scanner -skip-git -r $HOME/gitlab/abcd/broker/my-kyc-center
Scanning dir /Users/user/gitlab/abcd/broker/my-kyc-center
Attempted to scan lockfile but failed: /Users/user/gitlab/abcd/broker/my-kyc-center/kyc-api/pom.xml
Attempted to scan lockfile but failed: /Users/user/gitlab/abcd/broker/my-kyc-center/kyc-common/pom.xml
Attempted to scan lockfile but failed: /Users/user/gitlab/abcd/broker/my-kyc-center/kyc-job/pom.xml
Attempted to scan lockfile but failed: /Users/user/gitlab/abcd/broker/my-kyc-center/kyc-persist/pom.xml
Attempted to scan lockfile but failed: /Users/user/gitlab/abcd/broker/my-kyc-center/kyc-service/pom.xml
Scanning directory for vendored libs: /Users/user/gitlab/abcd/broker/my-kyc-center/kyc-service/src/main/java/com/my/kyc/service/vendor
Scanning potential vendored dir: /Users/user/gitlab/abcd/broker/my-kyc-center/kyc-service/src/main/java/com/my/kyc/service/vendor/model
Scanning potential vendored dir: /Users/user/gitlab/abcd/broker/my-kyc-center/kyc-service/src/main/java/com/my/kyc/service/vendor/security
Attempted to scan lockfile but failed: /Users/user/gitlab/abcd/broker/my-kyc-center/kyc-web/pom.xml
Attempted to scan lockfile but failed: /Users/user/gitlab/abcd/broker/my-kyc-center/pom.xml
No package sources found, --help for usage information.

After debugging the source code, I obtained more detailed error information:

"failed extracting /Users/user/gitlab/abcd/broker/my-kyc-center/kyc-api/pom.xml: failed to merge parents: failed to get Maven project com.my.kyc:kyc-center:2.0.0-SNAPSHOT: API query failed: Maven registry query status: 404 Not Found"

The directory /Users/user/gitlab/abcd/broker/my-kyc-center contains a multi-module Maven project. Through debugging, I think I discovered the following two points causing the scan failure:

1. Hardcoded Maven Central URL: In the file maven_registry.go, the Maven Central URL is hardcoded. Since my project is a private project and only submitted to the internal Nexus repository, directly querying the Maven Central results in a query failure and an error. I modified const MavenCentral to the internal Nexus URL and continued running, but found the same error, which led to discovering the second issue.

2. Snapshot Version Handling: When requesting the Maven repository for snapshot versions, the maven-metadata.xml file should be requested first. For example, the current GetProject function in the maven_registry.go file directly requests:

   https://nexus/repository/abcd-prod-proxy/com/my/kyc/kyc-center/2.0.0-SNAPSHOT/kyc-center-2.0.0-SNAPSHOT.pom

   This results in a 404 not found error because if there are multiple submissions of 2.0.0-SNAPSHOT, the POM file cannot be directly obtained. Instead, the maven-metadata.xml file at:

   https://nexus/repository/abcd-prod-proxy/com/my/kyc/kyc-center/2.0.0-SNAPSHOT/maven-metadata.xml

   should be read for the snapshotVersion, for example:

   <snapshotVersions>
    <snapshotVersion>
      <extension>pom</extension>
      <value>2.0.0-20240711.060746-8</value>
      <updated>20240711060746</updated>
    </snapshotVersion>
   </snapshotVersions>

   Then, the correct POM file can be requested:

   https://nexus/repository/abcd-prod-proxy/com/my/kyc/kyc-center/2.0.0-SNAPSHOT/kyc-center-2.0.0-20240711.060746-8.pom

   Environment Information:

   Operating System: macOS 14.5 Java Version: openjdk version "11.0.21" Maven Version: Apache Maven 3.9.6

Please let me know if you need any additional information or if there are any other steps I can take to assist in resolving this issue.

Thank you for your attention to this matter!

[another-rex]

Thanks for the detailed report! Can you also add the osv-scanner version you were using as well? @cuixq can you take a look?

[cuixq]

I think this is related to this feature request - supporting private registry for Maven.

For the snapshot versions in mavne-metadata.xml, I think this is for snapshot remote repositories based on the reference. I will rename this bug to reflect supporting consuming snapshot artifacts.

[cuixq]

@Malayke do you know any documentation or more examples about this? The link on this page returns 404 :(

[Malayke]

Hi @cuixq , you can also access this link, you find the right document, I discovered this mechanism independently by frequently running mvn dependency:tree. If the version includes snapshot, Maven first accesses maven-metadata.xml. Otherwise, it directly accesses the POM file. You can find an explanation of this feature here.

[Malayke]

Hi @cuixq I was wondering if I might kindly inquire about the possibility of supporting a custom Maven Central URL?

[cuixq]

Hi @Malayke do you mean a custom Maven registry URL instead of the hard-coded Maven Central URL? We definitely would like to support that.

[Malayke]

yes, thank you for your previous response! I appreciate your work on supporting snapshot version fetching from Maven registries. It's a great feature!

I was wondering if it might be possible to consider adding support for custom Maven registry URLs, rather than using a hard-coded Maven Central URL? I noticed this wasn't mentioned in the current implementation, which is why I thought to ask about it.

I remember bringing this up in this issue, and I'm curious if there have been any thoughts or plans regarding this feature. It would be incredibly helpful for those of us who might need to use alternative Maven repositories.

I completely understand if this isn't currently on the roadmap, but I'd be grateful for any insights you could share on this topic. Thank you so much for your time and for considering my question!

[cuixq]

Yes - supporting Maven private registries https://github.com/google/osv-scanner/issues/1045 is one of our priorities. We do plan to make the Maven repository URL configurable. I will update the mentioned bug with more details. Stay tuned!