search

google / osv-scanner

Vulnerability scanner written in Go which uses the data provided by https://osv.dev
https://google.github.io/osv-scanner/
Apache License 2.0
6.27k stars 363 forks source link

Failures when scanning pom.xml files with a <parent> tag #1321

Open LironJit opened 1 month ago

LironJit commented 1 month ago

Hey osv team, I've been running into some issues with scanning pom.xml files which have a <parent> tag:

osv-scanner v1.7.3 and latest (v1.9.0) are unable to scan this repository: https://github.com/javaee/javamail but after deleting the <parent> tags from pom.xml files (for example javamail/taglib/pom.xml) the files are scanned successfully.

Attempted to scan lockfile but failed: javamail/android/activation/pom.xml
Attempted to scan lockfile but failed: javamail/android/mail/pom.xml
Attempted to scan lockfile but failed: javamail/android/pom.xml
Attempted to scan lockfile but failed: javamail/client/pom.xml
Attempted to scan lockfile but failed: javamail/demo/pom.xml
Attempted to scan lockfile but failed: javamail/dsn/pom.xml
Attempted to scan lockfile but failed: javamail/gimap/pom.xml
Attempted to scan lockfile but failed: javamail/imap/pom.xml
Attempted to scan lockfile but failed: javamail/javadoc/pom.xml
Attempted to scan lockfile but failed: javamail/logging/pom.xml
Attempted to scan lockfile but failed: javamail/mail/pom.xml
Attempted to scan lockfile but failed: javamail/mailapi/pom.xml
Attempted to scan lockfile but failed: javamail/mailapijar/pom.xml
Attempted to scan lockfile but failed: javamail/mailhandler/pom.xml
Attempted to scan lockfile but failed: javamail/mbox/native/pom.xml
Attempted to scan lockfile but failed: javamail/mbox/pom.xml
Attempted to scan lockfile but failed: javamail/outlook/pom.xml
Attempted to scan lockfile but failed: javamail/parent-distrib/pom.xml
Attempted to scan lockfile but failed: javamail/pom.xml
Attempted to scan lockfile but failed: javamail/pop3/pom.xml
Attempted to scan lockfile but failed: javamail/publish/pom.xml
Attempted to scan lockfile but failed: javamail/servlet/pom.xml
Attempted to scan lockfile but failed: javamail/smtp/pom.xml
Attempted to scan lockfile but failed: javamail/taglib/pom.xml
Attempted to scan lockfile but failed: javamail/webapp/pom.xml

Also notice when I'm using v1.7.3 I'm able to scan the following (which has a <parent> tag):

✗ docker run --rm -v $(pwd):/code ghcr.io/google/osv-scanner:v1.7.3 scan -r /code/
Scanning dir /code/
Scanned /code/pom.xml file and found 1 package
+-------------------------------------+------+-----------+-------------------------+---------+--------------+
| OSV URL                             | CVSS | ECOSYSTEM | PACKAGE                 | VERSION | SOURCE       |
+-------------------------------------+------+-----------+-------------------------+---------+--------------+
| https://osv.dev/GHSA-5gj6-62g7-vmgf | 4.3  | Maven     | com.hazelcast:hazelcast | 5.1     | code/pom.xml |
| https://osv.dev/GHSA-8h4x-xvjp-vf99 | 6.5  | Maven     | com.hazelcast:hazelcast | 5.1     | code/pom.xml |
| https://osv.dev/GHSA-c5hg-mr8r-f6jp | 9.1  | Maven     | com.hazelcast:hazelcast | 5.1     | code/pom.xml |
| https://osv.dev/GHSA-c5vj-wp4v-mmvx | 7.6  | Maven     | com.hazelcast:hazelcast | 5.1     | code/pom.xml |
| https://osv.dev/GHSA-xh6m-7cr7-xx66 | 7.6  | Maven     | com.hazelcast:hazelcast | 5.1     | code/pom.xml |
+-------------------------------------+------+-----------+-------------------------+---------+--------------+
<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.apache.seatunnel</groupId>
        <artifactId>seatunnel-e2e</artifactId>
        <version>${revision}</version>
    </parent>

    <properties>
        <hazelcast.version>5.1</hazelcast.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>com.hazelcast</groupId>
            <artifactId>hazelcast</artifactId>
            <version>${hazelcast.version}</version>
        </dependency>
    </dependencies>
</project>

However, running v1.9.0 will not work unless I remove the <parent> tag:

✗ docker run --rm -v $(pwd):/code ghcr.io/google/osv-scanner:v1.9.0 scan -r /code/
Scanning dir /code/
Attempted to scan lockfile but failed: /code/pom.xml
<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <properties>
        <hazelcast.version>5.1</hazelcast.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>com.hazelcast</groupId>
            <artifactId>hazelcast</artifactId>
            <version>${hazelcast.version}</version>
        </dependency>
    </dependencies>
</project>

Thanks in advance!

oliverchang commented 1 month ago

Thanks for the report! @cuixq can you please take a look at this?

cuixq commented 1 month ago

@LironJit we launched transitive scanning for Maven pom.xml in version 1.8.1 which makes network requests to fetch parent POMs.

When I investigated this issue I noticed the following error:

xml: encoding "iso-8859-1" declared but Decoder.CharsetReader is nil

and I am going to make a fix for this.

However this issue still stops the scanner working for pom.xml without <parent> tag.

Could you give us more information on the error you see when running the scanner? Thanks.

desolatorxxl commented 1 month ago

I have the same problem and would like to help, however I'm not sure how.

For starters I added the parent error to Attempted to scan lockfile but failed here:

r.Errorf("Attempted to scan lockfile but failed: %s: %s\n", err, path)

Now I get:

$ go run . --verbosity verbose -skip-git -r myservice
Scanning dir myservice
Attempted to scan lockfile but failed: failed extracting myservice/api-mock/pom.xml: failed to merge parents: failed to get Maven project com.company.commons:maven.artifactory:2.0: failed to fetch Maven project com.company.commons:maven.artifactory@2.0: myservice/api-mock/pom.xml
Attempted to scan lockfile but failed: failed extracting myservice/myservice/pom.xml: failed to merge parents: failed to get Maven project com.company.commons:maven.artifactory:2.0: failed to fetch Maven project com.company.commons:maven.artifactory@2.0: myservice/myservice/pom.xml
Attempted to scan lockfile but failed: failed extracting myservice/myservice-st/pom.xml: failed to merge parents: failed to get Maven project com.company.commons:maven.artifactory:2.0: failed to fetch Maven project com.company.commons:maven.artifactory@2.0: myservice/myservice-st/pom.xml
Attempted to scan lockfile but failed: failed extracting myservice/pom.xml: failed to merge parents: failed to get Maven project com.company.commons:maven.artifactory:2.0: failed to fetch Maven project com.company.commons:maven.artifactory@2.0: myservice/pom.xml
No package sources found, --help for usage information.
exit status 128

(Just cloned osv-scanner from master at e054385a544da5981185fee64165c4d2ea4d889c)

Which seems to be related to osv-scanner not resolving artifacts from private artifact registries.

However if I try the javamail project, I get some different errors:

$ go run . --verbosity verbose -skip-git -r ./javamail
Scanning dir ./javamail
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/android/activation/pom.xml file and found 0 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/android/mail/pom.xml file and found 1 package
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/android/pom.xml file and found 0 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/client/pom.xml file and found 2 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/demo/pom.xml file and found 2 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/dsn/pom.xml file and found 2 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/gimap/pom.xml file and found 2 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/imap/pom.xml file and found 1 package
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/javadoc/pom.xml file and found 4 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/logging/pom.xml file and found 2 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/mail/pom.xml file and found 1 package
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/mailapi/pom.xml file and found 0 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/mailapijar/pom.xml file and found 0 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/mailhandler/pom.xml file and found 1 package
Attempted to scan lockfile but failed: failed extracting /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/mbox/native/pom.xml: failed resolving {Maven:com.sun.mail:libmbox[Concrete:1.6.2] {}}: version Maven:com.sun.mail:mbox[Concrete:1.6.2]: not found: /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/mbox/native/pom.xml
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/mbox/pom.xml file and found 2 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/outlook/pom.xml file and found 2 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/parent-distrib/pom.xml file and found 1 package
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/pom.xml file and found 0 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/pop3/pom.xml file and found 1 package
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/publish/pom.xml file and found 2 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/servlet/pom.xml file and found 3 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/smtp/pom.xml file and found 1 package
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/taglib/pom.xml file and found 4 packages
Attempted to scan lockfile but failed: failed extracting /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/webapp/pom.xml: failed resolving {Maven:com.sun.mail:webapp[Concrete:1.6.2] {}}:
version Maven:com.sun.mail:taglib[Concrete:1.6.2]: not found: /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/webapp/pom.xml
exit status 127
cuixq commented 1 month ago

@desolatorxxl thanks for the feedback!

May I ask where do you place your parent pom.xml? On local file system or remote private registry? OSV-Scanner at master should be able to track the registries defined in <repositories/> - for your case, where do you specify the registry URL?

desolatorxxl commented 1 month ago

@cuixq all pom.xml files are on a local file system. For clarity the directory structure looks like this:

$ find -name pom.xml
./pom.xml
./api-mock/pom.xml
./myservice/pom.xml
./myservice-st/pom.xml

But the ./pom.xml has, what I believe is, a remote parent, see further below.

I specified the registry URL in my ~/.m2/settings.xml, it looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<settings xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd" xmlns="http://maven.apache.org/SETTINGS/1.1.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <servers>
    <server>
      <username>...</username>
      <password>...</password>
      <id>central</id>
    </server>
    <server>
      <username>...</username>
      <password>...</password>
      <id>snapshots</id>
    </server>
  </servers>
  <profiles>
    <profile>
      <repositories>
        <repository>
          <snapshots>
            <enabled>false</enabled>
          </snapshots>
          <id>central</id>
          <name>libs-release</name>
          <url>https://artifactory.org.com:443/artifactory/libs-release</url>
        </repository>
        <repository>
          <snapshots />
          <id>snapshots</id>
          <name>libs-snapshot</name>
          <url>https://artifactory.org.com:443/artifactory/libs-snapshot</url>
        </repository>
      </repositories>
      <pluginRepositories>
        <pluginRepository>
          <snapshots>
            <enabled>false</enabled>
          </snapshots>
          <id>central</id>
          <name>libs-release</name>
          <url>https://artifactory.org.com:443/artifactory/libs-release</url>
        </pluginRepository>
        <pluginRepository>
          <snapshots />
          <id>snapshots</id>
          <name>libs-snapshot</name>
          <url>https://artifactory.org.com:443/artifactory/libs-snapshot</url>
        </pluginRepository>
      </pluginRepositories>
      <id>artifactory</id>
    </profile>
  </profiles>
  <activeProfiles>
    <activeProfile>artifactory</activeProfile>
  </activeProfiles>
</settings>

I digged around a bit in all the pom.xml files and found out that the top level pom (./pom.xml) references a remote parent, like this:

    <parent>
        <groupId>com.org.common</groupId>
        <artifactId>maven.artifactory</artifactId>
        <version>1.0</version>
    </parent>

The remote pom.xml contains a slightly different incarnation of my settings.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.org.common</groupId>
    <artifactId>maven.artifactory</artifactId>
    <version>1.0</version>
    <packaging>pom</packaging>
    <description>ORG POM for Artifactory</description>
    <organization>
        <name>Private ORG</name>
        <url>https://org.com</url>
    </organization>
    <distributionManagement>
        <repository>
            <id>central</id>
            <name>libs-release</name>
            <url>https://artifactory.org.com:443/artifactory/libs-release-local</url>
        </repository>
        <snapshotRepository>
            <id>snapshots</id>
            <name>libs-snapshot</name>
            <url>https://artifactory.org.com:443/artifactory/libs-snapshot-local</url>
        </snapshotRepository>
    </distributionManagement>
</project>

If I remove that parent, the scan runs successfully.

It seems that is a really odd way of centrally managing repository configuration?

Anyways, today that artifact is not required for the application to run or build, so I will try to get it removed.

cuixq commented 4 weeks ago

@desolatorxxl we currently do not support reading repositories from settings.xml and we have a issue opened for this https://github.com/google/osv-scanner/issues/1269.