search

jeremylong / Open-Vulnerability-Project

Java libraries for working with available vulnerability data sources (GitHub Security Advisories, NVD, EPSS, CISA Known Exploited Vulnerabilities, etc.)
Apache License 2.0
112 stars 34 forks source link

vulnz is failing cve caching from NVD due to an introduced 'cveTags' property #150

Closed jessebluemr closed 6 months ago

jessebluemr commented 6 months ago

I use vulnz to cache the NVD Data, since today morning this job runs endlessly.

In the debug log I observed that the same URL is fetched again and again:

requesting URI: https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-03-19T19%3A15%3A07Z&lastModEndDate=2024-07-17T19%3A15%3A07Z&resultsPerPage=2000&startIndex=0
Ticket taken At: 15:08:08; count: 1; by 26
Requested At: 15:08:08; URI: /rest/json/cves/2.0?lastModStartDate=2024-03-19T19%3A15%3A07Z&lastModEndDate=2024-07-17T19%3A15%3A07Z&resultsPerPage=2000&startIndex=0
Ticket returned At: 15:08:18; count: 2; by 26
Content-Type Received: application/json
requesting URI: https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-03-19T19%3A15%3A07Z&lastModEndDate=2024-07-17T19%3A15%3A07Z&resultsPerPage=2000&startIndex=0
Ticket taken At: 15:08:18; count: 2; by 26
Requested At: 15:08:18; URI: /rest/json/cves/2.0?lastModStartDate=2024-03-19T19%3A15%3A07Z&lastModEndDate=2024-07-17T19%3A15%3A07Z&resultsPerPage=2000&startIndex=0
Ticket returned At: 15:09:01; count: 3; by 26
Content-Type Received: application/json
requesting URI: https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-03-19T19%3A15%3A07Z&lastModEndDate=2024-07-17T19%3A15%3A07Z&resultsPerPage=2000&startIndex=0
Ticket taken At: 15:09:01; count: 3; by 26

...

Downloading the cve data https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-03-19T19%3A15%3A07Z&lastModEndDate=2024-07-17T19%3A15%3A07Z&resultsPerPage=2000&startIndex=0

And comparing it to the https://github.com/jeremylong/Open-Vulnerability-Project/blob/main/open-vulnerability-clients/src/test/resources/nvd.json

It seems that a new cveTags property was added:

   {
    "resultsPerPage": 2000,
    "startIndex": 0,
    "totalResults": 4439,
    "format": "NVD_CVE",
    "version": "2.0",
    "timestamp": "2024-03-21T15:53:35.133",
    "vulnerabilities": [
        {
            "cve": {
                "id": "CVE-2001-1517",
                "sourceIdentifier": "cve@mitre.org",
                "published": "2001-12-31T05:00:00.000",
                "lastModified": "2024-03-21T02:12:33.387",
                "vulnStatus": "Modified",
                "cveTags": [
                    {
                        "sourceIdentifier": "cve@mitre.org",
                        "tags": [
                            "disputed"
                        ]
                    }
                ],
                "descriptions": [ ... ]

This cveTags property is not described in https://github.com/jeremylong/Open-Vulnerability-Project/blob/main/open-vulnerability-clients/src/main/resources/json/cve_api_json_2.0.schema

In Line https://github.com/jeremylong/Open-Vulnerability-Project/blob/5988546bfa6c62d7342f2e583ba7e11882e5bdee/open-vulnerability-clients/src/main/java/io/github/jeremylong/openvulnerability/client/nvd/NvdCveClient.java#L341 the parsing error is not logged and a new request for the same data is issued again. The code endlessly repeats downloading the same data.

Logging in a test reports:

com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "cveTags" (class io.github.jeremylong.openvulnerability.client.nvd.CveItem), not marked as ignorable (18 known properties: "metrics", "sourceIdentifier", "evaluatorSolution", "cisaRequiredAction", "cisaExploitAdd", "published", "evaluatorComment", "vulnStatus", "evaluatorImpact", "descriptions", "id", "vendorComments", "weaknesses", "lastModified", "references", "cisaActionDue", "cisaVulnerabilityName", "configurations"])
     at [Source: (StringReader); line: 16, column: 17] (through reference chain: io.github.jeremylong.openvulnerability.client.nvd.CveApiJson20["vulnerabilities"]->java.util.ArrayList[0]->io.github.jeremylong.openvulnerability.client.nvd.DefCveItem["cve"]->io.github.jeremylong.openvulnerability.client.nvd.CveItem["cveTags"])
        at app//com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
        at app//com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1138)
        at app//com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:2224)
        at app//com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1709)
        at app//com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1687)
        at app//com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:320)
        at app//com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
        at app//com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
        at app//com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314)
        at app//com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
        at app//com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:359)
        at app//com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244)
        at app//com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28)
        at app//com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
        at app//com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314)
        at app//com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
        at app//com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
        at app//com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4825)
        at app//com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3772)
        at app//com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3740)

I'm not so firm with gradle, which makes it hard to create a pull request for me. But maybe this report helps others to locate the problem, too.