search

fkie-cad / nvd-json-data-feeds

Community reconstruction of the legacy JSON NVD Data Feeds. This project uses and redistributes data from the NVD API but is neither endorsed nor certified by the NVD.
114 stars 16 forks source link

Duplicate and invalid CVE-2014-0160 #10

Closed Phenix66 closed 9 months ago

Phenix66 commented 9 months ago

CVE-2014-0160 appears to be duplicated in recent releases and in one of the duplicates, the publish date is the epoch. 

"id": "CVE-2014-0160",
"sourceIdentifier": "secalert@redhat.com",
"published": "1970-01-01T00:00:00.000+00:00",

The source file, https://github.com/fkie-cad/nvd-json-data-feeds/blob/main/CVE-2014/CVE-2014-01xx/CVE-2014-0160.json, appears to be fine but there are 9 matches in CVE-2014.json when I download from it from releases and decompress it.

I also checked the NIST API at "https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2014-0160" to ensure the issue wasn't on their end and I only received the single, valid entry in response.

rhelmke commented 9 months ago

Oof. Strange. I'll investigate. Thanks for putting this to our attention.

rhelmke commented 9 months ago

Okay, I got sweaty palms there for a moment because that smelled like cache corruption.

First things first: I found the culprit and the next release will no longer return duplicate and/or invalid CVEs. The cache is not corrupted and I checked the whole dataset for consistency.

Now lets talk about what happened. We use Opensearch to cache NVD responses. CVEs get overwritten by published and lastModified dates. At some point, the NVD did not return any published timestamp for this CVE, which is why our tool inserted a default value and pushed it to Opensearch. Opensearch noticed that the published date is older, hence it did not overwrite the original record but created a new one. Our code to generate the release packages didn't expect this case and aggregated all documents without deduplication.

However, i'm not yet quite sure why multiple duplicates got created. I'll look into that tomorrow.

Thanks for the report and sorry for the inconvenience :-)

Edit: Also, I have no idea why it must always be Heartbleed..