CVEs missing in deltas

[lsdijk]

I downloaded the following files from the releases area:

2023-10-15_all_CVEs_at_midnight.zip.zip 2023-10-16_all_CVEs_at_midnight.zip.zip 2023-10-15_delta_CVEs_at_xx00Z.zip, where xx runs from 00 through 23 - i.e. 24 zip files 2023-10-15_delta_CVEs_at_end_of_day.zip

I then unzipped all those files and proceeded to apply the deltas in each of the 25 files (24 hourly ones, plus the end-of-day one) to the 10/15 midnight snapshot (just snapshot henceforth). After doing that, I compared the contents of the 10/15 snapshot with those of the 10/16. I thought that, after applying all the deltas in the 25 delta files to the 10/15 snapshot its contents would be identical to those of the 10/16 snapshot.

However, they are not. For example, there is a file called CVE-2023-5591.json under cves/2023/5xxx in the 10/16 snapshot which is not present in the 10/15 snapshot after (or before, at that) applying the deltas. Looking into the deltas for 10/15 themselves, CVE-2023-5591.json is also not present in any of them: in the directory obtained from 2023-10-15_delta_CVEs_at_end_of_day.zip the last file is CVE-2023-5590.json.

I have noticed a similar behavior downloading the corresponding files for different dates: for the most part there will be differences between the midnight snapshot on a given day, with all of the 25 deltas applied, and the midnight snapshot for the next day; it is only occasionally that they both are identical.

Any idea what is going on here? At what point during 10/15 was CVE-2023-5591.json added? Am I missing something?

[hkong-mitre]

@lsdijk, as nearly as I could tell, there is a race condition or timing issue in how these zip files are built that needs to be addressed.

In the meantime, as a temporary stop gap solution is it possible for your workflow to do one of the following to circumvent this problem for now?

• use git instead of the zip files. This is the fastest and most efficient approach—among the benefits is that you can issue the git pull command anytime to get all updates instead of waiting for hourly zip files
• use the cves/deltaLog.json file with a REST client. This provides similar benefits to the previous, but lets you use a library and language of your choice
• use the midnight build from the next day—the midnight build is a complete zip of the entire cves directory. This would mean, of course, that the most recent additions and changes are delayed 24 hours on your local machine.

[lsdijk]

Thanks. Hopefully you guys will fix this issue, for the deltas mechanism is very convenient for my purposes. In the meantime I can indeed make of use of the other approaches that you mention.