Sporadic failures from `cyhy-nvdsync`

[jsf9k]

🐛 Summary

The cyhy-nvdsync process occasionally fails due to a ValueError exception thrown from json.load(). When run later the command succeeds.

To reproduce

I am unable to reproduce this bug, but you can see instances of it in /var/log/syslog on the database1 instance:

$ sudo zgrep ValueError /var/log/syslog|less
Feb 23 08:16:55 database1 cyhy-nvdsync:     raise ValueError("No JSON object could be decoded")
Feb 23 08:16:55 database1 cyhy-nvdsync: ValueError: No JSON object could be decoded

Expected behavior

If the file published by NIST is corrupted, then replaced with an uncorrupted version, or if there are multiple download nodes and one of them contains a bad copy, then that would explain the behavior. If that is not the case then the observed behavior is unexplained.

Any helpful log output or screenshots

Here is a full stacktrace from a failure in /var/log/syslog:

ERROR:root:Unexpected exception
Feb 23 08:16:55 database1 cyhy-nvdsync: Traceback (most recent call last):
Feb 23 08:16:55 database1 cyhy-nvdsync:   File "/usr/local/bin/cyhy-nvdsync", line 97, in main
Feb 23 08:16:55 database1 cyhy-nvdsync:     process_url(db, url)
Feb 23 08:16:55 database1 cyhy-nvdsync:   File "/usr/local/bin/cyhy-nvdsync", line 77, in process_url
Feb 23 08:16:55 database1 cyhy-nvdsync:     parse_json(db, f)
Feb 23 08:16:55 database1 cyhy-nvdsync:   File "/usr/local/bin/cyhy-nvdsync", line 39, in parse_json
Feb 23 08:16:55 database1 cyhy-nvdsync:     data = json.load(json_stream)
Feb 23 08:16:55 database1 cyhy-nvdsync:   File "/usr/lib/python2.7/json/__init__.py", line 291, in load
Feb 23 08:16:55 database1 cyhy-nvdsync:     **kw)
Feb 23 08:16:55 database1 cyhy-nvdsync:   File "/usr/lib/python2.7/json/__init__.py", line 339, in loadsFeb 23 08:16:55 database1 cyhy-nvdsync:     return _default_decoder.decode(s)
Feb 23 08:16:55 database1 cyhy-nvdsync:   File "/usr/lib/python2.7/json/decoder.py", line 364, in decodeFeb 23 08:16:55 database1 cyhy-nvdsync:     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Feb 23 08:16:55 database1 cyhy-nvdsync:   File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
Feb 23 08:16:55 database1 cyhy-nvdsync:     raise ValueError("No JSON object could be decoded")
Feb 23 08:16:55 database1 cyhy-nvdsync: ValueError: No JSON object could be decoded
Feb 23 08:16:55 database1 cyhy-nvdsync:  . . . . . . . . . . . . . . . . . . . . . . x . . . . . . . . . . . . x x x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x . x . . x 
Feb 23 08:16:55 database1 cyhy-nvdsync: 
Feb 23 08:16:55 database1 cyhy-nvdsync: 
Feb 23 08:16:55 database1 cyhy-nvdsync: ---------- https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2024.json.gz ----------

Note that there is likely some interleaving due to both standard output and standard error being redirected to the same file.

[dav3r]

It might be helpful if we modified this script to output some info about the JSON file if there is an error like this (or maybe always), I'm thinking things like file length or contents.

[jsf9k]

I ran the sudo -u cyhy cyhy-nvdsync --use-network command thirteen times this morning without encountering a single error.

[jsf9k]

From an email @dav3r sent on Feb. 28, 2024:

It looks like whatever caused this issue has been resolved since we haven't gotten this alarm since Friday.

@dav3r - Feel free to go ahead and close this issue if you think that is appropriate.

[dav3r]

It appears that whatever was causing this failure has been resolved at the source, without any changes needed by us, so I am closing this issue.