missing CPE version 2.3 support

[cmuller]

cvechecker uses feeds, now in json format only, from NIST.

But it seems that while the "old" (e.g., two years old) feeds still have a mix of CPE 2.2 and CPE 2.3 entries, the newest feeds only have CPE 2.3 entries and I can't find any possibility to have those analyzed, unless we find a sort of conversion possibility (?).

For example if I type:

$ echo 'cpe:/a:sqlite:sqlite:3.6.0:::' > small.txt
$ cvechecker -w small.txt 
Adding CPE entries
 - Added watch for cpe:/a:sqlite:sqlite:3.6.0:::
$ cvechecker -r => nothing

It is not finding https://nvd.nist.gov/vuln/detail/CVE-2019-8457 which involves this version of sqlite.

It looks like in nvdcve-1.1-2019.json for example only the 2.3 version of CPEs are listed and not the 2.2 versions anymore..

I see for this CVE:

        "cpe_match": [
          {
            "vulnerable": true,
            "cpe23Uri": "cpe:2.3:a:sqlite:sqlite:*:*:*:*:*:*:*:*",
            "versionStartIncluding": "3.6.0",
            "versionEndIncluding": "3.27.2"
          }
        ]

Now if I try to use a CPE in the 2.3 version format with cvechecker, I get the following error:

$ echo 'cpe:2.3:a:sqlite:sqlite:3.6.0:*:*:*:*:*:*:*' > small.txt
$ cvechecker -w small.txt 
Adding CPE entries
 ! An error occurred while interpreting CPE on line 0

So.. I guess we are stuck until the tool can add CPEs to watch in the 2.3 version format.

[fcabaud]

I modified 2 files cvecheck.c and cvecheck_common.h for CPE 2.3 needs. As result, C structure struct cpe_data is modified (a boolean format_is_2_3 is added). This is compatible CPE 2.2. checkchecker -i is working but not pullcves:

sudo scripts/pullcves pull
Downloading nvdcve-1.1-Modified.json.gz... ok (downloaded)
Converting nvdcve-1.1-Modified.json to CSV... ok
Gathering differences with last pull... ok
Loading in nvdcve-1.1-modified.csv differences in cvechecker.
Loading CVE data from /usr/local/var/cvechecker/cache/nvdcve-1.1-modified.delta into database
100 records processed (0 already in db)...
200 records processed (0 already in db)...
300 records processed (0 already in db)...
400 records processed (0 already in db)...
500 records processed (0 already in db)...
600 records processed (0 already in db)...
700 records processed (0 already in db)...
800 records processed (0 already in db)...
900 records processed (0 already in db)...
1000 records processed (0 already in db)...
Failed to prepare statement "select cpeid from tb_cpe_p_4 where cpepart = "p" and cpevendor = "leap" and cpeproduct = "42.3" and cpeversion = "" and cpeupdate = "" and cpeedition = "*" and cpelanguage = "";".
*** Could not import nvdcve-1.1-modified.delta

It seems, there is an issue in ./src/sqlite3/sqlite3_impl.h in

#define PREPARE_SQLITE(rc, db, stmt, sqlstmt) \
  rc = sqlite3_prepare_v2(db, stmt, -1, &sqlstmt, NULL); \
  if (rc != SQLITE_OK) { \
    fprintf(stderr, "Failed to prepare statement \"%s\".\n", stmt); \
    exit(EXIT_FAILURE); \
  };

I dont know how to interface this modification with the DataBase. Can you help us ?

CVE_patch.tar.gz

[sjvermeu]

I have committed a larger update on cvechecker to support CPE 2.3 better. I haven't tested everything yet, but a clean install, followed by a "pullcves pull" and then loading a recent scanlist (cvechecker -l) does seem to work properly. The CVEs are loaded, the installed software is detected, and it finds software that is vulnerable (and my system hasn't updated yet).

Can you test it out and see how things work for you? You will need to do a clean install (cvechecker -i to reinitialize the databases at least) as the database tables have been changed.

[cmuller]

Hello. I tried this release and the compilation, installation, cvechecker -i and pullcves pull went flawlessly but then when I make a small watchlist file with:

cpe:2.3:a:gnu:zlib:1.2.2:*:*:*:*:*:*:*
cpe:2.3:a:json-c_project:json-c:1.0.1:*:*:*:*:*:*:*
cpe:2.3:a:openssl:openssl:1.0.2k:*:*:*:*:*:*:*

i get errors (I tried only with CPE2.3 entries as I didn't expect it to work with CPE2.2):

$ cvechecker --watchlist=w.txt --deltaonly
Adding CPE entries
 ! An error occurred while interpreting CPE on line 0
 ! An error occurred while interpreting CPE on line 1
 ! An error occurred while interpreting CPE on line 2

Am I missing something? I do not use the file to scan feature but more this watchlist specifying the list of CPEs. I also tried without the *:*:*:*:*:*:* parts with no success.

[sjvermeu]

I haven't checked the watchlist yet. I am currently working on supporting the version-less CPEs which you also mentioned in the initial report (where only the first non-vulnerable version is listed).

On Wed, Jul 1, 2020, 5:12 PM Christophe Muller notifications@github.com wrote:

Hello. I tried this release and the compilation, installation, cvechecker -i and pullcves pull went flawlessly but then when I make a small watchlist file with:

cpe:2.3:a:gnu:zlib:1.2.2::::::: cpe:2.3:a:json-c_project:json-c:1.0.1::::::: cpe:2.3:a:openssl:openssl:1.0.2k:::::::*

i get errors (I tried only with CPE2.3 entries as I didn't expect it to work with CPE2.2):

$ cvechecker --watchlist=w.txt --deltaonly Adding CPE entries ! An error occurred while interpreting CPE on line 0 ! An error occurred while interpreting CPE on line 1 ! An error occurred while interpreting CPE on line 2

Am I missing something? I do not use the file to scan feature but more this watchlist specifying the list of CPEs. I also tried without the ::::::* parts with no success.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/sjvermeu/cvechecker/issues/47#issuecomment-652478253, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACTRK52FYBB3PDGJMPNP4TRZNG7DANCNFSM4LQC3Y7A .

[cmuller]

Ok. Thank you for the answer. I will wait for the the one that can handle our CPEs with versions.

[sjvermeu]

The current code should handle the case you had now as well. I will see how I can best add test cases to cvechecker so that maintenance is easier towards the future (and that all the different use cases, CPE stuff and what not has better coverage), but could you see if things work for you?

$ cvechecker -w watchlist
Adding CPE entries
 - Added watch for cpe:2.3:a:apple:windows_migration_assistant:2.1.90:*:*:*:*:*:*:*

$ cvechecker -rH 
File "__provided__/cpe:2.3:a:apple:windows_migration_assistant:2.1.90:*:*:*:*:*:*:*" (CPE = cpe:2.3:a:apple:windows_migration_assistant:2.1.90:*:*:*:*:*:*:*) on host clevo (key clevo)
  Potential vulnerability found (CVE-2020-9858)
  CVSS Score is 7.8
  Match with potential higher version

You will need to use the -H (--reporthigher) for most recent CVEs though. cvechecker only records the vulnerable versions (as NVD previously iterated over all vulnerable versions), so when we receive a CVE with only a 'fixed from version xxx onward' we can't record all vulnerable software. With --reporthigher cvechecker will report on software whose CVE mentioned a higher version as being vulnerable (and in the code I then change the version xxx to xxx_alpha so that cvechecker assumes that version xxx_alpha is the last vulnerable version).

[cmuller]

Thanks. I will compile the latest version and test it on our data on Monday. I will keep you posted on our results.

[cmuller]

Hello, sorry for the delay.

I tried the latest version and it compiles and run ok now when using --watchlist=... on a file with CPE 2.3 entries (I translated the 2.2 ones). But...

I have a problem with the results:

• without the --reporthigher option I do not find CVEs that I had before, and
• with the --reporthigher option I have lots of CVEs that.. actually should not apply to the specified component/version

Let me take an example: with a watchlist containing an old version of Python: cpe:2.3:a:python:python:2.5.6:*:*:*:*:*:*:* watch-python.txt

For this version, I had 11 CVEs before.

• If I do not use the option, I get only 4 CVEs:

python-CVEs-without-reporthigher.txt

• But if I add the --reporthigher option, then I get 451 CVEs (!):

python-CVEs-with-reporthigher.txt

and some of them are actually not true, e.g., if I examine the 7th one (CVE-2012-2135) the affected software, according to https://nvd.nist.gov/vuln/detail/CVE-2012-2135 are Python versions 3.1, 3.2, 3.3 and not 2.5 at all.

As a consequence I cannot use the option to discover new 2020 CVEs as I run the risk of getting lots of false positive ones.

[sjvermeu]

Hi Christophe

Thanks for the update. The direct hits (those without --reporthigher) have the Python 2.5.6 explicitly mentioned as a vulnerable version. The other ones don't, but have higher versions listed. As you mentioned, this can cause for false positives if you ask for higher matches.

The problem we face is that cvechecker is built around the principle that the vulnerable versions are explicitly mentioned. I added the --reporthigher in the past because several CVEs did not list all vulnerable versions anymore, especially when the lower versions were no longer supported. For instance, support for Python v2.5 ended in May 2011, so it is likely that vulnerabilities found afterwards no longer list this version as vulnerable explicitly.

Recent CVE reports don't explicitly list versions anymore, but use constructions like 'all versions lower than X.Y.Z are vulnerable' (in the JSONs, search for versionEndExcluding). That means that the principle mentioned earlier no longer holds, but that reporting with --reporthigher is still useful, even though it can contain false positives.

Now, in your specific case (Python v2.5.6) it might be that those are not false positives, as the EOL of Python v2.5 has since long been reached. However, there will be situations that will provide much more obvious false positives. Taking Python again, it has multiple major versions active (at the time of writing, this is 3.5, 3.6, 3.7 and 3.8). It is very well possible that a vulnerability is found in a recent functionality, only affecting the 3.8 branch, which will cause false positives for the other major versions.

An example is CVE-2018-20852 which has explicit ranges for each of the major versions of Python. For cvechecker, this will currently result in possible false positives indeed. There is another issue/report (#16) to add flags about false positives to the database so they are no longer reported - when I initially wrote it, these flags were added later on in a general reporting system outside cvechecker, but it makes sense to include the capability in cvechecker itself.

While that would allow for managing the false positives, it does not provide a correct solution for this situation. The best way forward is most likely to drop the translation of CVEs and CPEs into the database, and parse the JSONs fully without dropping any information. That, however, is a major redesign, which I'm hesitant to tackle in a quick manner (I have some ideas, but that will probably just mean a full rewrite of everything, splitting cvechecker in a CPE detection module and a CVE JSON mapper - and perhaps such mapper already exist out there?)

[cmuller]

Hello,

I think I understand. After my holidays (starting tonight :) ) I am going to try and test such process maybe with some utilities to scan the actual contents of CVEs (there is a repository with all CVEs in json format I think). I will tell you what it gives.. and might be ideas for future developments. Anyway, as for CPE 2.3 support it is actually working well! :) so I guess you can close this issue and maybe reopen one for the redesign ideas, as you wish. Thanks for all your work with this tool anyway. And also have nice holidays if you have some.

[sjvermeu]

Great, thanks for the testing. I've now also released cvechecker v4.0.