tests: add tests for NVD 2.0 API

[terriko]

• related: #2330

@anthonyharrison added NVD API 2.0 support in #2330 and it currently doesn't have any 2.0-specific tests. It probably should have some.

Things to note:

1. We really want future NVD tests to not include connections to NVD, so these will probably be different than our existing tests
2. I suggest making use of Mock to make this happen, we may also make use of cached data.
3. Remember that eventually the 2.0 API will completely overtake the 1.0 one so we should plan to retain good code coverage after all the 1.0 tests are turned off.

[pdxjohnny]

2022-11-15 @pdxjohnny Engineering Logs

I just found myself implementing a http server for the NVD code to hit and saw this issue is open.

• Failure to run NVD tests (debugging currently)
  • https://gist.github.com/pdxjohnny/dcfaecadd743e773c8aed3e1d323e0bd
  • It worked!
  • Okay no, just non-deterministic. I guess that's the point of this issue :P
• References
  • 2022-11-15 @pdxjohnny Engineering Logs
  • https://github.com/intel/cve-bin-tool/pull/2330
  • NVD response.json() data dump

• Got the NVD tests parameterized to versions 1 and 2.

diff --git a/cve_bin_tool/nvd_api.py b/cve_bin_tool/nvd_api.py
index 6245c56..d151cd1 100644
--- a/cve_bin_tool/nvd_api.py
+++ b/cve_bin_tool/nvd_api.py
@@ -139,7 +139,7 @@ class NVD_API:

         if self.invalid_api:
             self.logger.warning(
-                f'Unable to access NVD using provided API key: {self.params["apiKey"]}'
+                f'Unable to access NVD using provided API key: {self.params.get("apiKey", "NO_API_KEY_GIVEN")}'
             )
         else:
             if time_of_last_update:
diff --git a/test/test_nvd_api.py b/test/test_nvd_api.py
index 29f14e9..109815c 100644
--- a/test/test_nvd_api.py
+++ b/test/test_nvd_api.py
@@ -8,6 +8,7 @@ from datetime import datetime, timedelta
 from test.utils import LONG_TESTS

 import pytest
+import aiohttp

 from cve_bin_tool.cvedb import CVEDB
 from cve_bin_tool.data_sources import nvd_source
@@ -42,14 +43,24 @@ class TestNVD_API:
         LONG_TESTS() != 1 or not os.getenv("nvd_api_key"),
         reason="NVD tests run only in long tests",
     )
-    async def test_total_results_count(self):
+    @pytest.mark.parametrize(
+        "api_version, feed",
+        [
+            ("1.0", None),
+            ("2.0", None),
+        ],
+    )
+    async def test_total_results_count(self, api_version, feed):
         """Total results should be greater than or equal to the current fetched cves"""
-        nvd_api = NVD_API(api_key=os.getenv("nvd_api_key") or "")
-        await nvd_api.get_nvd_params(
-            time_of_last_update=datetime.now() - timedelta(days=2)
-        )
-        await nvd_api.get()
-        assert len(nvd_api.all_cve_entries) >= nvd_api.total_results
+        async with aiohttp.ClientSession() as session:
+            nvd_api = NVD_API(api_key=os.getenv("nvd_api_key") or "",
+                              session=session)
+            nvd_api.logger.info("api_version: %s, feed: %s", api_version, feed)
+            await nvd_api.get_nvd_params(
+                time_of_last_update=datetime.now() - timedelta(days=2)
+            )
+            await nvd_api.get()
+            assert len(nvd_api.all_cve_entries) >= nvd_api.total_results

     @pytest.mark.asyncio
     @pytest.mark.skipif(

2022-11-16 @pdxjohnny Engineering Logs

• Got basic stats response saved from cache working
  • Got serving feed working with same page requested over fails test (as it should, paging broken currently, next is fix that).
• Side note: This asciinema was 12 MB uncut so I had to trim it up a bit

• single CVE import working

• Basic server
  • https://github.com/intel/dffml/blob/4101595a800e74f57cec5537ea2c65680135b71a/entities/alice/alice/threats/vulns/serve/nvdstyle.py#L1-L241
• WIP Test integration
  • https://gist.github.com/pdxjohnny/063a8111d48f66c1709c5102e138afaa

2022-11-18 @pdxjohnny Engineering Logs

• Still failing 2.0 tests, 1.0 works.
• https://gist.github.com/pdxjohnny/81977fa3effe189146bfbcfe04ed5fc8#file-validate-283492d554c095740c199f739dd4944bfab86a6db800993e16494209c1420061fe7c0e174570715ff7bd9132d26e9b47-json

$ nodemon -e py --exec 'clear; nvd_api_key=$NVD_API_KEY LONG_TESTS=1 timeout 10s python3.10 -um coverage run -m pytest -v --log-level=DEBUG --log-cli-level=DEBUG test/test_nvd_api.py::TestNVD_API::test_total_results_count -k 2.0; test 1'
...
___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ TestNVD_API.test_total_results_count[2.0-feed1-stats1] ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

self = <test.test_nvd_api.TestNVD_API object at 0x7f8dcaa7cf70>, api_version = '2.0', feed = <httptest.httptest.Server object at 0x7f8dcaa7c7c0>, stats = <httptest.httptest.Server object at 0x7f8dcaa7c700>
...
>               assert len(nvd_api.all_cve_entries) >= nvd_api.total_results
E               assert 0 >= 10
...
test/test_nvd_api.py:88: AssertionError
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Captured log setup ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DEBUG    asyncio:selector_events.py:54 Using selector: EpollSelector
DEBUG    asyncio:selector_events.py:54 Using selector: EpollSelector
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Fetching incremental metadata from NVD... ━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Downloading Feeds from NVD... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
127.0.0.1 - - [18/Nov/2022 08:38:09] "GET /?reporttype=countsbystatus HTTP/1.1" 200 -
127.0.0.1 - - [18/Nov/2022 08:38:09] "GET /2.0?startIndex=0&resultsPerPage=1 HTTP/1.1" 200 -
127.0.0.1 - - [18/Nov/2022 08:38:09] "GET /2.0?startIndex=0&resultsPerPage=2000&lastModStartDate=2022-11-16T16:36:09:895&lastModEndDate=2022-11-18T16:38:09:902 HTTP/1.1" 200 -
127.0.0.1 - - [18/Nov/2022 08:38:12] "GET /2.0?startIndex=0&resultsPerPage=2000&lastModStartDate=2022-11-16T16:36:09:895&lastModEndDate=2022-11-18T16:38:09:902 HTTP/1.1" 200 -
127.0.0.1 - - [18/Nov/2022 08:38:12] "GET /2.0?startIndex=2000&resultsPerPage=2000&lastModStartDate=2022-11-16T16:36:09:895&lastModEndDate=2022-11-18T16:38:09:902 HTTP/1.1" 200 -
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
INFO     cve_bin_tool.NVD_API:nvd_api.py:135 Fetching metadata from NVD...
DEBUG    alice.emulate.nvd.api:nvdstyle.py:158 ParseResult(scheme='', netloc='', path='/', params='', query='reporttype=countsbystatus', fragment='')
DEBUG    alice.emulate.nvd.api:nvdstyle.py:163 {'reporttype': ['countsbystatus']}
DEBUG    alice.emulate.nvd.api:nvdstyle.py:172 Serving stats...
INFO     cve_bin_tool.NVD_API:nvd_api.py:137 Got metadata from NVD: {'Total': 10, 'Rejected': 0, 'Received': 0, 'Modified': 0, 'Undergoing Analysis': 0, 'Awaiting Analysis': 0}
INFO     cve_bin_tool.NVD_API:nvd_api.py:140 self.total_results = Total: 10 - Rejected: 0
INFO     cve_bin_tool.NVD_API:nvd_api.py:144 Valiating NVD api...
DEBUG    alice.emulate.nvd.api:nvdstyle.py:158 ParseResult(scheme='', netloc='', path='/2.0', params='', query='startIndex=0&resultsPerPage=1', fragment='')
DEBUG    alice.emulate.nvd.api:nvdstyle.py:163 {'startIndex': ['0'], 'resultsPerPage': ['1']}
DEBUG    alice.emulate.nvd.api:nvdstyle.py:240 Serving validate NVD API: start_index: 0 results_per_page: 1...
DEBUG    alice.emulate.nvd.api:nvdstyle.py:274 Serving validate: results: {'format': 'NVD_CVE', 'resultsPerPage': 1, 'startIndex': 0, 'timestamp': '2022-11-18T08:38Z', 'totalResults': 10, 'version': '2.0', 'vulnerabilities': [{'cve': {'configurations': [{'nodes': [{'cpeMatch': [{'criteria': 'cpe:2.3:a:eric_allman:sendmail:5.58:*:*:*:*:*:*:*', 'matchCriteriaId': '1D07F493-9C8D-44A4-8652-F28B46CBA27C', 'vulnerable': True}], 'negate': False, 'operator': 'OR'}]}], 'descriptions': [{'lang': 'en', 'value': 'The debug command in Sendmail is enabled, allowing attackers to execute commands as root.'}, {'lang': 'es', 'value': 'El comando de depuración de Sendmail está activado, permitiendo a atacantes ejecutar comandos como root.'}], 'id': 'CVE-1999-0095', 'lastModified': '2019-06-11T20:29:00.263', 'metrics': {'cvssMetricV2': [{'acInsufInfo': False, 'cvssData': {'accessComplexity': 'LOW', 'accessVector': 'NETWORK', 'authentication': 'NONE', 'availabilityImpact': 'COMPLETE', 'baseScore': 10.0, 'baseSeverity': 'HIGH', 'confidentialityImpact': 'COMPLETE', 'integrityImpact': 'COMPLETE', 'vectorString': 'AV:N/AC:L/Au:N/C:C/I:C/A:C', 'version': '2.0'}, 'exploitabilityScore': 10.0, 'impactScore': 10.0, 'obtainAllPrivilege': True, 'obtainOtherPrivilege': False, 'obtainUserPrivilege': False, 'source': 'nvd@nist.gov', 'type': 'Primary', 'userInteractionRequired': False}]}, 'published': '1988-10-01T04:00:00.000', 'references': [{'source': 'cve@mitre.org', 'url': 'http://seclists.org/fulldisclosure/2019/Jun/16'}, {'source': 'cve@mitre.org', 'url': 'http://www.openwall.com/lists/oss-security/2019/06/05/4'}, {'source': 'cve@mitre.org', 'url': 'http://www.openwall.com/lists/oss-security/2019/06/06/1'}, {'source': 'cve@mitre.org', 'url': 'http://www.securityfocus.com/bid/1'}], 'sourceIdentifier': 'cve@mitre.org', 'vulnStatus': 'Modified', 'weaknesses': [{'description': [{'lang': 'en', 'value': 'NVD-CWE-Other'}], 'source': 'nvd@nist.gov', 'type': 'Primary'}]}}]}
INFO     cve_bin_tool.NVD_API:nvd_api.py:146 Valiated NVD api
INFO     cve_bin_tool.NVD_API:nvd_api.py:175 Fetching updated CVE entries after 2022-11-16T16:36:09:895
DEBUG    alice.emulate.nvd.api:nvdstyle.py:158 ParseResult(scheme='', netloc='', path='/2.0', params='', query='startIndex=0&resultsPerPage=2000&lastModStartDate=2022-11-16T16:36:09:895&lastModEndDate=2022-11-18T16:38:09:902', fragment='')
DEBUG    alice.emulate.nvd.api:nvdstyle.py:163 {'startIndex': ['0'], 'resultsPerPage': ['2000'], 'lastModStartDate': ['2022-11-16T16:36:09:895'], 'lastModEndDate': ['2022-11-18T16:38:09:902']}
DEBUG    alice.emulate.nvd.api:nvdstyle.py:284 Serving feed: start_index: 0 results_per_page: 2000...
DEBUG    alice.emulate.nvd.api:nvdstyle.py:336 Serving feed with 10 results
INFO     cve_bin_tool.NVD_API:nvd_api.py:189 Adding 10 CVE entries
DEBUG    alice.emulate.nvd.api:nvdstyle.py:158 ParseResult(scheme='', netloc='', path='/2.0', params='', query='startIndex=0&resultsPerPage=2000&lastModStartDate=2022-11-16T16:36:09:895&lastModEndDate=2022-11-18T16:38:09:902', fragment='')
DEBUG    alice.emulate.nvd.api:nvdstyle.py:158 ParseResult(scheme='', netloc='', path='/2.0', params='', query='startIndex=2000&resultsPerPage=2000&lastModStartDate=2022-11-16T16:36:09:895&lastModEndDate=2022-11-18T16:38:09:902', fragment='')
DEBUG    alice.emulate.nvd.api:nvdstyle.py:163 {'startIndex': ['0'], 'resultsPerPage': ['2000'], 'lastModStartDate': ['2022-11-16T16:36:09:895'], 'lastModEndDate': ['2022-11-18T16:38:09:902']}
DEBUG    alice.emulate.nvd.api:nvdstyle.py:163 {'startIndex': ['2000'], 'resultsPerPage': ['2000'], 'lastModStartDate': ['2022-11-16T16:36:09:895'], 'lastModEndDate': ['2022-11-18T16:38:09:902']}
DEBUG    alice.emulate.nvd.api:nvdstyle.py:284 Serving feed: start_index: 0 results_per_page: 2000...
DEBUG    alice.emulate.nvd.api:nvdstyle.py:284 Serving feed: start_index: 2000 results_per_page: 2000...
DEBUG    alice.emulate.nvd.api:nvdstyle.py:336 Serving feed with 10 results
DEBUG    alice.emulate.nvd.api:nvdstyle.py:336 Serving feed with 0 results
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Captured log teardown --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DEBUG    asyncio:selector_events.py:54 Using selector: EpollSelector
=================================================================================================================================================================================================================================================================================================================== short test summary info ==============================================================================================================================================================  - 
=====================================================================================================================================================
FAILED test/test_nvd_api.py::TestNVD_API::test_total_results_count[2.0-feed1-stats1] - assert 0 >= 10
=============================================================================================================================================================================================================================================================================================================== 1 failed, 1 deselected in 6.51s ===============================================================================================================================================================================================================================================================================================================
[nodemon] clean exit - waiting for changes before restart

• Ah ha! Enabled debug logging because noticed we weren't seeing the "Send Request" log client side.

diff --git a/cve_bin_tool/log.py b/cve_bin_tool/log.py
index 85b7009..749b867 100644
--- a/cve_bin_tool/log.py
+++ b/cve_bin_tool/log.py
@@ -30,4 +30,4 @@ logging.basicConfig(
 root_logger = logging.getLogger()

 LOGGER = logging.getLogger(__package__)
-LOGGER.setLevel(logging.INFO)
+LOGGER.setLevel(logging.DEBUG)
diff --git a/cve_bin_tool/nvd_api.py b/cve_bin_tool/nvd_api.py
index 28bc102..0f82748 100644
--- a/cve_bin_tool/nvd_api.py
+++ b/cve_bin_tool/nvd_api.py
@@ -130,14 +130,20 @@ class NVD_API:

         if not self.session:
             connector = aiohttp.TCPConnector(limit_per_host=19)
-            self.session = RateLimiter(
-                aiohttp.ClientSession(connector=connector, trust_env=True)
-            )
+            self.session = aiohttp.ClientSession(connector=connector, trust_env=True)

         self.logger.info("Fetching metadata from NVD...")
         cve_count = await self.nvd_count_metadata(self.session, self.stats)
+        self.logger.info("Got metadata from NVD: %r", cve_count)
+
+        self.total_results = cve_count["Total"] - cve_count["Rejected"]
+        self.logger.info(
+            f'self.total_results = Total: {cve_count["Total"]} - Rejected: {cve_count["Rejected"]}'
+        )

+        self.logger.info("Valiating NVD api...")
         await self.validate_nvd_api()
+        self.logger.info("Valiated NVD api")

         if self.invalid_api:
             self.logger.warning(
@@ -180,8 +186,6 @@ class NVD_API:
                     progress.update(task)
                 progress.update(task, advance=1)

-            else:
-                self.total_results = cve_count["Total"] - cve_count["Rejected"]
             self.logger.info(f"Adding {self.total_results} CVE entries")

     async def validate_nvd_api(self):
@@ -227,7 +231,6 @@ class NVD_API:
                     self.logger.debug(f"Response received {response.status}")
                     if response.status == 200:
                         fetched_data = await response.json()
-
                         if start_index == 0:
                             # Update total results in case there is discrepancy between NVD dashboard and API
                             reject_count = (
@@ -238,6 +241,9 @@ class NVD_API:
                             self.total_results = (
                                 fetched_data["totalResults"] - reject_count
                             )
+                            self.logger.info(
+                                f'self.total_results = Total: {fetched_data["totalResults"]} - Rejected: {reject_count}'
+                            )
                         if self.api_version == "1.0":
                             self.all_cve_entries.extend(
                                 fetched_data["result"]["CVE_Items"]
diff --git a/test/test_nvd_api.py b/test/test_nvd_api.py
index 91cf1fb..e7e2a96 100644
--- a/test/test_nvd_api.py
+++ b/test/test_nvd_api.py
@@ -2,16 +2,26 @@
 # SPDX-License-Identifier: GPL-3.0-or-later

 import os
+import types
 import shutil
 import tempfile
+import contextlib
 from datetime import datetime, timedelta
 from test.utils import LONG_TESTS

 import pytest
+import aiohttp
+import httptest
+
+import alice.threats.vulns.serve.nvdstyle

 from cve_bin_tool.cvedb import CVEDB
 from cve_bin_tool.data_sources import nvd_source
-from cve_bin_tool.nvd_api import NVD_API
+from cve_bin_tool.nvd_api import (
+    NVD_API,
+    FEED as NVD_API_FEED,
+    NVD_CVE_STATUS,
+)

 class TestNVD_API:
@@ -42,14 +52,40 @@ class TestNVD_API:
         LONG_TESTS() != 1 or not os.getenv("nvd_api_key"),
         reason="NVD tests run only in long tests",
     )
-    async def test_total_results_count(self):
+    @pytest.mark.parametrize(
+        "api_version, feed, stats",
+        [
+            (
+                "1.0",
+                httptest.Server(alice.threats.vulns.serve.nvdstyle.NVDStyleHTTPHandler),
+                httptest.Server(alice.threats.vulns.serve.nvdstyle.NVDStyleHTTPHandler),
+            ),
+            (
+                "2.0",
+                httptest.Server(alice.threats.vulns.serve.nvdstyle.NVDStyleHTTPHandler),
+                httptest.Server(alice.threats.vulns.serve.nvdstyle.NVDStyleHTTPHandler),
+            ),
+        ],
+    )
+    async def test_total_results_count(self, api_version, feed, stats):
         """Total results should be greater than or equal to the current fetched cves"""
-        nvd_api = NVD_API(api_key=os.getenv("nvd_api_key") or "")
-        await nvd_api.get_nvd_params(
-            time_of_last_update=datetime.now() - timedelta(days=2)
-        )
-        await nvd_api.get()
-        assert len(nvd_api.all_cve_entries) >= nvd_api.total_results
+        # TODO alice.nvd.TestHTTPServer will become either
+        # alice.nvd.TestNVDVersion_1_0 or alice.nvd.TestNVDVersion_2_0
+        # lambda *args: alice.nvd.TestHTTPServer(*args, directory=pathlib.Path(__file__).parent)
+        with feed as feed_http_server, stats as stats_http_server:
+            async with aiohttp.ClientSession() as session:
+                nvd_api = NVD_API(
+                    feed=feed_http_server.url(),
+                    stats=stats_http_server.url(),
+                    api_key=os.getenv("nvd_api_key") or "",
+                    session=session,
+                    api_version=api_version,
+                )
+                await nvd_api.get_nvd_params(
+                    time_of_last_update=datetime.now() - timedelta(days=2)
+                )
+                await nvd_api.get()
+                assert len(nvd_api.all_cve_entries) >= nvd_api.total_results

     @pytest.mark.asyncio- 

     @pytest.mark.skipif(

• Enabling debug logging resulted in the following statement being logged.
  • This failure should probably be an ERROR level rather than DEBUG log.

DEBUG    cve_bin_tool.NVD_API:nvd_api.py:274 Failed to connect to NVD list indices must be integers or slices, not str

• Added traceback
• Is NVD2 code needing to index? fetched_data["vulnerabilities"][index]["cve"]?

ERROR    cve_bin_tool.NVD_API:nvd_api.py:276 Pausing requests for 3 seconds
DEBUG    cve_bin_tool.NVD_API:nvd_api.py:277 TypeError('list indices must be integers or slices, not str')
Traceback (most recent call last):
  File "/home/pdxjohnny/Documents/python/cve-bin-tool/cve_bin_tool/nvd_api.py", line 254, in load_nvd_request
    fetched_data["vulnerabilities"]["cve"]
TypeError: list indices must be integers or slices, not str

• Found and fixed two issues
  • intel/cve-bin-tool@afc4a9254683d2a7027bc6574e99d1b0d406d5bc
  • fix(nvd_api): Align v2 rejection handling with description schema updates
  • intel/cve-bin-tool@46cd825b126dd167158cae4f5e4ac7a32de2e08d
  • fix(nvd_api): extend all cve entries from v2 query top level vulnerabilities key

• Pushed 9f0a41ad55bdc7f295c435ebd51db77e3343b915
  • alice: threats: vulns: serve: nvdstyle: Fix serving of v2 style CVEs

2022-11-21 @pdxjohnny Engineering Logs

• Pushed NVDStyleHTTPHandler to tests/nvd_style.py
• Finding and updating some of the v2 related parsing code...
• Looks like Anthony pushed related changes recently, rebasing in main...

2022-11-22 @pdxjohnny Engineering Logs

• Rebased in main
• Still seeing some issues

FAILED test/test_nvd_api.py::TestNVD_API::test_get_nvd_params[1.0] - assert (10 == 0)
FAILED test/test_nvd_api.py::TestNVD_API::test_get_nvd_params[2.0] - assert (10 == 0)
FAILED test/test_nvd_api.py::TestNVD_API::test_nvd_incremental_update[1.0] - assert 1 == 10
FAILED test/test_nvd_api.py::TestNVD_API::test_nvd_incremental_update[2.0] - assert 2 == 10

[terriko]

No one's working on it yet!

And yeah, my guess as to why the windows instances fail so much more often than the linux ones is either that they have fewer IPs or more people running similar scan jobs, but it seems like the windows instances just have some networking issues too so it might be something about the cloud setup or their underlying network.

I really wish github and NVD would partner up to fix this problem ASAP. ;)

[terriko]

I think we're as done with the NVD tests as we're going to get for now (especially given the limitations of the NVD servers), so I'm going to close this.