MaineK00n commented 11 months ago

What did you implement:

Fixes #272

Type of change

[x] New feature (non-breaking change which adds functionality)

How Has This Been Tested?

$ go-cve-dictionary fetch nvd
INFO[11-17|15:56:23] Inserting NVD into DB (sqlite3). 
INFO[11-17|15:56:23] Deleting NVD tables... 
INFO[11-17|15:56:23] Fetching CVE information from NVD. 
INFO[11-17|15:56:23] Fetching... https://github.com/vulsio/vuls-data-raw-nvd-api-cve/archive/refs/heads/main.tar.gz 
INFO[11-17|15:56:34] Fetching CVE information from NVD(2002). 
INFO[11-17|15:56:35] Inserting fetched CVEs(2002)... 
2392 / 2392 [---------------------------------------------------------------------------------------] 100.00% 4053 p/s
INFO[11-17|15:56:35] Refreshed 2392 CVEs. 
INFO[11-17|15:56:35] Fetching CVE information from NVD(2003). 
INFO[11-17|15:56:36] Inserting fetched CVEs(2003)... 
1553 / 1553 [---------------------------------------------------------------------------------------] 100.00% 2912 p/s
INFO[11-17|15:56:36] Refreshed 1553 CVEs. 
INFO[11-17|15:56:36] Fetching CVE information from NVD(2004). 
INFO[11-17|15:56:37] Inserting fetched CVEs(2004)... 
2707 / 2707 [---------------------------------------------------------------------------------------] 100.00% 1402 p/s
INFO[11-17|15:56:39] Refreshed 2707 CVEs. 
INFO[11-17|15:56:39] Fetching CVE information from NVD(2005). 
INFO[11-17|15:56:39] Inserting fetched CVEs(2005)... 
4766 / 4766 [---------------------------------------------------------------------------------------] 100.00% 1759 p/s
INFO[11-17|15:56:42] Refreshed 4766 CVEs. 
INFO[11-17|15:56:42] Fetching CVE information from NVD(2006). 
INFO[11-17|15:56:43] Inserting fetched CVEs(2006)... 
7142 / 7142 [---------------------------------------------------------------------------------------] 100.00% 1901 p/s
INFO[11-17|15:56:47] Refreshed 7142 CVEs. 
INFO[11-17|15:56:47] Fetching CVE information from NVD(2007). 
INFO[11-17|15:56:47] Inserting fetched CVEs(2007)... 
6580 / 6580 [---------------------------------------------------------------------------------------] 100.00% 1889 p/s
INFO[11-17|15:56:51] Refreshed 6580 CVEs. 
INFO[11-17|15:56:51] Fetching CVE information from NVD(2008). 
INFO[11-17|15:56:51] Inserting fetched CVEs(2008)... 
7176 / 7176 [---------------------------------------------------------------------------------------] 100.00% 1569 p/s
INFO[11-17|15:56:56] Refreshed 7176 CVEs. 
INFO[11-17|15:56:56] Fetching CVE information from NVD(2009). 
INFO[11-17|15:56:57] Inserting fetched CVEs(2009)... 
5039 / 5039 [----------------------------------------------------------------------------------------] 100.00% 505 p/s
INFO[11-17|15:57:07] Refreshed 5039 CVEs. 
INFO[11-17|15:57:07] Fetching CVE information from NVD(2010). 
INFO[11-17|15:57:08] Inserting fetched CVEs(2010)... 
5216 / 5216 [----------------------------------------------------------------------------------------] 100.00% 828 p/s
INFO[11-17|15:57:15] Refreshed 5216 CVEs. 
INFO[11-17|15:57:15] Fetching CVE information from NVD(2011). 
INFO[11-17|15:57:15] Inserting fetched CVEs(2011)... 
4859 / 4859 [----------------------------------------------------------------------------------------] 100.00% 617 p/s
INFO[11-17|15:57:24] Refreshed 4859 CVEs. 
INFO[11-17|15:57:24] Fetching CVE information from NVD(2012). 
INFO[11-17|15:57:25] Inserting fetched CVEs(2012)... 
5890 / 5890 [----------------------------------------------------------------------------------------] 100.00% 612 p/s
INFO[11-17|15:57:35] Refreshed 5890 CVEs. 
INFO[11-17|15:57:35] Fetching CVE information from NVD(2013). 
INFO[11-17|15:57:36] Inserting fetched CVEs(2013)... 
6779 / 6779 [----------------------------------------------------------------------------------------] 100.00% 729 p/s
INFO[11-17|15:57:45] Refreshed 6779 CVEs. 
INFO[11-17|15:57:45] Fetching CVE information from NVD(2014). 
INFO[11-17|15:57:46] Inserting fetched CVEs(2014)... 
8976 / 8976 [---------------------------------------------------------------------------------------] 100.00% 1482 p/s
INFO[11-17|15:57:52] Refreshed 8976 CVEs. 
INFO[11-17|15:57:52] Fetching CVE information from NVD(2015). 
INFO[11-17|15:57:53] Inserting fetched CVEs(2015)... 
8738 / 8738 [---------------------------------------------------------------------------------------] 100.00% 1679 p/s
INFO[11-17|15:57:58] Refreshed 8738 CVEs. 
INFO[11-17|15:57:58] Fetching CVE information from NVD(2016). 
INFO[11-17|15:57:59] Inserting fetched CVEs(2016)... 
10544 / 10544 [-------------------------------------------------------------------------------------] 100.00% 1629 p/s
INFO[11-17|15:58:06] Refreshed 10544 CVEs. 
INFO[11-17|15:58:06] Fetching CVE information from NVD(2017). 
INFO[11-17|15:58:07] Inserting fetched CVEs(2017)... 
16977 / 16977 [-------------------------------------------------------------------------------------] 100.00% 1587 p/s
INFO[11-17|15:58:17] Refreshed 16977 CVEs. 
INFO[11-17|15:58:17] Fetching CVE information from NVD(2018). 
INFO[11-17|15:58:18] Inserting fetched CVEs(2018)... 
17341 / 17341 [-------------------------------------------------------------------------------------] 100.00% 2046 p/s
INFO[11-17|15:58:27] Refreshed 17341 CVEs. 
INFO[11-17|15:58:27] Fetching CVE information from NVD(2019). 
INFO[11-17|15:58:28] Inserting fetched CVEs(2019)... 
16968 / 16968 [-------------------------------------------------------------------------------------] 100.00% 1868 p/s
INFO[11-17|15:58:37] Refreshed 16968 CVEs. 
INFO[11-17|15:58:37] Fetching CVE information from NVD(2020). 
INFO[11-17|15:58:39] Inserting fetched CVEs(2020)... 
20406 / 20406 [-------------------------------------------------------------------------------------] 100.00% 1562 p/s
INFO[11-17|15:58:52] Refreshed 20406 CVEs. 
INFO[11-17|15:58:52] Fetching CVE information from NVD(2021). 
INFO[11-17|15:58:54] Inserting fetched CVEs(2021)... 
21973 / 21973 [-------------------------------------------------------------------------------------] 100.00% 1536 p/s
INFO[11-17|15:59:08] Refreshed 21973 CVEs. 
INFO[11-17|15:59:08] Fetching CVE information from NVD(2022). 
INFO[11-17|15:59:10] Inserting fetched CVEs(2022)... 
24397 / 24397 [-------------------------------------------------------------------------------------] 100.00% 1732 p/s
INFO[11-17|15:59:24] Refreshed 24397 CVEs. 
INFO[11-17|15:59:24] Fetching CVE information from NVD(2023). 
INFO[11-17|15:59:25] Inserting fetched CVEs(2023)... 
20194 / 20194 [-------------------------------------------------------------------------------------] 100.00% 1866 p/s
INFO[11-17|15:59:36] Refreshed 20194 CVEs. 
INFO[11-17|15:59:39] Finished fetching NVD.

Checklist:

You don't have to satisfy all of the following.

[ ] Write tests
[ ] Write documentation
[x] Check that there aren't other open pull requests for the same issue/feature
[x] Format your source code by make fmt
[x] Pass the test by make test
[x] Provide verification config / commands
[x] Enable "Allow edits from maintainers" for this PR
[x] Update the messages below

Is this ready for review?: YES

Reference

jbmaillet commented 11 months ago

Thanks a lot. Tested OK with some basic tests. :+1:

One question though: shouldn't there be some API key parameter somewhere on the CLI?

Beginning six months after the release of the API keys, users transmitting requests without a key will see a reduction in the number of requests they can make in a rolling 60 second window. Users transmitting requests that include their API key will see no change in service and may continue to make requests at the current rate.

From API-Key-Announcement Keys can be obtained here: Request an API Key

I have such a key ready to be used (a bit more difficult to actually test).

I'll do my best to do further tests this week, but in the course of these tests and deployment I'll need to switch from v0.5.6 from 2020 I currently use to up to date v0.9.0, and probably re-initiate the MySQL DB is use for custom queries.

jbmaillet commented 11 months ago

(Note that of course, the API key support could as well be added later on, as another enhancement. As for me, I already have a lot to test here... Again, thanks for your work last weekend. :100: )

MaineK00n commented 11 months ago

Since we are not fetching directly from the NVD API, an API KEY is not required. This time, the results of the NVD API are saved in this repository(https://github.com/vulsio/vuls-data-raw-nvd-api-cve), and go-cve-dictionary uses that repository.

jbmaillet commented 11 months ago

I tested this successfully in a MySQL setup and my custom client application. :+1:

I don't know Go, so this is a functional test only, not a code review, sorry for that.

The next step for me will be to update my production server to v0.9.0, to be ready for the next version with this new development. Thanks again!

fredericg78 commented 11 months ago

Hi, is it related to https://nvd.nist.gov/vuln/data-feeds ? "On December 15th, 2023, the NVD plans to retire all legacy data feeds while guiding any remaining data feed users to updated application-programming interfaces (APIs)." https://nvd.nist.gov/General/News/change-timeline: The NVD plans to retire the remaining legacy data feeds as well as all 1.0 APIs on December 15th. Which is the go-cve-dictionary version which starts to be compliant with this breaking change for fetching datas ?

Best regards

MaineK00n commented 11 months ago

@fredericg78

Although the version has not been clearly determined, go-cve-dictionary built after this PR was merged will use the results of the NVD API.

witchcraze commented 11 months ago

Let me report v0.10.0 roughly. I use server-mode with php. In my code, two update was required, but almost no problem.

Under "Cvss2", "Cvss3" is empty
["Cvss2"]["BaseScore"] → ["Cvss2"][0]["BaseScore"] (layer was changed ?)

MaineK00n commented 11 months ago

@witchcraze This was changed because there may be multiple CVSSv2 and CVSSv3.

witchcraze commented 11 months ago

Ah, OK. In this case, two CVSS 3.x was registered. (But we can not judje which is NVD's one) https://nvd.nist.gov/vuln/detail/CVE-2023-20254

$ curl -s http://127.0.0.1:1323/cves/CVE-2023-20254 | jq ".Nvds[].Cvss3"
[
  {
    "VectorString": "CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H",
    "AttackVector": "NETWORK",
    "AttackComplexity": "LOW",
    "PrivilegesRequired": "LOW",
    "UserInteraction": "NONE",
    "Scope": "UNCHANGED",
    "ConfidentialityImpact": "HIGH",
    "IntegrityImpact": "HIGH",
    "AvailabilityImpact": "HIGH",
    "BaseScore": 8.8,
    "BaseSeverity": "HIGH",
    "ExploitabilityScore": 2.8,
    "ImpactScore": 5.9
  },
  {
    "VectorString": "CVSS:3.1/AV:N/AC:L/PR:H/UI:N/S:U/C:H/I:H/A:H",
    "AttackVector": "NETWORK",
    "AttackComplexity": "LOW",
    "PrivilegesRequired": "HIGH",
    "UserInteraction": "NONE",
    "Scope": "UNCHANGED",
    "ConfidentialityImpact": "HIGH",
    "IntegrityImpact": "HIGH",
    "AvailabilityImpact": "HIGH",
    "BaseScore": 7.2,
    "BaseSeverity": "HIGH",
    "ExploitabilityScore": 1.2,
    "ImpactScore": 5.9
  }
]

MaineK00n commented 11 months ago

As for the original data, depending on the source, you can tell whether it is evaluated by NVD or another vendor...... https://github.com/vulsio/vuls-data-raw-nvd-api-cve/blob/main/2023/CVE-2023-20254.json

vulsio / go-cve-dictionary

feat(fetch/nvd): use NVD API feed #348

What did you implement:

Type of change

How Has This Been Tested?

Checklist:

Reference