DependencyTrack / dependency-track

Dependency-Track is an intelligent Component Analysis platform that allows organizations to identify and reduce risk in the software supply chain.
https://dependencytrack.org/
Apache License 2.0
2.69k stars 578 forks source link

Remove NpmAuditAnalysisTask - Create GitHubAdvisoryAnalysisTask #1225

Closed StyleT closed 2 years ago

StyleT commented 3 years ago

See https://github.blog/2021-10-07-github-advisory-database-now-powers-npm-audit/

Current Behavior:

12:30:52.442 INFO [VulnerabilityAnalysisTask] (244 out of 245) Analyzing 2018 components in project: cf2aa72a-d64e-449e-9f53-10ff24ddd808
12:30:52.482 INFO [NpmAuditAnalysisTask] Starting Node Audit analysis task for 2018 components
12:30:57.730 INFO [NpmAuditAnalysisTask] Processing NPM advisories
12:30:57.735 ERROR [VulnerabilityAnalysisTask] An unexpected error occurred performing a vulnerability analysis task
java.lang.NullPointerException: null
    at org.dependencytrack.persistence.VulnerabilityQueryManager.contains(VulnerabilityQueryManager.java:258)
    at org.dependencytrack.persistence.QueryManager.contains(QueryManager.java:635)
    at org.dependencytrack.util.NotificationUtil.analyzeNotificationCriteria(NotificationUtil.java:56)
    at org.dependencytrack.tasks.scanners.NpmAuditAnalysisTask.processResults(NpmAuditAnalysisTask.java:205)
    at org.dependencytrack.tasks.scanners.NpmAuditAnalysisTask.analyze(NpmAuditAnalysisTask.java:163)
    at org.dependencytrack.tasks.scanners.NpmAuditAnalysisTask.inform(NpmAuditAnalysisTask.java:74)
    at org.dependencytrack.tasks.VulnerabilityAnalysisTask.performAnalysis(VulnerabilityAnalysisTask.java:166)
    at org.dependencytrack.tasks.VulnerabilityAnalysisTask.analyzeComponents(VulnerabilityAnalysisTask.java:126)
    at org.dependencytrack.tasks.VulnerabilityAnalysisTask.inform(VulnerabilityAnalysisTask.java:88)
    at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:99)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)

Steps to Reproduce:

Upload NPM project with vulnerabilities to DTrack

Expected Behavior:

Should be working as normal

Environment:

Additional Details:

Commit wit temp fix: https://github.com/StyleT/dependency-track/commit/205ab0a66d59fa0f8cb799de8923b7489468aaee

But generally - we need to remove NpmAdvisoryMirrorTask (now it's useless), and create GithubAdvisoryMirrorTask which would work not only for NPM projects (see API here).

stevespringett commented 3 years ago

Looks like GitHub isn't correlating vulnerability identifiers properly. This is a defect in GitHub and should be resolved by them.

As an example, I will take npm advisory 1673 which affects lodash.

Pagination of npm advisories will return the following snippet (currently accessible at https://registry.npmjs.org/-/npm/v1/security/advisories?perPage=20&page=4)

{
    "access": "public",
    "created": "2021-05-06T16:14:39.514Z",
    "cves": [
        "CVE-2021-23337"
    ],
    "cwe": "CWE-77",
    "deleted": null,
    "found_by": {
        "email": "",
        "link": "",
        "name": "Anonymous"
    },
    "id": 1673,
    "metadata": {
        "affected_components": "",
        "exploitability": 7,
        "module_type": ""
    },
    "module_name": "lodash",
    "overview": "`lodash` versions prior to 4.17.21 are vulnerable to Command Injection via the template function.",
    "patched_versions": ">=4.17.21",
    "recommendation": "Upgrade to version 4.17.21 or later",
    "references": "- [CVE](https://nvd.nist.gov/vuln/detail/CVE-2021-23337)\n- [GitHub Advisory](https://github.com/advisories/GHSA-35jh-r3h4-6jhm)\n- [Snyk Advisory](https://snyk.io/vuln/SNYK-JS-LODASH-1040724)",
    "reported_by": {
        "email": "",
        "link": "",
        "name": "Anonymous"
    },
    "severity": "high",
    "title": "Command Injection",
    "updated": "2021-05-06T16:24:12.299Z",
    "vulnerable_versions": "<4.17.21"
}

However, when performing an npm audit against lodash 4.17.15, the following snippet is returned:

"1002847": {
  "access": "public",
  "created": "2021-10-07T07:31:50.564Z",
  "cves": [
    "CVE-2021-23337"
  ],
  "cwe": "CWE-77",
  "deleted": null,
  "findings": [
    {
      "paths": [
        "lodash"
      ],
      "version": "4.17.15"
    }
  ],
  "found_by": null,
  "github_advisory_id": "GHSA-35jh-r3h4-6jhm",
  "id": 1002847,
  "metadata": null,
  "module_name": "lodash",
  "npm_advisory_id": null,
  "overview": "`lodash` versions prior to 4.17.21 are vulnerable to Command Injection via the template function.",
  "patched_versions": ">=4.17.21",
  "recommendation": "Upgrade to version 4.17.21 or later",
  "references": "- https://nvd.nist.gov/vuln/detail/CVE-2021-23337\n- https://github.com/advisories/GHSA-35jh-r3h4-6jhm",
  "reported_by": null,
  "severity": "high",
  "title": "Command Injection in lodash",
  "updated": "2021-03-31T23:59:26.000Z",
  "url": "https://github.com/advisories/GHSA-35jh-r3h4-6jhm",
  "vulnerable_versions": "<4.17.21"
}

Dependency-Track mirrors the contents of NPM Advisories and when performing an npm audit, Dependency-Track will look up ID 1002847 but that record does not exist, because that ID does not exist when performing the mirror.

Inserting another record into the DT database will result in duplicate vulnerabilities. In order for me to properly fix this, GitHub needs to ensure the value of npm_advisory_id matches what the previous npm advisory ID was. In this case it was 1673. Once this is resolved, I can create a proper fix on my end. Until then, I would recommend disabling the NPM Advisory analyzer and ensure the OSS Index analyzer is enabled.

stevespringett commented 3 years ago

https://github.com/advisories does not provide any way to contact the team regarding the feed. I have a message posted in the GitHub Security Lab Slack workspace trying to get more information on how to report the issue so it can get resolved.

Mario-Hofstaetter commented 3 years ago

Just noticed the same exception in logs after upgrading dtrack to 4.3.6 which got me nervous, good to see there is already an issue for it.

StyleT commented 3 years ago

@stevespringett I think it's bad idea to rely on GitHub team's hypothetical fix. As well as continue use of the https://registry.npmjs.org/-/npm/v1/security/advisories endpoint. Even if they'll fix it - it's just a temporarily workaround since NPM generally moves over to GitHub advisories DB. Also endpoint from above - already returns obsolete info (lates advisories available only in GitHub DB).

stevespringett commented 3 years ago

@S-Callier agreed. I'm just trying to get a short-term fix in, as DT 4.4 isn't close to being done and GitHub doesn't have an easy way to move all tickets back one release across multiple milestones.

Someone also needs to do the mapping regardless of whether GitHub does it or someone in the DT community does it.

ddurham2 commented 2 years ago

I'm also seeing this today. I'm concerned that it's leaving some vulns showing that we've fixed i.e. still showing packages that aren't in the uploaded sbom anymore.

stevespringett commented 2 years ago

NPM Advisory mirroring will cease to work after February 16, 2022. https://github.blog/changelog/2021-11-16-deprecating-non-audit-related-advisory-fetch-endpoints-for-the-npmjs-com-registry-api/

stevespringett commented 2 years ago

Observations while working on this ticket:

stevespringett commented 2 years ago

Mirroring of GitHub Advisories is nearly complete.

Screen Shot 2021-12-04 at 3 46 11 PM
stevespringett commented 2 years ago

Here's an update on this ticket. Now that all GitHub Advisories are mirrored to the DT database (just like NVD ones are), I needed to make some changes to the Internal Analyzer - specifically to add support for using Package URL to identify known vulnerabilities. Previously, the internal analyzer relied solely on CPE. With this change, it also provides a prerequisite for #96 (a private/internal vulnerability repository).

Screen Shot 2022-01-25 at 9 49 42 AM

GitHub Advisories currently supports:

DT incorporates support for all of these ecosystems when using GitHub Advisories.

stevespringett commented 2 years ago

Also debuting in v4.4 is finer control over mirroring of vulnerability sources, GitHub being one of them.

Screen Shot 2022-01-26 at 4 54 41 PM
iamrahul127 commented 2 years ago

@stevespringett With this change is following a valid setting or will be removed in near future? image

stevespringett commented 2 years ago

@iamrahul127 The npm Audit analyzer has already been removed, but it looks like I missed removing its configuration from the frontend.

github-actions[bot] commented 2 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.