load advisories directly

[g-k]

Fetch and save advisories directly to the database.

With #237 package version matching this would let us:

• save us from needing to run audit CLI commands for each package for new advisories
• allow us to see what impact unpublished advisories reported to our bug bounty program might have or published advisories GH can't match (e.g. against central or before vuln alerts go out).

refs: #156

Advisory databases:

• https://cve.mitre.org/
• https://github.com/RustSec/advisory-db
• https://github.com/advisories with GraphQL API https://developer.github.com/v4/object/securityadvisory/ (we already fetch this as part of the github metadata fetcher https://github.com/mozilla-services/dependency-observatory/blob/967eb8ca4d817ed27bc29e21b6d4b857d620e3f6/depobs/scanner/models/github.py#L409-L421)
• https://www.npmjs.com/advisories scrapable by autoincrementing IDs (latest as of 2020-04-28: https://www.npmjs.com/advisories/1484) with API reverse engineered in https://dzone.com/articles/how-to-use-npm-rest-api-to-get-audit-npm-audit-res

$ curl -X POST -d '{ 
    "name": "npm_audit_test",
    "version": "1.0.0",
    "requires": {
        "marked": "^0.6.3"
    },
    "dependencies": {
        "marked": {
            "version": "0.6.3",
            "integrity": "sha1-ebq614r2OLpNUiqecVzf3SQp6UY=234"
        }
    }}' https://registry.npmjs.org/-/npm/v1/security/audits | jq ''
{                                                                                                                                                                                                                                                                                                                                                                          
  "actions": [                                                                                                                                                                                                                                                                                                                                                             
    {                                                                                                                                                                                                                                                                                                                                                                      
      "isMajor": true,
      "action": "install",
      "resolves": [
        {
          "id": 1076,
          "path": "marked",
          "dev": false,
          "optional": false,
          "bundled": false
        }
      ],
      "module": "marked",
      "target": "1.0.0"
    }
  ],
  "advisories": {
    "1076": {
      "findings": [
        {
          "version": "0.6.3",
          "paths": [
            "marked"
          ]
        }
      ],
      "id": 1076,
      "created": "2019-07-17T20:26:14.214Z",
      "updated": "2019-07-25T18:15:12.932Z",
      "deleted": null,
      "title": "Regular Expression Denial of Service",
      "found_by": {
        "link": "",
        "name": "Bart Grantham"
      },
      "reported_by": {
        "link": "",
        "name": "Bart Grantham"
      },
      "module_name": "marked",
      "cves": [],
      "vulnerable_versions": ">=0.4.0 <0.7.0",
      "patched_versions": ">=0.7.0",
      "overview": "Affected versions of `marked` are vulnerable to Regular Expression Denial of Service (ReDoS). The `_label` subrule may significantly degrade parsing performance of malformed input.",
      "recommendation": "Upgrade to version 0.7.0 or later.",
      "references": "",
      "access": "public",
      "severity": "low",
      "cwe": "CWE-400",
      "metadata": {
        "module_type": "",
        "exploitability": 5,
        "affected_components": ""
      },
      "url": "https://npmjs.com/advisories/1076"
    }
  },
  "muted": [],
  "metadata": {
    "vulnerabilities": {
      "info": 0,
      "low": 1,
      "moderate": 0,
      "high": 0,
      "critical": 0
    },
    "dependencies": 1,
    "devDependencies": 0,
    "optionalDependencies": 0,
    "totalDependencies": 1
  }
}

Others are public with embargo (pyup's https://github.com/pyupio/safety-db, snyk.io).

There's also standardization work in the package url area https://github.com/package-url/purl-spec

[psiinon]

I was hoping that we could just load all of the NPM advisories directly, but that could be tricky. The API available require a package name and version to be supplied. The advisory pages do actually contain JSON, but its deep in the HTML and I'm concerned that extracting and parsing it could be very flaky. Calling the available API when we know the package and version is great, so lets do that. But I'm also worried that it will make it difficult to take account of new advisories. If we just use the existing API then we'll have to regularly poll for all of the packages and versions we have in our db, and that just gets more and more time consuming (and expensive).

So - I think we should use the existing API when we score packages and versions. But we should also parse https://www.npmjs.com/advisories and maintain a list of package names and the associated advisories. When we detect a new advisory we then call the existing API for all of the versions we have for that package - a significantly lower number of calls.

[g-k]

Huh yeah, there might be a slight delay compared to hitting NPM directly, but we could go the GH API route for the GraphQL equivalent of https://github.com/advisories?query=ecosystem%3Anpm+sort%3Aupdated-desc

Bonus would be that we can cover new GH-only advisories and other ecosystems.

[g-k]

There's some example data for that in https://github.com/mozilla-services/dependency-observatory/blob/master/tests/scanner/fixtures/REPO_VULN_ALERTS_first_page_response_next_page.json

[g-k]

The big question would be whether GH would let us pull that data without having an impacted repo.

I think we'd have to play with the GQL API explorer to confirm.

[psiinon]

Is the GH data a superset of the NPM data? I thought it was missing a load, but I can check...

[g-k]

I think the GH data should be, but it might be slightly delayed.

[g-k]

The GQL API supports ordering and filtering by ecosystem, package name, and severity.

https://developer.github.com/v4/object/securityadvisory/#connections

[g-k]

Looks like we can pull it from the GQL API, but I'd have to confirm it covers the npm advisories:

{
  securityVulnerabilities(ecosystem: NPM, first: 10, orderBy: {field: UPDATED_AT, direction: DESC}) {
    nodes {
      advisory {
        databaseId
        description
        ghsaId
        id
        identifiers {
          type
          value
        }
        origin
        permalink
        publishedAt
        references {
          url
        }
        severity
        summary
        updatedAt
        withdrawnAt
      }
      firstPatchedVersion {
        identifier
      }
      severity
      package {
        ecosystem
        name
      }
      updatedAt
      vulnerableVersionRange
    }
    pageInfo {
      endCursor
      hasNextPage
      hasPreviousPage
      startCursor
    }
    totalCount
  }
}

{
  "data": {
    "securityVulnerabilities": {
      "nodes": [
        {
          "advisory": {
            "databaseId": 2075,
            "description": "### Impact\nPassing HTML from untrusted sources - even after sanitizing it - to one of jQuery's DOM manipulation methods (i.e. `.html()`, `.append()`, and others) may execute untrusted code.\n\n### Patches\nThis problem is patched in jQuery 3.5.0.\n\n### Workarounds\nTo workaround the issue without upgrading, adding the following to your code:\n\n```js\njQuery.htmlPrefilter = function( html ) {\n\treturn html;\n};\n```\n\nYou need to use at least jQuery 1.12/2.2 or newer to be able to apply this workaround.\n\n### References\nhttps://blog.jquery.com/2020/04/10/jquery-3-5-0-released/\nhttps://jquery.com/upgrade-guide/3.5/\n\n### For more information\nIf you have any questions or comments about this advisory:\n* Open an issue in [the jQuery repo](https://github.com/jquery/jquery/issues)\n* Email us at [security@jquery.com](mailto:security@jquery.com)",
            "ghsaId": "GHSA-gxr4-xjj5-5px2",
            "id": "MDE2OlNlY3VyaXR5QWR2aXNvcnlHSFNBLWd4cjQteGpqNS01cHgy",
            "identifiers": [
              {
                "type": "GHSA",
                "value": "GHSA-gxr4-xjj5-5px2"
              },
              {
                "type": "CVE",
                "value": "CVE-2020-11022"
              }
            ],
            "origin": "UNSPECIFIED",
            "permalink": "https://github.com/advisories/GHSA-gxr4-xjj5-5px2",
            "publishedAt": "2020-04-29T22:18:55Z",
            "references": [
              {
                "url": "https://github.com/jquery/jquery/security/advisories/GHSA-gxr4-xjj5-5px2"
              },
              {
                "url": "https://nvd.nist.gov/vuln/detail/CVE-2020-11022"
              },
              {
                "url": "https://github.com/advisories/GHSA-gxr4-xjj5-5px2"
              }
            ],
            "severity": "MODERATE",
            "summary": "Potential XSS vulnerability in jQuery",
            "updatedAt": "2020-04-29T22:18:55Z",
            "withdrawnAt": null
          },
          "firstPatchedVersion": {
            "identifier": "3.5.0"
          },
          "severity": "MODERATE",
          "package": {
            "ecosystem": "NPM",
            "name": "jquery"
          },
          "updatedAt": "2020-04-29T22:18:38Z",
          "vulnerableVersionRange": ">= 1.2, < 3.5.0"
        },
        {
          "advisory": {
            "databaseId": 2074,
            "description": "### Impact\nPassing HTML containing `<option>` elements from untrusted sources - even after sanitizing it - to one of jQuery's DOM manipulation methods (i.e. `.html()`, `.append()`, and others) may execute untrusted code.\n\n### Patches\nThis problem is patched in jQuery 3.5.0.\n\n### Workarounds\nTo workaround this issue without upgrading, use [DOMPurify](https://github.com/cure53/DOMPurify) with its `SAFE_FOR_JQUERY` option to sanitize the HTML string before passing it to a jQuery method.\n\n### References\nhttps://jquery.com/upgrade-guide/3.5/\n\n### For more information\nIf you have any questions or comments about this advisory:\n* Open an issue in [the jQuery repo](https://github.com/jquery/jquery/issues)\n* Email us at [security@jquery.com](security@jquery.com)",
            "ghsaId": "GHSA-jpcq-cgw6-v4j6",
            "id": "MDE2OlNlY3VyaXR5QWR2aXNvcnlHSFNBLWpwY3EtY2d3Ni12NGo2",
            "identifiers": [
              {
                "type": "GHSA",
                "value": "GHSA-jpcq-cgw6-v4j6"
              },
              {
                "type": "CVE",
                "value": "CVE-2020-11023"
              }
            ],
            "origin": "UNSPECIFIED",
            "permalink": "https://github.com/advisories/GHSA-jpcq-cgw6-v4j6",
            "publishedAt": "2020-04-29T22:19:14Z",
            "references": [
              {
                "url": "https://github.com/jquery/jquery/security/advisories/GHSA-jpcq-cgw6-v4j6"
              },
              {
                "url": "https://nvd.nist.gov/vuln/detail/CVE-2020-11023"
              },
              {
                "url": "https://github.com/advisories/GHSA-jpcq-cgw6-v4j6"
              }
            ],
            "severity": "MODERATE",
            "summary": "Potential XSS vulnerability in jQuery",
            "updatedAt": "2020-04-29T22:26:56Z",
            "withdrawnAt": null
          },
          "firstPatchedVersion": {
            "identifier": "3.5.0"
          },
          "severity": "MODERATE",
          "package": {
            "ecosystem": "NPM",
            "name": "jquery"
          },
          "updatedAt": "2020-04-29T22:15:52Z",
          "vulnerableVersionRange": ">= 1.0.3, < 3.5.0"
        },
        {
          "advisory": {
            "databaseId": 2073,
            "description": "### Impact\nIf consumers of the http-client:\n  1. make an http request with an authorization header\n  2. that request leads to a redirect (302) and\n  3. the redirect url redirects to another domain or hostname \n\nThe authorization header will get passed to the other domain.\n\nNote that since this library is for actions, the GITHUB_TOKEN that is available in actions is generated and scoped per job with [these permissions](https://help.github.com/en/actions/configuring-and-managing-workflows/authenticating-with-the-github_token#permissions-for-the-github_token).\n\n### Patches\nThe problem is fixed in 1.0.8 at [npm here](https://www.npmjs.com/package/@actions/http-client).  In 1.0.8, the authorization header is stripped before making the redirected request if the hostname is different.\n\n### Workarounds\nNone.\n\n### References\nhttps://github.com/actions/http-client/pull/27\n\n### For more information\nIf you have any questions or comments about this advisory:\n* Open an issue in https://github.com/actions/http-client/issues",
            "ghsaId": "GHSA-9w6v-m7wp-jwg4",
            "id": "MDE2OlNlY3VyaXR5QWR2aXNvcnlHSFNBLTl3NnYtbTd3cC1qd2c0",
            "identifiers": [
              {
                "type": "GHSA",
                "value": "GHSA-9w6v-m7wp-jwg4"
              },
              {
                "type": "CVE",
                "value": "CVE-2020-11021"
              }
            ],
            "origin": "UNSPECIFIED",
            "permalink": "https://github.com/advisories/GHSA-9w6v-m7wp-jwg4",
            "publishedAt": "2020-04-29T17:58:53Z",
            "references": [
              {
                "url": "https://github.com/actions/http-client/security/advisories/GHSA-9w6v-m7wp-jwg4"
              },
              {
                "url": "https://nvd.nist.gov/vuln/detail/CVE-2020-11021"
              },
              {
                "url": "https://github.com/advisories/GHSA-9w6v-m7wp-jwg4"
              }
            ],
            "severity": "MODERATE",
            "summary": "Http request which redirect to another hostname do not strip authorization header in @actions/http-client",
            "updatedAt": "2020-04-29T17:58:53Z",
            "withdrawnAt": null
          },
          "firstPatchedVersion": {
            "identifier": "1.0.8"
          },
          "severity": "MODERATE",
          "package": {
            "ecosystem": "NPM",
            "name": "@actions/http-client"
          },
          "updatedAt": "2020-04-29T17:58:43Z",
          "vulnerableVersionRange": "< 1.0.8"
        },
        {
          "advisory": {
            "databaseId": 2068,
            "description": "npm-programmatic through 0.0.12 is vulnerable to Command Injection.The packages and option properties are concatenated together without any validation and are used by the 'exec' function directly.",
            "ghsaId": "GHSA-426h-24vj-qwxf",
            "id": "MDE2OlNlY3VyaXR5QWR2aXNvcnlHSFNBLTQyNmgtMjR2ai1xd3hm",
            "identifiers": [
              {
                "type": "GHSA",
                "value": "GHSA-426h-24vj-qwxf"
              },
              {
                "type": "CVE",
                "value": "CVE-2020-7614"
              }
            ],
            "origin": "UNSPECIFIED",
            "permalink": "https://github.com/advisories/GHSA-426h-24vj-qwxf",
            "publishedAt": "2020-04-23T20:09:09Z",
            "references": [
              {
                "url": "https://nvd.nist.gov/vuln/detail/CVE-2020-7614"
              },
              {
                "url": "https://github.com/advisories/GHSA-426h-24vj-qwxf"
              }
            ],
            "severity": "HIGH",
            "summary": "npm-programmatic is vulnerable to Command Injection",
            "updatedAt": "2020-04-23T20:09:09Z",
            "withdrawnAt": null
          },
          "firstPatchedVersion": null,
          "severity": "HIGH",
          "package": {
            "ecosystem": "NPM",
            "name": "npm-programmatic"
          },
          "updatedAt": "2020-04-23T20:02:35Z",
          "vulnerableVersionRange": "< 0.0.12"
        },
        {
          "advisory": {
            "databaseId": 2049,
            "description": "lix through 15.8.7 allows man-in-the-middle attackers to execute arbitrary code by modifying the HTTP client-server data stream so that the Location header is associated with attacker-controlled executable content in the postDownload field.",
            "ghsaId": "GHSA-q8xg-8xwf-m598",
            "id": "MDE2OlNlY3VyaXR5QWR2aXNvcnlHSFNBLXE4eGctOHh3Zi1tNTk4",
            "identifiers": [
              {
                "type": "GHSA",
                "value": "GHSA-q8xg-8xwf-m598"
              },
              {
                "type": "CVE",
                "value": "CVE-2020-10800"
              }
            ],
            "origin": "UNSPECIFIED",
            "permalink": "https://github.com/advisories/GHSA-q8xg-8xwf-m598",
            "publishedAt": "2020-04-16T03:14:59Z",
            "references": [
              {
                "url": "https://nvd.nist.gov/vuln/detail/CVE-2020-10800"
              },
              {
                "url": "https://github.com/advisories/GHSA-q8xg-8xwf-m598"
              }
            ],
            "severity": "MODERATE",
            "summary": "man-in-the-middle attack in lix",
            "updatedAt": "2020-04-16T03:14:59Z",
            "withdrawnAt": null
          },
          "firstPatchedVersion": null,
          "severity": "MODERATE",
          "package": {
            "ecosystem": "NPM",
            "name": "lix"
          },
          "updatedAt": "2020-04-16T03:10:40Z",
          "vulnerableVersionRange": "< 15.8.11"
        },
        {
          "advisory": {
            "databaseId": 2046,
            "description": "node-uuid before 1.4.4 uses insufficiently random data to create a GUID, which could make it easier for attackers to have unspecified impact via brute force guessing.",
            "ghsaId": "GHSA-265q-28rp-chq5",
            "id": "MDE2OlNlY3VyaXR5QWR2aXNvcnlHSFNBLTI2NXEtMjhycC1jaHE1",
            "identifiers": [
              {
                "type": "GHSA",
                "value": "GHSA-265q-28rp-chq5"
              },
              {
                "type": "CVE",
                "value": "CVE-2015-8851"
              }
            ],
            "origin": "UNSPECIFIED",
            "permalink": "https://github.com/advisories/GHSA-265q-28rp-chq5",
            "publishedAt": "2020-04-16T03:14:50Z",
            "references": [
              {
                "url": "https://nvd.nist.gov/vuln/detail/CVE-2015-8851"
              },
              {
                "url": "https://github.com/advisories/GHSA-265q-28rp-chq5"
              }
            ],
            "severity": "MODERATE",
            "summary": "Insufficiently random GUIDs in node-uuid",
            "updatedAt": "2020-04-16T03:14:50Z",
            "withdrawnAt": null
          },
          "firstPatchedVersion": {
            "identifier": "1.4.4"
          },
          "severity": "MODERATE",
          "package": {
            "ecosystem": "NPM",
            "name": "node-uuid"
          },
          "updatedAt": "2020-04-16T02:58:24Z",
          "vulnerableVersionRange": "< 1.4.4"
        },
        {
          "advisory": {
            "databaseId": 2045,
            "description": "sanitize-html before 1.4.3 has XSS.",
            "ghsaId": "GHSA-3j7m-hmh3-9jmp",
            "id": "MDE2OlNlY3VyaXR5QWR2aXNvcnlHSFNBLTNqN20taG1oMy05am1w",
            "identifiers": [
              {
                "type": "GHSA",
                "value": "GHSA-3j7m-hmh3-9jmp"
              },
              {
                "type": "CVE",
                "value": "CVE-2016-1000237"
              }
            ],
            "origin": "UNSPECIFIED",
            "permalink": "https://github.com/advisories/GHSA-3j7m-hmh3-9jmp",
            "publishedAt": "2020-04-16T03:14:47Z",
            "references": [
              {
                "url": "https://nvd.nist.gov/vuln/detail/CVE-2016-1000237"
              },
              {
                "url": "https://github.com/advisories/GHSA-3j7m-hmh3-9jmp"
              }
            ],
            "severity": "MODERATE",
            "summary": "XSS in sanitize-html",
            "updatedAt": "2020-04-16T03:14:47Z",
            "withdrawnAt": null
          },
          "firstPatchedVersion": {
            "identifier": "1.4.3"
          },
          "severity": "MODERATE",
          "package": {
            "ecosystem": "NPM",
            "name": "sanitize-html"
          },
          "updatedAt": "2020-04-16T02:54:09Z",
          "vulnerableVersionRange": "< 1.4.3"
        },
        {
          "advisory": {
            "databaseId": 2044,
            "description": "### Impact\n_What kind of vulnerability is it? Who is impacted?_\n\nIf you're running a vulnerable application on your computer and an attacker can trick you into visiting a malicious website, they could use [DNS rebinding](https://en.wikipedia.org/wiki/DNS_rebinding) and [CSRF](https://en.wikipedia.org/wiki/Cross-site_request_forgery) attacks to read/write to vulnerable applications. \n\n**There is no evidence that suggests that this has been used in the wild.**\n\n### Patches\n_Has the problem been patched? What versions should users upgrade to?_\n\nYes, 2.15.0.\n\n### Workarounds\n_Is there a way for users to fix or remediate the vulnerability without upgrading?_\n\nNo.\n\n### References\n_Are there any links users can visit to find out more?_\n\nNo.\n\n### For more information\nIf you have any questions or comments about this advisory:\n* Open an issue in [fraction/oasis](http://github.com/fraction/oasis)\n* Email me at [christianbundy@fraction.io](mailto:christianbundy@fraction.io)",
            "ghsaId": "GHSA-j438-45hc-vjhm",
            "id": "MDE2OlNlY3VyaXR5QWR2aXNvcnlHSFNBLWo0MzgtNDVoYy12amht",
            "identifiers": [
              {
                "type": "GHSA",
                "value": "GHSA-j438-45hc-vjhm"
              },
              {
                "type": "CVE",
                "value": "CVE-2020-11003"
              }
            ],
            "origin": "UNSPECIFIED",
            "permalink": "https://github.com/advisories/GHSA-j438-45hc-vjhm",
            "publishedAt": "2020-04-16T03:14:39Z",
            "references": [
              {
                "url": "https://github.com/fraction/oasis/security/advisories/GHSA-j438-45hc-vjhm"
              },
              {
                "url": "https://nvd.nist.gov/vuln/detail/CVE-2020-11003"
              },
              {
                "url": "https://github.com/advisories/GHSA-j438-45hc-vjhm"
              }
            ],
            "severity": "MODERATE",
            "summary": "CSRF and DNS Rebinding in Oasis",
            "updatedAt": "2020-04-16T03:14:39Z",
            "withdrawnAt": null
          },
          "firstPatchedVersion": {
            "identifier": "2.15.0"
          },
          "severity": "MODERATE",
          "package": {
            "ecosystem": "NPM",
            "name": "@fraction/oasis"
          },
          "updatedAt": "2020-04-15T22:57:24Z",
          "vulnerableVersionRange": "< 2.15.0"
        },
        {
          "advisory": {
            "databaseId": 2015,
            "description": "ctorName in index.js in kind-of v6.0.2 allows external user input to overwrite certain internal attributes via a conflicting name, as demonstrated by 'constructor': {'name':'Symbol'}. Hence, a crafted payload can overwrite this builtin attribute to manipulate the type detection result.",
            "ghsaId": "GHSA-6c8f-qphg-qjgp",
            "id": "MDE2OlNlY3VyaXR5QWR2aXNvcnlHSFNBLTZjOGYtcXBoZy1xamdw",
            "identifiers": [
              {
                "type": "GHSA",
                "value": "GHSA-6c8f-qphg-qjgp"
              },
              {
                "type": "CVE",
                "value": "CVE-2019-20149"
              }
            ],
            "origin": "UNSPECIFIED",
            "permalink": "https://github.com/advisories/GHSA-6c8f-qphg-qjgp",
            "publishedAt": "2020-03-31T15:59:54Z",
            "references": [
              {
                "url": "https://nvd.nist.gov/vuln/detail/CVE-2019-20149"
              },
              {
                "url": "https://github.com/advisories/GHSA-6c8f-qphg-qjgp"
              }
            ],
            "severity": "MODERATE",
            "summary": "Type checking vulnerability in kind-of",
            "updatedAt": "2020-04-14T20:47:33Z",
            "withdrawnAt": null
          },
          "firstPatchedVersion": {
            "identifier": "6.0.3"
          },
          "severity": "MODERATE",
          "package": {
            "ecosystem": "NPM",
            "name": "kind-of"
          },
          "updatedAt": "2020-04-14T20:47:33Z",
          "vulnerableVersionRange": "= 6.0.2"
        },
        {
          "advisory": {
            "databaseId": 2036,
            "description": "devcert-sanscache before 0.4.7 allows remote attackers to execute arbitrary code or cause a Command Injection via the exec function. The variable `commonName` controlled by user input is used as part of the `exec` function without any sanitization.",
            "ghsaId": "GHSA-4gp3-p7ph-x2jr",
            "id": "MDE2OlNlY3VyaXR5QWR2aXNvcnlHSFNBLTRncDMtcDdwaC14Mmpy",
            "identifiers": [
              {
                "type": "GHSA",
                "value": "GHSA-4gp3-p7ph-x2jr"
              },
              {
                "type": "CVE",
                "value": "CVE-2019-10778"
              }
            ],
            "origin": "UNSPECIFIED",
            "permalink": "https://github.com/advisories/GHSA-4gp3-p7ph-x2jr",
            "publishedAt": "2020-04-14T23:09:00Z",
            "references": [
              {
                "url": "https://nvd.nist.gov/vuln/detail/CVE-2019-10778"
              },
              {
                "url": "https://github.com/advisories/GHSA-4gp3-p7ph-x2jr"
              }
            ],
            "severity": "HIGH",
            "summary": "OS Command Injection in devcert-sanscache",
            "updatedAt": "2020-04-14T23:09:00Z",
            "withdrawnAt": null
          },
          "firstPatchedVersion": {
            "identifier": "0.4.7"
          },
          "severity": "HIGH",
          "package": {
            "ecosystem": "NPM",
            "name": "devcert-sanscache"
          },
          "updatedAt": "2020-04-14T15:32:51Z",
          "vulnerableVersionRange": "< 0.4.7"
        }
      ],
      "pageInfo": {
        "endCursor": "Y3Vyc29yOnYyOpK0MjAyMC0wNC0xNFQxNTozMjo1MVrNDLw=",
        "hasNextPage": true,
        "hasPreviousPage": false,
        "startCursor": "Y3Vyc29yOnYyOpK0MjAyMC0wNC0yOVQyMjoxODozOFrNDPA="
      },
      "totalCount": 769
    }
  }
}

[g-k]

Simon mentions: https://registry.npmjs.org/-/npm/v1/security/advisories/1

[g-k]

Possible implementation using the npm registry endpoint:

1. add an async fetch_advisories function probably taking an advisory ID or IDS to the npm registry client similar to fetch_npm_registry_metadata: https://github.com/mozilla-services/dependency-observatory/blob/b687e82844ce70d7b181719cf9aed8052873ad79/depobs/scanner/clients/npm_registry.py#L71-L105

2. extract a save_advisory function into database models from https://github.com/mozilla-services/dependency-observatory/blob/b687e82844ce70d7b181719cf9aed8052873ad79/depobs/scanner/pipelines/save_to_db.py#L156-L181 (edit: #272 moved these to database.models and #275 extracted a insert_advisories fn in db.models)

3. add a task that gets the highest npm advisory ID in our DB and fetches more until it gets a 404 like fetch_and_save_registry_entries to worker tasks https://github.com/mozilla-services/dependency-observatory/blob/b687e82844ce70d7b181719cf9aed8052873ad79/depobs/worker/tasks.py#L341-L368

4. call the task after a scan before scoring e.g. https://github.com/mozilla-services/dependency-observatory/blob/b687e82844ce70d7b181719cf9aed8052873ad79/depobs/worker/tasks.py#L229 (optionally set up a celery beat worker with a cron schedule)

5. If necessary, tweak npms aiohttp client config for advisories (has format default args then we can override them in following lines) https://github.com/mozilla-services/dependency-observatory/blob/b687e82844ce70d7b181719cf9aed8052873ad79/depobs/website/config.py#L113-L134

6. optionally add a query to find missing or incomplete advisories to backfill like get_package_names_with_missing_npms_io_scores https://github.com/mozilla-services/dependency-observatory/blob/b687e82844ce70d7b181719cf9aed8052873ad79/depobs/database/models.py#L808

[g-k]

@alex-bellon you can ignore https://github.com/mozilla-services/dependency-observatory/issues/236#issuecomment-624139215 and probably reuse https://github.com/alex-bellon/internship-mozilla/blob/f8e6a9ba28b6ddb91b4061667e857411a4293467/mozilla-services/find-vulnerable-packages.py#L159-L254 (don't worry about the existing GH client & quiz since we'll probably replace that)

[g-k]

worker / admin commands landed in https://github.com/mozilla-services/dependency-observatory/pull/398 and https://github.com/mozilla-services/dependency-observatory/pull/394