osv reports pyyaml needs upgrade when it is already upgraded

[matthewdeanmartin]

Hello, I can't tell is this is because osv is corrupt and y'all can't do anything about it or if pip-audit can't do version number comparisons. I'm still learning, but I can't find a way to skip this.

It only takes one unresolvable false positive and I have to disable a tool like this from the build server. safety has a --ignore switch, does pip-audit have one?

$ pip-audit -s osv
Found 1 known vulnerabilities in 1 packages
Name   Version ID             Fix Versions
------ ------- -------------- ------------
pyyaml 5.4.1   PYSEC-2021-142 5.4

$ pip freeze | grep PyY
PyYAML==5.4.1

[woodruffw]

Huh! Yeah, that's either some bad data coming from OSV or a version compare bug on our side.

Do you see the same error when you use the default (PyPI) vulnerability service?

[woodruffw]

Looking at the code, I strongly suspect this is bad data on the OSV side: we query OSV for all vulnerabilities matching pyyaml==5.4.1, and it should only return fix versions that don't match the request. We don't validate that, since it's (IIUC) an intended invariant of the API.

[woodruffw]

Yeah, here's the raw response from OSV:

{
  "vulns": [
    {
      "id": "PYSEC-2021-142",
      "details": "A vulnerability was discovered in the PyYAML library in versions before 5.4, where it is susceptible to arbitrary code execution when it processes untrusted YAML files through the full_load method or with the FullLoader loader. Applications that use the library to process untrusted input may be vulnerable to this flaw. This flaw allows an attacker to execute arbitrary code on the system by abusing the python/object/new constructor. This flaw is due to an incomplete fix for CVE-2020-1747.",
      "aliases": [
        "CVE-2020-14343"
      ],
      "modified": "2021-08-27T03:22:18.913334Z",
      "published": "2021-02-09T21:15:00Z",
      "references": [
        {
          "type": "REPORT",
          "url": "https://bugzilla.redhat.com/show_bug.cgi?id=1860466"
        }
      ],
      "affected": [
        {
          "package": {
            "name": "pyyaml",
            "ecosystem": "PyPI",
            "purl": "pkg:pypi:pyyaml"
          },
          "ranges": [
            {
              "type": "ECOSYSTEM",
              "events": [
                {
                  "introduced": "0"
                },
                {
                  "fixed": "5.4"
                }
              ]
            }
          ],
          "versions": [
            "3.01",
            "3.02",
            "3.03",
            "3.04",
            "3.05",
            "3.06",
            "3.07",
            "3.08",
            "3.09",
            "3.10",
            "3.11",
            "3.12",
            "3.13",
            "3.13b1",
            "3.13rc1",
            "4.2b1",
            "4.2b2",
            "4.2b4",
            "5.1",
            "5.1.1",
            "5.1.2",
            "5.1b1",
            "5.1b3",
            "5.1b5",
            "5.1b7",
            "5.2",
            "5.2b1",
            "5.3",
            "5.3.1",
            "5.3b1",
            "5.4b1",
            "5.4b2"
          ],
          "database_specific": {
            "source": "https://github.com/pypa/advisory-db/blob/main/vulns/pyyaml/PYSEC-2021-142.yaml"
          }
        }
      ]
    }
  ]
}

So 5.4.1 should not return this response, since it's not listed in the vulnerable versions. Looks very likely to be a data error.

[matthewdeanmartin]

It takes 16 seconds, but with no switches, I get "no problem"

$ time pip-audit
WARNING:pip_audit._service.pypi:Failed to read from cache directory, 
performance may be degraded:
 [Errno 22] Invalid argument: 'c:\\users\\matth\\appdata\\local\\pip\\cache\r\\http\\8\\a\\5\\8\\3\\8a58387382be80a4c8c82318278eef0ae97a371dd8b8b5a61af54380'

No known vulnerabilities found

real    0m16.607s
user    0m0.015s
sys     0m0.015s

[woodruffw]

Gotcha. That performance is terrible, and looks related to the other cache issues we've seen with Windows (#169).

So yes, this looks like an upstream data error.

[woodruffw]

cc @di / @oliverchang for triage -- I can create an upstream issue with OSV or leave it be, whatever's best.

[woodruffw]

@matthewdeanmartin Out of curiosity: it looks like you're running in a Unix shell, but your cache directory looks like a Windows one. Are you running on Windows with WSL/some other POSIX layer?

[di]

I filed https://github.com/google/osv/issues/272, let's take the discussion about windows to #169.

@matthewdeanmartin, out of curiosity, why are you using the OSV service over PyPI?

[matthewdeanmartin]

OSV has pyyaml screwed up, Pypi doesn't (according to pip-audit)

HOWEVER, I created a dockerfile (alpine python) so I could run pip-audit vs pip-audit -r requirements.txt and the -r version is unbearably slow (I've never seen it finish.) So I've repro'd the hang in alpine and OFF VPN, so my .pypirc and VPN isn not the cause.

Worked example on github: https://github.com/matthewdeanmartin/pip_audit_in_docker/

If you'd like I can open a bug for the perf for -r

[woodruffw]

If you'd like I can open a bug for the perf for -r

Yes, please do! Please include the host information for your Docker setup as well (Docker's filesystem performance on macOS and Windows is notoriously bad, so it's possible that we're hitting some kind of worse case with the cache we use).

[woodruffw]

Closing this for now, since we (will) have other issues for the perf problems.

[matthewdeanmartin]

Sorry, I misread your question regarding why I tried the osv switch... I haven't researched this project beyond the youtube vid & the README, so I really do not understand how the vulnerabilities lists compare across the similar products in this space. If I had to guess, all of these projects are querying the exact same DB, I wish people pushing these would explain if they're all the same. (for all I known osv is the same as pypi)

• safety
• pipenv (vendorizes safety?)
• ochrana
• pipenv
• synk

[woodruffw]

There's a YouTube video? 🙂

W/R/T the vulnerability services: we don't use Safety's, or Ochrana's or Snyk's. As far as I know, each of those use their own independently maintained DBs.

Ours are:

• OSV, which is a Google-maintained vulnerability database for multiple languages
• PyPI, which is the official source of package metadata for the Python packaging world

AFAIK, pip-audit is also the only tool that uses the two sources. I'm pretty sure of that with PyPI; less sure for OSV.

[di]

There's a YouTube video? 🙂

I had the same reaction, and then I remembered I gave a sneak peek of pip-audit in my PackagingCon talk: https://www.youtube.com/watch?v=PlGcg-MUtXk

[di]

@matthewdeanmartin FYI, this should be fixed by https://github.com/google/osv/pull/273/.