Add dependency vulnerability analysis to CCO CI/CD pipelines

[woodt]

Probably the easiest way to QA this work would be to add a package with a known vulnerability on a branch, and confirm that the CI/CD pipeline detects the problem and fails.

A "nice" package to try out is rdflib==4.2.2

[woodt]

This work is in progress. Expect to have a pull request ready end of day on Monday.

[woodt]

Oops. Turned out to be a little more complicated than I thought. E.g., there are some conflicting requirements in ckan-cloud-provisioning-api. And dhq-auth uses flask-oauthlib, which appears to be replaced by authlib.

/src # pip-compile requirements.in
Could not find a version that matches oauthlib!=2.0.3,!=2.0.4,!=2.0.5,<3.0.0,>=1.1.2,>=3.0.0 (from flask-oauthlib==0.9.5->dhq-auth==0.1.3->-r requirements.in (line 1))
Tried: 0.0.1, 0.0.2, 0.1.0, 0.1.1, 0.1.2, 0.1.3, 0.3.0, 0.3.2, 0.3.3, 0.3.4, 0.3.5, 0.3.6, 0.3.7, 0.3.8, 0.4.0, 0.4.1, 0.4.2, 0.5.0, 0.5.1, 0.6.0, 0.6.1, 0.6.2, 0.6.3, 0.7.0, 0.7.1, 0.7.2, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.1.0.post1, 1.1.1, 1.1.2, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.0.5, 2.0.6, 2.0.7, 2.0.7, 2.1.0, 2.1.0, 3.0.0, 3.0.0, 3.0.1, 3.0.1, 3.0.2, 3.0.2, 3.1.0, 3.1.0
There are incompatible versions in the resolved dependencies:
  oauthlib!=2.0.3,!=2.0.4,!=2.0.5,<3.0.0,>=1.1.2 (from flask-oauthlib==0.9.5->dhq-auth==0.1.3->-r requirements.in (line 1))
  oauthlib>=3.0.0 (from requests-oauthlib==1.2.0->flask-oauthlib==0.9.5->dhq-auth==0.1.3->-r requirements.in (line 1))

Still investigating this.

Also ... all three of these repos are on github.com, so once the requirements are pinned, I would like to just enable dependabot which will examine requirements for known vulnerabilities and updates.

[woodt]

We're not going to go with dependabot. I will implement the safety scanning as done with previous Python repos, and probably snyk for the UI repo (see https://github.com/GSA/datagov-ckan-multi/issues/186)

[woodt]

Figured out a way to resolve the conflicting dependency issue. Will finish this work (and spin up another ticket to deal with the no longer supported flask-oauthlib.

[woodt]

See also https://github.com/ViderumGlobal/ckan-cloud-provisioning-api/pull/9

[kimwdavidson]

@woodt to add QA instructions, then pass to @akariv for QA

[woodt]

I've confirmed locally that when a vulnerable library is installed, the safety scanner detects the problem and the travis shell script will report and exit. Unfortunately, the actual travis CI pipeline is failing before the scan is run -- seems to have something to do with the minikube setup.

[estebanruseler]

Iceboxed as not improving CCO currently