[Spike] Address Snyk Critical errors for DSP & Investigate Ci capabilities

[HumairAK]

A downstream forker of DSP has scanned the DSP images via Snyk which are the same as the ones we use in upstream and found various Critical level issues that need to be addressed.

https://app.snyk.io/org/red-hat-openshift-data-science-rhods/projects?groupBy=targets&searchQuery=&sortBy=highest+severity&filters[Show]=vuln-groups&filters[Integrations]=&before&after&fromGitHubAuth=true

Acceptance criteria:

• Outline all the issues that need to be fix and create approriate gh issues with scores on their complexity
• Suggest how to integrate Snyk as part of our CI process, figure out when/where these checks should run

[HumairAK]

Okay looked into the current Snyk set up, and found that having Snyk only in downstream is super inconvenient.

Having Snyk in only downstream is not great

• We do upstream first development, so having these vulnerabilities exposed downstream is a very late in the process
• We want the fixes to go upstream first, so the automated pull request feature is totally useless if it's configured to a downstream repo
• Snyk scans downstream dockerfiles, which is also not very useful, as these dockerfiles are used for upstream builds. Downstream uses cpaas builds and those dockerfiles live in gitlab
• Snyk can be configured with github security scanning, and gh actions, but again these would be great to have in upstream instead, as that's where development happens

Free version

I was able to successfully use the free Snyk version to import upstream repositories, and send some automated prs made by our dsp-developers bot

• Free version has limited test runs, at 200 tests/mo, but I'm not sure what exactly a "test" in this metric is comprised of. But definitely not something we can probably get away with by using for PRs. Maybe nightlies.
• Easy to set up in UI
• Nightly GH actions seem straight forward

Lingering questions

I resolved most of the errors for DSP repo, as they came from just upgrading the go version for apiserver, but some errors are coming from older python libraries, which we aren't really using, not direclty anyway. For example the sdk requirements.txt is reporting:

As a critical vulnerability, but I'm not sure there's much we can do here, other than reporting it upstream and applying a fix there.

There are also certain components with similar vulnerable python packages, but we don't use these components, like visualization server. Should we just ignore these? or find a rule to ignore them in snyk?

[HumairAK]

My conclusion is that as a low hanging fruit we should follow up with a task to set up dsp-devs with the dspo/dsp repos configured to a free account upstream, but no automation to fix prs just yet. And follow up steps can be to set up alerting if vulnerabilities are found.