extending to other purls

JosiahParry commented 4 years ago

This pull request is inspired by the dialogue around https://github.com/sonatype-nexus-community/oysteR/issues/18 and https://github.com/sonatype-nexus-community/oysteR/issues/23 as well as implements functionality to address reproducible package environments such as virtual env and renv.

Creates a function audit_pkgs() which can be used to audit any of the OSS Index ecosystems.
Creates function for auditing requirements.txt and renv.lock files.
Includes base and recommended packages in get_pkgs() so that audit_deps() doesn't skip over base installations.
Marks internal functions as internal

`audit_pkgs`

This function is intended to extend the functionality to oysteR to any other supported purl. There are three main arguments pkg, version, and type. pkg is a character vector of the packages to be searched for in the OSS Index. By default version = "*" which searches for vulnerabilities for all package versions. type refers to the repository type this is set to "cran" by default but can be changed to conda, npm, etc. Currently there is no stop for packages that are not listed in the Ecosystems. This lends itself to the conversation in https://github.com/sonatype-nexus-community/oysteR/issues/20. A check can be instituted (probably in the new gen_purl()) that validates if any types are unsupported.

Auditing requirements.txt and renv.lock

One topic that I discuss frequently in my work is the need for reproducible environments.

In the R space there are the libraries packrat and renv where renv is intended to supersede packrat.

In the python space, as I understand it, the two most common methods are via pip and conda where pip creates a requirements.txt and conda creates a environment.yml file. In an effort to support further language interoperability and bilingual project, the renv package supports restoring package environments from pip and conda. renv will utilize pip freeze to create a requirements.txt if one is not already created.

In the spirit of further reproducibility and also audit-ability of those environments, two functions were created to parse these file and run packages through their respective repositories.

`audit_req_txt()`

audit_req_txt() takes a file path input, reads the requirements.txt and sends the packages and their versions to the OSS Index with type = "pypi". The type is set to PyPI since pip is a product of it. Open to taking a type argument so this can be set.

I am unsure how robust this function is in parsing requirements.txt. I don't think it could handled comments in the file etc.

`audit_renv_lock()`

audit_renv_lock() reads an renv.lock file and extracts the package names and versions. It then searches the OSS Index with type = "cran" to return results. Note here, however, that renv.lock files will list package sources this could be github, or an internal package repository. Given that OSS Index only works with packages hosted on CRAN, this makes sense.

Aside: after further thought, it would make sense for a single R OSS index envrionment and excluding GitHub. Most R packages that are of some significance make their way to CRAN at some point. Bioconductor still remains important. However, if OSS Index pulls from cvedetails.com a la https://www.cvedetails.com/cve/CVE-2016-8714/, I imagine there might not be a perfect delineation between CRAN, bioconductor, or Github making searching for R packages more difficult.

csgillespie commented 4 years ago

Thanks @JosiahParry I'll try and look through this weekend.

csgillespie commented 4 years ago

@JosiahParry Looks nice. I've made a few tweaks to your PR on this branch (https://github.com/sonatype-nexus-community/oysteR/tree/updates) but my git skills aren't up to merging it into your PR (@DarthHater can you help here?)

A summary of my changes:

bookkeeping: update NEWS and version
Add in some line breaks for the docs
Use your gen_purls() in the audit_deps()
Specify the directory when parsing the renv and reqs' files
Use the superior = to -> ;)

TODO/ Discuss

It would be nice to have a simple test for your file parsing
Should we change audit_dep() to audit_r_deps() in light of your excellent PR?

JosiahParry commented 4 years ago

This all sounds good to me! With the exception of an = for object assignment but that's a hill i'll die on at a later date ;)

I'll give your PR a poke around. The extent of my git knowledge is essentially "tear it down and do it again" so this works for me!

JosiahParry commented 4 years ago

oh, would it be possible to add myself as an author/contributor in the description file? I'd love some bragging rights. 🙃

csgillespie commented 4 years ago

Authorship: of course! I should have suggested it - sorry.

@DarthHater Can you advise on what Sonatype's usual mechanism?

csgillespie commented 4 years ago

Closing this PR - see #25

sonatype-nexus-community / oysteR