Recommend CII certifications for rOpenSci's packages?

[maelle]

For context, the R Consortium might recommend CII Best Practices Badge for R packages (https://www.r-consortium.org/blog/2018/07/26/should-r-consortium-recommend-cii-best-practices-badge-for-r-packages-latest-survey-results ).

Should we recommend it? It means filling a form that's partly automatically filled once one has entered a GitHub repo link. Before recommending it we might need to amend some of our guidelines to make sure the process is as smooth as possible once packages respect our guidelines.

Cc @seaaan who I see has been working on this for plater. Any experience report, Sean?

[maelle]

Note that it means one more badge... cf #56

[noamross]

I think it would be useful to do a comparison of how many things RO standards cover in CII, and I think a blog post from @seaaan or someone else going through the process would be really interesting. But I don't think we should recommend it. First, I don't think there's enough info and resources on for new authors that are based on other author's experience - we try not to push things on authors until there are the adequate guidance materials. Secondarily, I interpret this certification as being about and for core infrastructure, and not necessarily well designed for long-tail niche packages that we tend to get. It might make sense to pursue certification for some of our packages that we might deem core infrastructure.

[seaaan]

I spent an hour or so going through the form, but didn't put that much effort into it. I didn't find it to be too helpful. The main changes I made were adding a couple of sentences saying where to report bugs/feature requests/security vulnerabilities and giving some guidelines about how to contribute.

A lot of the questions didn't feel very relevant to me. For example, there's a lot about reporting security vulnerabilities, using cryptographic hashes correctly, protecting against man-in-the-middle attacks, and other security issues. Maybe I am just naive, but none of this felt relevant to my package, which is all written in pure R and doesn't do any network access or authentication. I think that @noamross's point about it being designed for core infrastructure might explain some of this.

I also felt bewildered by some of the questions, like about the use of a "dynamic analysis tool". I think this might just mean an automated test suite, but the description talks about fuzzing and other stuff (and there's a previous question about an automated test suite so probably not).

In summary, I felt like the questions asked about either things I had already done (many in response to the rOpenSci guidelines!) or things that didn't feel relevant/important, so I didn't end up making many changes to the package. To be fair though, I only really spent an hour or so looking at it and maybe the aspects that felt irrelevant to me are in fact gaps in the package that I should have addressed.

RE a blog post: I don't feel like I spent enough time with it to give it a fair hearing and I wouldn't want to write a post that's all about how it wasn't very valuable to me, because I'm sure that it's very valuable for some people!

[maelle]

Thanks a lot, both of you!

[maelle]

Having filled the form for HIBPwned, I tend to agree with @seaaan. I didn't have to change much to the package, so I don't see much value.

Moreover, even if it had value, we'd need a step-by-step explanation of what some things mean for R packages.

[jimhester]

I agree with the general sentiments as well, most of the categories either don't really apply to R packages, or are already covered by common tools in the R community (notably R CMD check).

Also agree on the security questions being somewhat off base, particularly

The project MUST have at least one primary developer who knows how to design secure software

seems to be overkill for most R packages, particularly those that contain only R code. I think if people are being honest only a tiny minority of the R community could say that requirement has been met. I certainly don't feel comfortable saying so for my packages.

[juliasilge]

I started working through the requirements for tidytext and started to get bogged down about halfway through, feeling unclear how much the questions applied to me and/or if I could honestly answer them "yes".

After looking at what the requirements are and then thinking about my own personal decision on using an R package or not, I think I would find just as helpful info in an informative GitHub (I'd check out the README, vignettes, issues, is there CI? is code coverage tracked?). I know that is multiple things to look at instead of one badge, but I don't know how helpful/applicable the badge is, in its current form.

[nealrichardson]

I just did the CII review for httptest ( !), and I think that any package that's following ROpenSci's standards should "pass" fine, assuming a liberal interpretation of what security means for an R package. If y'all were to provide guidance on how to interpret some of the sections as @maelle suggested, it wouldn't be burdensome to fill out the form. So while I agree with the consensus here that it doesn't seem to add much value, if R Consortium were to recommend it and you wanted to follow that, it doesn't seem like it would be a problem for those who onboard packages here. Indeed, the onboarding process is more rigorous because it is peer reviewed rather than self reported.

@noamross's point about core infrastructure is good, and looking at the "silver" and "gold" CII levels, maybe those standards would mean something for truly core projects.

[maelle]

Thanks everyone for chiming in! I think we can close this for now, and revisit if the CII starts getting more relevant for R packages.