feedback from Titus Brown

[jennybc]

He alerted us via Twiter: https://twitter.com/ctitusbrown/status/774279820758753280

Comments are here:

https://via.hypothes.is/https://swcarpentry.github.io/good-enough-practices-in-scientific-computing/

In follow up tweets, asks:

is there any data or numerical evidence on adoption or retention of approaches? paper claims to be evidence based but it seems like anecdata...?

cc @ctb

[jkitzes]

This is mostly resolved in 94110dfaf9df86efe5438145e8a415ec3575b71a. Two lingering issues that I haven't changed, and I think could remain as is:

• Recommends generalizing our list of "permissive open source licenses" from "MIT, BSD, or Apache" to "OSI" or something like that, to include GPL and others. I don't know if this was a conscious decision to recommend only the "more free" ones, so I've left it for now.
• I didn't have anything to add to "is there any data or numerical evidence on adoption or retention of approaches? paper claims to be evidence based but it seems like anecdata...?" I don't think that we make an evidence based claim for the paper at this point, more of a claim that it's a broad collection of anecdote. I'm comfortable leaving as is.

[ctb]

On Fri, Sep 23, 2016 at 09:52:05AM -0700, Justin Kitzes wrote:

This is mostly resolved in 94110dfaf9df86efe5438145e8a415ec3575b71a. Two lingering issues that I haven't changed, and I think could remain as is:

• Recommends generalizing our list of "permissive open source licenses" from "MIT, BSD, or Apache" to "OSI" or something like that, to include GPL and others. I don't know if this was a conscious decision to recommend only the "more free" ones, so I've left it for now.

This is a contentious enough issue that I would suggest putting in a justification or comment for recommending the more-free ones. @jakevdp has a nice blog post on it -- http://www.astrobetter.com/blog/2014/03/10/the-whys-and-hows-of-licensing-scientific-code/

• I didn't have anything to add to "is there any data or numerical evidence on adoption or retention of approaches? paper claims to be evidence based but it seems like anecdata...?" I don't think that we make an evidence based claim for the paper at this point, more of a claim that it's a broad collection of anecdote. I'm comfortable leaving as is.

I agree. Can't find the original text, but ISTR that at some point there was a statement about being evidence based; I don't see it any more.

[jkitzes]

Thanks @ctb, going to ping @gvwilson for thoughts on how to proceed on the first point.

[gvwilson]

I took out the statement about all of our claims being evidence-based because the brouhaha over the last couple of years about reproducibility and p-hacking in science (esp. psych) has made me go back and look more critically at studies in software engineering (including those in "Making Software"). Long story short, I'm not as confident as I'd like to be that some of them would stand up to close scrutiny; that, coupled with the complete lack of studies in some key areas, makes me feel I shouldn't sound as certain as I have in the past. (There are no published studies showing that version control works better than something like a shared drive or Dropbox folder.)

[jennybc]

@gvwilson Sounds good. I think it's fine to state that we are speaking in good faith from personal experience and experience teaching hundreds / thousands of people in SWC and DC contexts.