Please leave feedback here

[gvwilson]

We welcome feedback on this draft - please either:

1. Add a comment to this issue.
2. File a separate issue in this repository.
3. Submit a pull request proposing a change to robust-software.tex (by preference) or index.md.
4. Email the authors.

[morgantaschuk]

Thanks to our pre-print reviewers: jon_petters, morungos, froggleston, JungeAlexander, apmasell, michaelhoffman, Moran Neuhof, Spencer Smith, Radovan Bast, Patrick O'Beirne, Phillip Lord , Konrad Hinsen, Emily Davenport, ctitusbrown, aurelg

If I've missed you or misinterpreted your comments, please leave a comment and let us know.

On what "Robust" means

• You say that it's what makes the software usable by others, but in the initial definition this is limited to getting the software working. But rule #10 goes beyond installation towards what the program actually does. In my opinion, this is just as important as installation, and deserves a more in-depth discussion.
  • Not completely sure what "robust" means in the title (but I think I know). If it means sustainable then external contributions are important. For external contributions it is very useful to clearly mark what is API and what are internals. In many codes this is not clear. Also a contribution guide is then very useful for external people.
• Having distinct yet concurrent software development phases (early investigation -> prototyping -> usable software) seems to be ignored by most of my colleagues. Since that’s the one of the key selling point of the article - transforming prototypes into usable software -, I wonder if it might be worth, beyond the 10 relevant and valuable rules you mention, to add a specific paragraph about these phases. If needed, I can propose one with a pull request.

Different disciplines

• In several places you mention and focus on "running" the software, e.g. "impossible to run outside", "difference between running and being robust", "tests to show to that it actually runs". This is good and understandable in the Python/R communities but in Fortran/C/C++-heavy physics/chemistry communities building the software can be a barrier for users and is of huge concern for portability (so is running but before the running comes the building). If you want to mention this and go this way, then you could recommend to use standard build tools (Make, Autotools, CMake; the last would be my fav) and not home-cooked configuration and build scripts.
• Web apps:
  • Actually deploying web applications, even research ones, will be a huge hurdle. All organizations I’ve worked with actually make it considerably harder. For example, databases are very often required. Deploying without root can be less than fun, although it’s doable. I used to run MySQL from the command line but it meant I actually really needed to know what I am doing.
  • OS portability is usually less of an issue for web applications. They’ll need to run for users on newer versions of IE/Edge, but it’s rarer for them to be deployed on Windows, at least where I’ve seen. However, using system() is frowned upon, as it’s escaping from the web context into an underlying system, and web applications should try to minimize that.
• Command-line options for a web application should ideally be replaced by environment variables. In effect, follow the 12 factor application guidelines (https://12factor.net/) which directly aims for maximum portability between environments. In fact, the 12 factor application guidelines are a wonderful start for any research web application.

0. Introduction

• "produces s/excellent/intended results"
• "Everyone with a few years of experience feels a tremor of fear when told to use a graduated student’s code to analyze their data." For some reason that's reading a little harsh to me. Perhaps graduated student could be changed to "another person's"?
  • Some citations to support this would be useful. How big a problem is this? Do we know?
• "Even the existing programs and initiatives rarely have the time to cover engineering topics in-depth, especially since the field is so broad and rapidly developing." Is there a modifier you could use in front of engineering to make it more specific (perhaps computational or bioinformatic)?
• My overall sense of this Introduction is that you are primarily preaching to the choir - only people who are already convinced that this is a problem will buy. While I personally appreciate the op-ed it's not going to be convincing to a broader audience. That may be OK, but I think it could be made about half as long and twice as convincing by focusing on integrating the citations you provide into a coherent, well-supported argument.
• "Not every coding effort requires such rigor": This does not follow from previous paragraph. In fact, I'm not sure what "such rigor" refers to?
• there are other reasons for duplication, including the drive to publish, no? It's not all about robustness. In fact, I think it would be bad for science if there was only one of each tool

1. READMEs

• "This is entirely reasonable as long as it is properly documented." It is?
• "Often, multiple libraries exist with the same or very similar names," They do?
• "If the dependency must remain closed, place it at an internal location on shared disk, remove all write permissions, and link to it from your README, although this method is discouraged because of the potential for sharing and risks accidental removal." I don't really understand what this means. This is for software that will be used internally only, or...?

2. Usage

• This is not "Unix" mkdir but GNU mkdir (as the help screen says). It runs in many places which are not unix, as well as few which are.

3. Version number

• You may want to mention in section 3 (meaningful version number) to always print the version of the code and all its dependencies in the output file(s) to enhance reproducibility. You do mention this in section 8 but before I arrived there I thought it got left out.
• The last paragraph of section three feels a bit out of place. Perhaps move it to the next section, about making past version accessible?
• Spelling: Bitbutcket, Github

4. Older versions

• "Programs’ authors should therefore ensure that every version" Maybe "A program's authors"?

5. Reuse software

• not "standard error output"
• "a call is made out to the other program, the results are incorporated into the" > "and the results"
• "Popular" [projects] doesn't always equate to "vetted"
• "Unfortunately, the interface between two software packages can be a source of considerable frustration. Support requests descend into debugging errors produced by the other project." ...hence semantic versioning, suggested above!
• "Reuse software (within reason)": "Build on other software" perhaps? I am also not sure this leads to robustness :)
• "To strike a balance between these two, developers should document all of the packages that theirs depends on, preferably in a machine-readable form." Typically this is something that a user could also use (via pip, apt-get, whatever) instead of having the list of software in README. Doesn't this get a bit redundant?Not spelling out the disadvantages of reusing software -- you introduce a new dependency. "left-pad" shows us the risks there. As an ontology developer, I get frustrated by silliness like "always reuse other ontologies." No, no, no. You allude to this, with your sort example, but integration is not the main concern.
• In section 5 you recommend to avoid depending on packages which are installed locally and may not be available to the user and not listed as dependency. My recommendation to avoid this is to always install dependencies through the file that you recommend one paragraph up (e.g. requirements.txt in Python). Then you "dogfood" on your own requirements.txt file and make sure that you don't forget a dependency. In other words I never install a dependency with "pip install package" but always through "pip install -r requirements.txt". This makes my local installations documented. Also I always use and recommend virtualenv (or the corresponding alternative in other languages). This enhances portability.

6. No root

• test the installation of their software on a fresh virtual machine
• Reduce assumptions on development language versions. I like Node.js, but I did have fun when I developed an app using Node.js version 4. Deploying on a CentOS VM in the end required me to build the whole of gcc (I kid you not) because a new version of gcc was needed by a few library dependencies. If I’d such to good old 0.12, I’d have been just fine with the system gcc. Keeping backwards compatibility might be a bit more annoying, but using bleeding edge versions will reduce deployment options.

7. No hard-coding

8. Parameters

• The key here is "useful" -- bioinformaticians have a terrible tendency to thing that "stuff a configuration parameter in" is a good way to avoid making a decision. I would try and expand that.
• In particular, I am missing the point "check that all input values are in a reasonable range, choose reasonable defaults where possible and no defaults at all otherwise". I have seen plenty of research software that assumes an in-depth familiarity of the user with the algorithms and their restrictions. Parameters out of the expected range silently lead to meaningless results. The recently publicized problems with fMRI analyses are an extreme case of this; there was nothing formally wrong with the code, but getting good results required using non-default parameters that most users didn't even know about.
• "Keep in mind that if a parameter can be adjusted, users will want to be able to turn off the feature entirely to have a baseline comparison." I don't entirely understand this. Parameter == feature?
• I'm not so sure the recommendation of command-line parameters over of configuration files is relevant as is it currently stated. IMHO, it depends on the software itself and cannot be generalized. For instance, in a research field I know quite well (molecular simulations software), there are so many parameters that need to be specified that relying on a command line would be really cumbersome. Moreover, simulations can run for days, and it would be easy to forget the exact set of parameters it was started with. For this latter point, your recommendation of dumping parameters together with the output of the program makes sense. However, in the early phase of (unstable) software development, this dump might not so reliable because features are likely to be added/removed quite often, and corresponding parameters hardcoded/externalized. This unstable phase is also the most critical with regard to the underlying scientific reality that is investigated, and is therefore its importance is beyond that of software engineering, especially while tracking the scientific performance for even very tiny yet significant improvements.

9. Test sets

• For codes that produce mainly numbers (many physics and quantum chemistry codes run for days and produce very few floating point results), the tests should test with tolerance for limited numerical precision. This does not have to be explicitly stated since it may be part of "output must be easy to interpret" but I mention this anyway since this is a nuisance with many codes in my community.
• complicated code is hard to test, simple code is easy to test -> tests will automatically guide me towards good (= simple) code structure (simple is again good for external contributions)

10. Identical results

• "Many bioinformatics applications rely on randomized algorithms" For example, ... (also, not limited to biology :)

11. Conclusion

• This seems out of line with the rest of the article, which was about robustness and reuse, which is, to my mind, somewhat separate from sustainability. I would ditch the sustainability argument and summarize according to the original intent of this paper...

Code documentation.

• I think lack of documentation is one of the biggest problems when downloading a bioinformatician's source code and trying to make it work. Student/employee wrote it a long time ago, either has moved on or doesn't remember the details, and doesn't have time/incentive to figure out problem. Documentation inside the code is so important: It's the difference between working 30 minutes on changing one line of code, to having to work 5-6 hours for that.
• "Add some documentation". Much scientific software does stuff that is complicated, and it's hard to work out how it achieves what it purports to achieve. Possibly, quite a lot of scientific software probably doesn't achieve what it purports to achive. I think this is more important than number 10 (which is, as you say, actually wrong in some cases). My experience is that publications do not really help, because it's often not obvious how the algorithm in the publication relates to the code. There's no replacement for code documentation!

Reference suggestions:

• A couple of relevant refs perhaps?: https://t.co/fNs1jjyBrg https://t.co/7Gf3aPiGeI
• Perhaps point to conda or similar alternatives when talking about dependencies
• documentation best practices https://t.co/vsoHezeFCU
• https://sysmod.wordpress.com/2016/08/28/excel-gene-mutation-and-curation/

Other suggestions:

• Did you consider some code review by another coder as a robustness increaser? Worth a mention, or not 'simple'?
• an appendix checklist might be useful. Ideal would be to get something that journals can point to as a standard, kind of like CONSORT for software
• To minimize code complexity I would recommend the Unix philosophy to create programs which aim at doing one thing and one thing only (and well). I have written and seen many programs that tried to do many things, "everything", and eventually crushed under their own complexity. It is better to keen functionality minimal and to leverage small units by composition into new ideas. Again following Unix philosophy
• Authentication. Is a huge problem. Every organization is different, and it’s very very easy to regress to a lowest common denominator of a database of usernames and passwords (even if done right with bcrypt). It’s definitely a good plan to use a buffer layer to allow multiple authentication backends, e.g., Shiro for Java, Passport for Node.js, etc. Even if you do use LDAP, many organizations actually use Active Directory so don’t like to issue service accounts, especially for random application
• Most research organizations never want to issue email addresses to systems rather than people, unless they are deployed under the central IT units, and this is a huge barrier for research-developed applications. You’ll need to avoid notifying through email, or find another way to achieve it

Allowing/promoting community contributions

• Decide who will maintain the software: the PI, the student, an RA, or no one? Anything is fine if clearly stated.
• in-line documentation: "I know that I might not be able to support this program forever, and I would be really happy if a young enthusiastic student could make use of my code in the future without hating me."
• As long as your program is not used commercially, and is used mostly for research, there's no reason not to allow contributions. I would like to see as many people as possible giving other people access to their code - allowing others to easily modify it and use it for their purpose, and then pushing it back or sharing it with others.
• Providing support to your software: Replying to emails and questions about your software is a pretty basic requirement, definitely when it's still new and doesn't have dozens of Stackoverflow or Biostars posts discussing it.
• Tests simplify external contributions (thus help make the code live longer), as an external contributor I am scared to modify a code without tests, as a maintainer I am scared to accept changes if the changes are not tested
• make it open source if possible. It's implicit in a lot of what you say but people will always want to be able to adjust your software in ways you didn't intend.
• provide a link to a user forum or issue tracker. This is tough to do in some cases b/c you might not want to provide support, but specifying a place that users can go to talk about your software might be useful.

[hlapp]

A couple of comments that I didn't find covered above (though may have overlooked):

• Consider citing HMMER as an example for superb scientific software engineering. It's very widely used, and Sean Eddy regularly makes a point that this is important (including in his 2013 keynote address at BOSC), and can even pay off for one's own career. I'd even contact him for additional comments he might have.
• One of the most frequent issues with scientific software that isn't robust is that it crashes (or worse, produces erroneous output without any error) for user input it didn't expect. All too often this happens half way through a 72hr computational pipeline, and tracking this stuff down must cumulatively be wasting the equivalent of multiple postdocs FTEs. Input files too large, not in the right format, too many or too few columns, too few samples, a parameter value expected to convert errorless to a number, a numeric parameter supposed to be an integer but accidentally passed as float, and the list goes on an on. How many programs check the return value of file open, read, write, and close commands every single time? The critical importance of testing every single bit of input specified by a user as to whether it satisfies expectations, and very clearly stating those expectations, ought to be covered in this article, no?
• Another common issue is memory leaks. This isn't restricted to programming languages that require one to expressly allocate and free memory such as C. It's equally possible in Java or Python to accidentally keep a pointer in scope to some object so it can't be garbage collected. Who regularly runs the tools for a language (lint etc) that pinpoint these issues? More generally, always enabling all warnings that a compiler or interpreter can issue, and doing the extra work needed to silence them. I don't know how many scientific software programs I've compiled that generate tons of scary scary warnings. Why should we trust these programs to be robust? Why should I trust a programmer to bother catching my user input errors if they can't be bothered to silence the compiler warnings they could have ready access to themselves. I don't think this is covered yet?
• Another frequently encountered issue is poor behavior resulting from ignoring rules that have been made to allow citizens to live in close proximity. One of the most common infractions is to ignore Unix/Linux conventions, such as environment variables or programming APIs. For example, most programs used in science create temporary files, some create 10x of thousands. Many times, they don't honor TMPDIR, and/or don't use the language's built-in API for creating temporary files. Or they do but assume every user already knows how to control where temp files are put. As a result, shared compute machines are frequently rendered unusable for everyone because someone ran some program that filled up /tmp (which, especially on an HPC cluster, is often quite limited in size). Another one is not clearly separating output between stdout and stderr. Or not using exit codes to indicate error status. And of course parsing command line arguments with custom code rather than using the language's built-in API (though I suppose that would go under Reuse).
• A capability that can be non-trivial to implement (and thus missing from almost all scientific software) but can make software hugely more robust when running for several days or even weeks or months is checkpointing. How many thousand CPU years are lost to cyberspace every year due to lack of checkpointing? Having to rerun from the start an analysis that runs for 4 weeks only because the final computational step was interrupted by something unexpected (which can be as trivial as underestimating the memory one should have allocated from the HPC cluster scheduler) is a real tangible slowdown of scientific progress.
• Versioning and Providing previous versions could well be consolidated into a single point. It feels like otherwise that horse is kind of beaten to death, and while versioning is surely important, it's not like un-versioned software cannot possibly be robust.

[michaelhoffman]

I don’t think I belong on this thread, though it is very interesting.

-MH

On Sep 29, 2016, at 8:36 AM, Hilmar Lapp notifications@github.com<mailto:notifications@github.com> wrote:

A couple of comments that I didn't find covered above (though may have overlooked):

• Consider citing HMMERhttp://hmmer.org/ as an example for superb scientific software engineering. It's very widely used, and Sean Eddy regularly makes a point that this is important (including in his 2013 keynote address at BOSChttp://www.open-bio.org/bosc2013/day2/BOSC2013Keynote-_Sean_Eddy.pdf), and can even pay off for one's own career. I'd even contact him for additional comments he might have.
• One of the most frequent issues with scientific software that isn't robust is that it crashes (or worse, produces erroneous output without any error) for user input it didn't expect. All too often this happens half way through a 72hr computational pipeline, and tracking this stuff down must cumulatively be wasting the equivalent of multiple postdocs FTEs. Input files too large, not in the right format, too many or too few columns, too few samples, a parameter value expected to convert errorless to a number, a numeric parameter supposed to be an integer but accidentally passed as float, and the list goes on an on. How many programs check the return value of file open, read, write, and close commands every single time? The critical importance of testing every single bit of input specified by a user as to whether it satisfies expectations, and very clearly stating those expectations, ought to be covered in this article, no?
• Another common issue is memory leaks. This isn't restricted to programming languages that require one to expressly allocate and free memory such as C. It's equally possible in Java or Python to accidentally keep a pointer in scope to some object so it can't be garbage collected. Who regularly runs the tools for a language (lint etc) that pinpoint these issues? More generally, always enabling all warnings that a compiler or interpreter can issue, and doing the extra work needed to silence them. I don't know how many scientific software programs I've compiled that generate tons of scary scary warnings. Why should we trust these programs to be robust? Why should I trust a programmer to bother catching my user input errors if they can't be bothered to silence the compiler warnings they could have ready access to themselves. I don't think this is covered yet?
• Another frequently encountered issue is poor behavior resulting from ignoring rules that have been made to allow citizens to live in close proximity. One of the most common infractions is to ignore Unix/Linux conventions, such as environment variables or programming APIs. For example, most programs used in science create temporary files, some create 10x of thousands. Many times, they don't honor TMPDIR, and/or don't use the language's built-in API for creating temporary files. Or they do but assume every user already knows how to control where temp files are put. As a result, shared compute machines are frequently rendered unusable for everyone because someone ran some program that filled up /tmp (which, especially on an HPC cluster, is often quite limited in size). Another one is not clearly separating output between stdout and stderr. Or not using exit codes to indicate error status. And of course parsing command line arguments with custom code rather than using the language's built-in API (though I suppose that would go under Reuse).
• A capability that can be non-trivial to implement (and thus missing from almost all scientific software) but can make software hugely more robust when running for several days or even weeks or months is checkpointing. How many thousand CPU years are lost to cyberspace every year due to lack of checkpointing? Having to rerun from the start an analysis that runs for 4 weeks only because the final computational step was interrupted by something unexpected (which can be as trivial as underestimating the memory one should have allocated from the HPC cluster scheduler) is a real tangible slowdown of scientific progress.
• Versioning and Providing previous versions could well be consolidated into a single point. It feels like otherwise that horse is kind of beaten to death, and while versioning is surely important, it's not like un-versioned software cannot possibly be robust.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/oicr-gsi/robust-paper/issues/20#issuecomment-250502674, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJNO9N4xMY5BIFYHRMijqBc5HuzMmoVxks5qu9rsgaJpZM4J6wTH.

[morgantaschuk]

@michaelhoffman Sorry about that. It was a Twitter handle/GitHub name overlap.

[michaelhoffman]

No problem! Just thought you'd like to include the intended "Michael Hoffman"

-MH

This was sent from my phone. I blame autocorrect.

On Sep 29, 2016, at 10:21 AM, Morgan Taschuk notifications@github.com<mailto:notifications@github.com> wrote:

@michaelhoffmanhttps://github.com/michaelhoffman Sorry about that. It was a Twitter handle/GitHub name overlap.

You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/oicr-gsi/robust-paper/issues/20#issuecomment-250533193, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJNO9IibZ64GaOaZVoourkhzwHMRNAv-ks5qu_NxgaJpZM4J6wTH.

[morgantaschuk]

Closing this issue as a pre-submission thread. Please open a new issue if you'd like to address these comments further.

[hlapp]

The submitted manuscript seems to lack acknowledgments, including to this thread. Intention or oversight? And most of the above feedback seems to have gone neither into a rule nor into "What was left out". Intention or oversight?

[morgantaschuk]

Good point about the acknowledgements. We should definitely add a section. That was an oversight.

We definitely took all of the feedback into account. You can see the diffs and our discussion in pull requests here: https://github.com/oicr-gsi/robust-paper/pull/21 and https://github.com/oicr-gsi/robust-paper/pull/22.

I looked at your feedback in particular. We agree with each of your points about memory leaks and checkpointing, but we decided those topics are beyond the scope of "simple rules". These rules are intended to be made on software in order to make code reuse simpler and more straightforward, not necessarily to make it the best software out there. Randomly, the term "robust" in this paper is probably a misnomer on our part. We took your suggestion about combining versioning and old versions sections, and we also incorporated your feedback regarding checking input values into the second part of rule 9.

[hlapp]

Randomly, the term "robust" in this paper is probably a misnomer on our part.

Perhaps. As a reviewer, one of my major comments would be that the set of rules conflates robustness to run ("use") and robustness to "reuse". E.g., software can run quite robustly even if it has no tests at all that a user would be able to run. For example, the quoted mkdir doesn't come with tests that execute when it's installed (most installations are from binary, and this is no different for scientific software).

Personally, I feel these are better separated than commingled, even though I believe quite strongly that collaboratively developed software tends to be much more robust than software developed by a single individual. But the manuscript isn't putting up develop collaboratively as a rule (which is fair enough), and arguably individually developed software need not lack robustness.

we also incorporated your feedback regarding checking input values into the second part of rule 9.

I see. I didn't recognize that when I read that section. (And checking user input is not specifically mentioned as to what is most useful to test to improve robustness -- certain tests contribute to robustness much more than others.)