(Software) review resources

[betatim]

Collection of resources on the topic of "review". Mostly software focussed but some of the points also apply to any kind of scientific review process.

• 11 Best Practices for Peer Code Review
• Code reviews, the lab meeting
• Have a checklist so it is clear to reviewer and reviewee what will happen and define when they are done
• "A Foolish Consistency is the Hobgoblin of Little Minds", don't get distracted by discussing "{code,writing} style", discuss what the {code,writing} does. (Use a robot to determine if it passes style or not)
• When you give people rules/guidelines they will start applying them to the letter ("No lines longer than 80characters!!!") which is useless and means they miss the gorilla. This is a hard problem to solve. If you become unsure/lack sufficient expertise on the topic you are reviewing it is easy to simply "apply the rules" instead of review the substance. The alternative would be to admit (in public) that you lack the expertise, which is extremely hard to do. It also takes a lot of practice and conscious effort not to get sucked into bible slinger mode.

Thoughts on computing in general: http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1001745

[betatim]

I can't find the reference for this but there are several studies that show that review is most efficient and effective when done in small steps. This supports the idea of starting the review as early as possible (or earlier).

A 30 line diff is much easier (more than a factor 10 easier) to understand and hold in your head than a 300 line diff.

[particleist]

@betatim : yes, that must be true. It is also true of the physics review, of course, though you do have to check for consistency at the end of the process in both cases, so that you don't make divergent choices in different review steps. The maven would presumably play a large role in making sure this didn't happen.

On Fri, Aug 21, 2015 at 9:09 AM, Tim Head notifications@github.com wrote:

I can't find the reference for this but there are several studies that show that review is most efficient and effective when done in small steps. This supports the idea of starting the review as early as possible (or earlier).

A 30 line diff is much easier (more than a factor 10 easier) to understand and hold in your head than a 300 line diff.

— Reply to this email directly or view it on GitHub https://github.com/seneubert/maven-review/issues/2#issuecomment-133312195 .

[apuignav]

I think the minimal code review cannot go much further than checking that provided code runs and spits out the results claimed by the analysts, maybe checking for obvious and gross mistakes. Detailed code review is quite time consuming (even for small chunks) and I am not sure it is realistic to assume is going to be up for that.

[betatim]

Mentally we should get away from the idea that this reviewing is somehow a competition/adversarial. I trust that (because I could check) you will not mislead me about the output/result of your analysis. A reviewer is not a policeman, they are there to help/guide you/provide a fresh set of eyes.

[seneubert]

I wholeheartedly agree with you Tim. Is there anything we can do to help this change of mental models?

[particleist]

I think this mentality arises at least in part from the fact that the current review model hides a lot of things (e.g. the code, but not only) from the reviewers, and the pressure on both analysts and reviewers to converge as quickly as possible. In this setup it is inevitable that the review becomes adversarial : for example, the moment an analyst tries to explain why they should not answer a question, instead of just answering it, because they are pressed for time or whatever. I suspect that making the code available would actually help with this because the reviewer would never feel that things are being hidden from them.

There is also the issue of best-practice. Often as a reviewer you see that the analysts are using a suboptimal method compared to other things done in the WG. Here even if you try to be helpful, it often appears to the analysts that you are actually lecturing them or making them look bad, and it becomes adversarial from that perspective. This is partly an issue of the culture of the WG of course.

V

On Fri, Sep 25, 2015 at 11:55 AM, Sebastian Neubert < notifications@github.com> wrote:

I wholeheartedly agree with you Tim. Is there anything we can do to help this change of mental models?

— Reply to this email directly or view it on GitHub https://github.com/seneubert/maven-review/issues/2#issuecomment-143175353 .