Review Matrix

[nuest]

Current version: https://github.com/nuest/how-to-read-a-research-compendium/blob/master/research-compendium-review-matrix.pdf

---

Based on http://blizzard.cs.uwaterloo.ca/keshav/home/Papers/data/07/paper_review_matrix.pdf

• 1st PASS
  • code license - can I use it (checkbox)?
  • is the software useful if it works?
• 2nd PASS
  • code repository - is it active? what is the bus factor?
  • published on CRAN/PyPi or "source only"
• 3rd PASS
  • ...

[benmarwick]

We assume that the reader has already gone through the three passes of Keshav's paper, and then;

• 1st PASS Relevance, accessibility and usefulness indicators, by a quick glance -- language and environment: is it a language that I can use (currently I'm not having fun with a compendium that uses OCaml :sob:)? is the compute infrastructure (OS, cores, memory, etc.) available to me? -- dependencies: few or many? can I install them? -- code license - can I use it (checkbox)? is the software useful if it works?
• 2nd PASS Quality indicators, by a look into the materials -- are there code comments? readme? responsiveness to issues? CI, stars/forks -- ropensci onboarding criteria: is there documentation? tests? NEWS? authorship clear? -- on CRAN? has a supporting publication?
• 3rd PASS Application, by a time consuming very close study of the materials -- work through examples included in compendium -- extend application of compendium to new problem/data

[nuest]

@benmarwick Thanks! I've made another iteration, see "Content" below.

I would like to keep the one page format, and also integrate the seperate steps. Can you elaborate why you would expect a reader to complete Keshav's passes and then look at the "compendium" aspects?

I also took over many of your ideas (accessibility, dependencies, code comments, responsiveness, forks/stars) over to the manuscript text (see https://github.com/nuest/how-to-read-a-research-compendium/commit/cecff5a2e774de2b93fb0b7decf6b447901602d8). Given the number of aspects the two new Cs, I like the idea of having keywords in "light gray" in the matrix to help users, so I've added those below.

Content

• 1st PASS
  • Construction [keywords: building blocks, programming language, algorithms, accessibility, usefulness]
  • Complexity [keywords: scale, OS, cores, memory, execution time, data size, software dependencies]
• 2nd PASS
  • add subcategory to "How Does it Do it?": Tool quality [are there code comments? is there a README? is there (structured) documentation? tests? semantic versioning and releases? authorship clear? published in suitable software repository (CRAN, PyPi, npm)?]
  • add subcategory to "What did I find?": Reproducibility [results of following the instructions and reproduction, differences?]
• 3rd PASS
  • new category (next to Strenghts, Weaknesses): Application [very close study of the materials, examplesl, extend application to new problem/data]

Design

Do you have any ideas how to best design the matrix? Inkscape + SVG come to mind, or a spreadsheet. Or is there now way around LaTeX? I don't think Markdown will work, and I would really like to keep the matrix to one printed page.

A high resolution version of the original matrix can be found at http://www.ryanberg.info/2014/11/15/how-to-read-a-scholarly-paper-literature-review-matrix/, but also only a PDF.

[nuest]

Although not the most open tool, I went to a Google Spreadsheet to design the extended review matrix. It should at least allow easy collaboration.

Editable link: https://docs.google.com/spreadsheets/d/18VrkcvTWDyvsldRKfgw2jnhfd5C6xbMwGZNE4ILjXlw/edit?usp=sharing

Here is a screenshot of the current version, exported to PDF (A4 paper size in portrait mode, using custom margins: 0.25'', scale "Fit to height" and then manually tinkered with the width of the content column E)