(for transparency and future understanding, this message is added based on the discussion of our manifesto during CHIIR 2021, just so the contributions of others are visible and can be used as starting points to update the manifesto where needed.)
Q&A: is the notion of reuse in the manifesto focused on reproducibility?
Definition of reuse
Mark Hall: we use the broadest possible definition of re-use, which is not just replication, but also picking bits out of previous studies.
Reuse for reproducibility
Udo Krusewitz and Arthur Barbosa Câmara: There are some community efforts, e.g. reproducibility tracks. Several IR-related conferences have reproducibility tracks (ECIR, SIGIR, RecSys).
Marijn Koolen: Repeatability, replicability and reproducibility (using Christoph Schöch's definitions in [1]) is difficult for IIR for many reasons, although reproducibility in the sense of establishing robustness of findings should be attainable (but perhaps only within a limited timeframe while the user-base doesn't change too much).
Definition of reproducibility
Andrew Macfarlane: the ACM definitions and terminology in this area are confusing and conflicting. Its a question of what you mean by reproduciblity - the end goal (results) or methodology?
Vivien Petras: we try to get at this distinction (results / methodology) a bit by distinguishing between research design (methodology) / data (results) / infrastructure (System, processing). They need different efforts for re-use.
Max L. Wilson: HCI has largely rejected reproducibility as a relevant issue, so CHIIR should be in an interesting struggle point between HCI and IR.
Mark Sanderson: Steve Whittaker [2] argued for reuse ideas for CHI about 20 years ago. He pointed to the success of TREC and tried to argue that the CHI community should work on common reference tasks (which then suggests reuse)
Mark Hall: Unfortunately not too many of the things we put forward have not been put forward before.
Max L. Wilson: Interactive TREC died off right, for all the reasons that people realised that replication in IIR (and HCI) isn’t possible. Because society, technology, and people change (as Dan Russell said at RepliCHI)
Nils Pharo: Inex' interactive track survived quite many years and reused study designs. Not to replicate, but making comparison over years possible.
Reproducibilty vs. Documentation
Dana McKay and Stephann Makri: It is a challenge with qualitative work in some ways, but it doesn't mean we shouldn't be explicit. That may mean going beyond being reflective to opening our approaches up to far more scrutiny. We get back to the difficulties of making the implicit (cognitive approach to qual coding beyond any structure imposed by QDA approach) as explicit as possible. More is possible in this regard. But only so much is possible. It's hard to open ourselves up to such scrutiny, because it can invite not only positive, well-structured critique, but (sometimes unfair) criticism/'this is not good research'.
David Elsweiler: It is not always easy to judge if you have provided enough detail to replicate a study. In my experience it only becomes clear when you actually try to do it.
Maria Gäde: I think studies should be as comprehensible as possible even if they are not entirely repeatable, that’s what we argue and would increase quality (from my point of view)
[1] Christof Schöch. 2017. Wiederholende Forschung in den digitalen Geisteswis- senschaften. In DHd2017: Digital Nachhaltigkeit (DHd2017), Bern, Switzerland, 13-18 February 2017. Zenodo, Genève, Swizterland, 207–212. https://doi.org/10.5281/zenodo.277113
[2] Whittaker, S., Terveen, L., & Nardi, B. A. (2000). Let's stop pushing the envelope and start addressing it: a reference task agenda for HCI. Human–Computer Interaction, 15(2-3), 75-106.
(for transparency and future understanding, this message is added based on the discussion of our manifesto during CHIIR 2021, just so the contributions of others are visible and can be used as starting points to update the manifesto where needed.)
Q&A: is the notion of reuse in the manifesto focused on reproducibility?
Definition of reuse
Mark Hall: we use the broadest possible definition of re-use, which is not just replication, but also picking bits out of previous studies.
Reuse for reproducibility
Udo Krusewitz and Arthur Barbosa Câmara: There are some community efforts, e.g. reproducibility tracks. Several IR-related conferences have reproducibility tracks (ECIR, SIGIR, RecSys).
Marijn Koolen: Repeatability, replicability and reproducibility (using Christoph Schöch's definitions in [1]) is difficult for IIR for many reasons, although reproducibility in the sense of establishing robustness of findings should be attainable (but perhaps only within a limited timeframe while the user-base doesn't change too much).
Definition of reproducibility
Andrew Macfarlane: the ACM definitions and terminology in this area are confusing and conflicting. Its a question of what you mean by reproduciblity - the end goal (results) or methodology?
Vivien Petras: we try to get at this distinction (results / methodology) a bit by distinguishing between research design (methodology) / data (results) / infrastructure (System, processing). They need different efforts for re-use.
Max L. Wilson: HCI has largely rejected reproducibility as a relevant issue, so CHIIR should be in an interesting struggle point between HCI and IR.
Mark Sanderson: Steve Whittaker [2] argued for reuse ideas for CHI about 20 years ago. He pointed to the success of TREC and tried to argue that the CHI community should work on common reference tasks (which then suggests reuse)
Mark Hall: Unfortunately not too many of the things we put forward have not been put forward before.
Max L. Wilson: Interactive TREC died off right, for all the reasons that people realised that replication in IIR (and HCI) isn’t possible. Because society, technology, and people change (as Dan Russell said at RepliCHI)
Nils Pharo: Inex' interactive track survived quite many years and reused study designs. Not to replicate, but making comparison over years possible.
Reproducibilty vs. Documentation
Dana McKay and Stephann Makri: It is a challenge with qualitative work in some ways, but it doesn't mean we shouldn't be explicit. That may mean going beyond being reflective to opening our approaches up to far more scrutiny. We get back to the difficulties of making the implicit (cognitive approach to qual coding beyond any structure imposed by QDA approach) as explicit as possible. More is possible in this regard. But only so much is possible. It's hard to open ourselves up to such scrutiny, because it can invite not only positive, well-structured critique, but (sometimes unfair) criticism/'this is not good research'.
David Elsweiler: It is not always easy to judge if you have provided enough detail to replicate a study. In my experience it only becomes clear when you actually try to do it.
Maria Gäde: I think studies should be as comprehensible as possible even if they are not entirely repeatable, that’s what we argue and would increase quality (from my point of view)
[1] Christof Schöch. 2017. Wiederholende Forschung in den digitalen Geisteswis- senschaften. In DHd2017: Digital Nachhaltigkeit (DHd2017), Bern, Switzerland, 13-18 February 2017. Zenodo, Genève, Swizterland, 207–212. https://doi.org/10.5281/zenodo.277113
[2] Whittaker, S., Terveen, L., & Nardi, B. A. (2000). Let's stop pushing the envelope and start addressing it: a reference task agenda for HCI. Human–Computer Interaction, 15(2-3), 75-106.