Reviews on submission 68

[fabianodalp]

The assigned reviewers are going to post their reviews on this submission within this issue. The same thread will be used also to support the interaction with the authors.

Reviewers, please check STATUS.md to determine which badges the artifact is applying for. A description of the badges can be found here: https://re20.org/index.php/artifacts/. You will also receive an e-mail with further instructions shortly.

[jenhork]

Thanks to the paper authors for providing the material and the DOI link. There's nothing in the individual files that is problematic, in my view.

My problem going through the material is trying to understand how it would be used, in what order, and in what way.

Having access to the paper, I'm trying to map the material to the Pedagogical Design in Fig. 1, which has Task 1 Elicitation, Task 2 Specification, and Task 3 Inspection. Item 1, the lecture, would be learning resources for Task 3? Item 2, the tutorial, is tutorial for task 3 Item 3, is a concrete SRS to use in this tutorial Item 4, are use cases for this example, but how are they used? Also in the tutorial? In the pedagogical design the use case are and output of Task 1 and an input to Task 2? Item 5 is what the students have to do for Task 3 Item 6, is an example problem description, but it's not an input to Task 3, but to Task 1, the Requirements elicitation? Item 7 is how to evaluate Task 3 Item 8 is how to evaluate Task 2?

These are my guesses, but it's a bit confusing that we are provided mostly with material for the inspections, but also a bit of material for the elicitation and specification.

I think it is not clear how to use the material without also having the CR paper copy, and even then there is some mapping work required. Perhaps the paper will be in the final repo, but even so the mapping to the paper can be clarified.

Also, more minor, but why just the lecture outline in item 1 and not the PDFs of the full lectures?

And there is only one TutorialActivity file for both 2 hour tutorial sessions?

Finally, I think it would help to clarify that all this material is to make the training reusable, and whether or not this is also to support replicating the qualitative study. In other words, there is nothing about the analysis of the data in the course, no coding hierarchies, etc. Given "To answer RQ4 and RQ5, we perform a thematic analysis of the reflection reports." in a repo for a qualitative study, I would expect the results of the thematic analysis and the raw data, as much as possible. In other words, in the paper, there are two contributions: 1) how to teach inspections, and 2) the authors coding, analysis and reporting of results for some course instances. The repo is only about 1, and not 2. I'm guessing for 2: the student data cannot be provided, and the list of codes is in the paper, but good to explain this upfront. Perhaps the authors can upload their coding hierarchies without raw data, e.g., themes for coding artifacts X, Y, and Z.

Another way to think about this, is whether or not all the material for point 1, is enough for others to redo the study to get new data towards point 2. I think with the material for 1, plus the paper and the themes in the paper, this is likely the case.

Best,

Jennifer

[microlina]

The authors are applying for an Available badge. The material have a DOI an all the files are publicly accessible in Zenodo, which is an adequate choice.

The description presented in the Zenodo page clearly describes the provided materials. However, a better explanation on the usage (and order) of the materials would be helpful, even if a version of the paper is added to the repository.

It would be interesting to have access to the entire lecture. Why have the authors decided to only present the 2-hours lecture outline?

In section 3 of the paper, the authors state that "They also attended two (one-hour session) tutorials on requirements inspection where they were presented with examples of some types of defects followed by carrying out an exercise of finding defects in a very small SRS using checklists". The examples presented to the students, prior to the exercise are not available, only the exercises. Regarding the exercises, was there a gold standard for them? Such as a resolution and list of defects the students should be able to encounter. It would be helpful to have both the problem and the solution.

Furthermore, it would also be interesting to have access to the resources listed in the InspectionAssignment_Decription.pdf document, which were available to the students.

Finally, and now related to the performed study and analysis, I was excepting to have access, if not to the raw data, at least to the entire results of the statistical analysis.

Best regards, Catarina

[fabianodalp]

Hello all, thank you for the discussions so far. I think the badge "available" can be granted here. Regarding the "reusable" badge, it is not required to have full alignment between the paper and the artifact, although that would help. In particular, my suggestion would be that, if some clarifications can be made, those would be useful to include here. However, for missing data, e.g., the raw data or the results of the statistical analysis, as @microlina said, it is up to the authors to decide what information they want and can make open: they don't have to include ALL data used in the paper.

[paolaspoletini]

Thank you all for your comments! We are working on both providing the needed clarifications and improving the package and we will get back to you by the end of the week if this is fine with you.

[jenhork]

Sure, just let us know when things are updated.

[paolaspoletini]

Sorry it took a little longer than expected to make the modifications and thanks for the useful comments and the additional requests that helped us to improve the package. Below are the response to the specific comments and how we addressed them.

Let us know if there are further doubts and need of providing additional material. Thanks!

@jenhork The files in the package are carefully described in the readme file. In the same file, it is explained why each file is provided. In order to make more clear why we provide the rubric for evaluating the SRSs, we added the description of the context in which the approach was originally used at the beginning of the readme file (instead of using the beginning of footnote 1). Furthermore, we described the file SRSAssignment_Rubric.pdf separately from the list of the inspection-related files. We also explicitly mentioned what could be changed to use the approach in a different context (i.e., as a stand-alone activity).

@microlina In the readme file, we have added an explicit description of the order in which the steps of the approach are carried out.

We believe that the specific examples presented to the students are not particularly relevant. In the case of the described course, they were from Kotonya and Somerville's book. We have explicitly mentioned this in the readme file. Also, there was no gold standard for the exercises used in the tutorial. The students only received some examples of defects from Kotonya and Soomerville's book.

Finally, unfortunately, the specific resources listed in the InspectionAssignment_Decription.pdf document cannot be currently shared as they are not owned by us, but instructors who want to reuse the material can use simple log issue lists and suggests a subset of the many checklists available online (such as https://www.cs.purdue.edu/homes/bxd/307/Software%20Requirements%20Specification.pdf or http://www.cs.toronto.edu/~sme/CSC340F/2005/assignments/inspections/JPL_reqts_clist.pdf)

@jenhork and @microlina Thank you both for pointing out different scenarios on how to integrate our package. We are actually applying for the “available” and “reusable” badges, and not “replicated” or “reproduced”. The idea is to let other instructors reuse the teaching material that was used in our experiment. This is why we do not share the raw data. We have now specified in the README file that the raw data cannot be shared, and the resulting codes are reported in the original paper. The preprint of the paper is also included in the package so that the reader can refer to the paper. If you think it could add value to the package we can add the file with the data used for the correlation analysis, even if it is not part of the teaching approach.

As for the slides used in the lecture, unfortunately, they cannot be shared (due to UTS’s copyright policy).

As for the tutorial, the same case has been used for both the tutorials. For the first tutorial the SRS is used, for the second one, students used the use case specification.

[fabianodalp]

Thank you @paolaspoletini for your response. Small note: replicated and reproduced actually mean that your paper replicates or reproduces previous results; so, your point regarding available vs. replicated/reproduced is note relevant for the discussion.

@jenhork @microlina please take a look at the changes and see if the reusable badge can be awarded.

[jenhork]

@paolaspoletini Thanks to you and your co-authors for the changes.

I think the usage of the artefacts, what is there and what is not and why, is more clear now.

I think it's reasonable to provide available and reusable as badges, as the educational material to replicate the experiment is now provided.

Best,

Jennifer

[fabianodalp]

@jenhork thank you for checking again. I will assign both badges. @microlina if you feel you disagree, just respond ASAP :)