Reviews on submission 126

[fabianodalp]

The assigned reviewers are going to post their reviews on this submission within this issue. The same thread will be used also to support the interaction with the authors.

Reviewers, please check STATUS.md to determine which badges the artifact is applying for. A description of the badges can be found here: https://re20.org/index.php/artifacts/. You will also receive an e-mail with further instructions shortly.

[jin-guo]

@fabianodalp, would you please confirm if the survey design, instead of the raw data is within the scope of the artifact track?

@jtiz003 would you please include the paper submission so that I can have a better understanding of the context of the survey?

[fabianodalp]

The survey design is within the scope, yes. We are taking a pretty liberal perspective in this edition of the track.

[jin-guo]

@fabianodalp Thanks for the confirmation and sharing the paper submission over email. I will evaluate the survey design then.

[jin-guo]

The authors are applying for the badges of Reusable and Available.

Available

The survey design is available on both Zenodo and the link is included in the repo.

Reusable

A lot of information according to the standard of Reusable badge is missing:

The artifacts associated with the research are found to be documented, consistent, complete, exercisable, and include appropriate evidence of verification and in addition, they are very carefully documented and well-structured to the extent that reuse and repurposing is facilitated. In particular, norms and standards of the research community for artifacts of this type are strictly adhered to.

The authors should consider adding the following information in the ReadMe:

1. The DOI of the artifact published in Zenodo;
2. The objectives of the study and the link to the paper (once it's published);
3. The overview of the survey design (Section III.A in the paper);
4. Information related to ethical approval on the study (Section III.B in the paper);
5. A brief description of the protocols of recruitment and data collection and analysis (Section III.C-E in the paper).

While most of the information is in the paper already, I suggest the authors include a concise version here for the sake of exercisable and completeness of the artifact itself.

[fabianodalp]

Thank you @jin-guo for your review. @jtiz003 , can you please fix these things in the ReadMe so to achieve the "reusable" badge? I will grant already the "available" badge.

[emitza]

I agree with Jin that the submission complies with the requirements for the available badge.

I also agree with the previous review that there is missing information in the readme file so that the artifact is more independent of the paper itself. I think the points raised by Jin would allow the paper to obtain the badge if they are added to the readme file.

[jtiz003]

@jin-guo @fabianodalp @Emitza Thanks for your review. I will begin addressing the missing details

[jtiz003]

@jin-guo @fabianodalp @Emitza I have updated the readme based on your suggestions. Please let me know if anything else is required. Thanks, James

[jin-guo]

@jtiz003 Thanks for the update of the README file. It looks good to me. The only thing I suggest is to add a description in the last section Recruitment, Data Collection, and Analysis to make it clear that they are the procedures used in your paper (not necessarily required for reuse of the survey).

[jtiz003]

@jin-guo Thanks. I have added a sentence in that section to explain this point

[fabianodalp]

Thank you both to @jtiz003 and @jin-guo for the changes and the guidance on the reusable badge, respectively. I am awarding the badge now, unless @jin-guo or @Emitza disagree.

James, make sure the DOI of your artifact is referenced in the camera-ready version!

[emitza]

I am fine with awarding the badge!