Review of submission ReddyMekala_RE_AE

[LloydThinks]

Author: @crowdre Reviewer 1: @ape810 Reviewer 2: @re-ae

[ape810]

Hello @crowdre this artefact seems to be very interesting. I'm trying to run the code available at the figshare link, but I've some difficulties... could you please give a short guide on steps to be followed? Thanks

[LloydThinks]

Dear @ape810 and @re-ae, it appears as though the authors have only applied for the Available badge. Please evaluate the merits of the submission based on the criteria for the Available badge only. In such a case, the authors do not need to provide a short guide on steps to be followed. The authors may, however, choose to extend their submission to Reusable with such documents, but due to time reasons this is likely not going to be possible.

Lloyd & Nash

[re-ae]

@crowdre

An Available badge is requested for this artifact.

The artifact is hosted online on Figshare. I am able to download the artifact that includes Python scripts, data files, and model files. The artifact's visibility is currently set to private because of which there is no DOI associated. Authors are aware of the issue and request an exception. They suggest that they will make the artifact public with the camera ready version of the paper and include a DOI.

There is no readme with the artifact that makes it difficult to understand which file or folder is for what purpose. I suggest the authors to add a minimal readme file that provides at least a directory listing to help understand what all is available.

I recommend the Available badge to the artifact, conditional to authors' replacing the private Figshare URL with a DOI that redirects to immutable public URL and the readme.

[ape810]

@crowdre I'd also recommend the Available badge for this artefact, provided that the authors use a public Figshare URL and a DOI, to be referred in the paper as well, in accordance with the description of "Artefact Available" in the RE'21 Artifact Track guidelines. I was able to:

• access the Figshare repository and download the artefact (python code, data, models and a .ipynb file)
• I could open the .ipynb file with jupyter-lab and see how P1 and P2 tasks can be performed with ML (Naive Bayes, TF-IDF) and obtain results presented in Table I in the paper. But I'd recommend to include a readme with instructions on how to run the code and of what can be reproduced running it.

[crowdre]

Dear reviewers @re-ae and @ape810 , we thank you kindly for your considerations, your positive judgment of our artifact, and your recommendation for the Available badge!

We also appreciate your understanding that the artifact still used a private link. As promised, we have now published our artifact on Figshare with DOI 10.6084/m9.figshare.14273594.

We will align as authors on a readme file as requested. We might currently not be in the capacity to create one that qualifies for a Reusable badge, unfortunately. Thanks again for these insights!

[crowdre]

Dear reviewers @re-ae and @ape810 ,

We have prepared a readme file and included the CC-BY-SA 4.0 license. We have actually gone one step further, and included instructions on how to replicate our study.

If time permits, would you be willing to review whether our dataset now qualifies for a reusable badge? If this is possible, we would greatly appreciate it, but if not, we certainly understand.