Contact information

Jon Whittle (jon.whittle[at]monash.edu)
Gillian Oliver (gillian.oliver[at]monash.edu)
Waqar Hussain (waqar.hussain[at]monash.edu) -- Corresponding Author
Davoud Mougouei (davoud.mougouei[at]monash.edu)
Harsha Perera (harsha.perera[at]monash.edu)
Rifat Ara Shams (rifat.shams[at]monash.edu)
Arif Nurwidyantoro (arif.nurwidyantoro[at]monash.edu)

GitHub Id

https://github.com/ovislabmonash

applying for Available

blind-reviewer-2 commented 4 years ago

Hi Authors,

The repository is overall well structured and contains the data utilised in the attached paper. The data is clean and simple. One downside is that is appears a large portion of the links in the URL column are no longer accessible. See below for my notes including which links appear to be broken. However, I don't feel this creates any issue for assigning this submission a badge. Links, no matter to what resource, may eventually become unavailable. If possible, the authors should correct these links in the dataset and update the version in FigShare. With these comments in mind, I find it clear that this repository is at least "Reusable."

The authors have uploaded the data to FigShare. This resource means that the data will be available for an extended period of time for all interested parties to use. With this included, I find it clear that this repository is at least "Available".

Given the current state of the repository, I recommend a badge of "Available."

Detailed Comments

Reusable Badge (+ Functional)

ReadMe

Well written, easy to follow, and full explanation of dataset (CSV)

Checking CSV for Quality

All fields appear to be clean: no missing or strangely formatted values
All https://conferences.computer.org/ links appear to be broken. Now sure if it is possible to replace those links with working ones.
All dl.acm.org links appear to be working
All https://www.computer.org/ links appear to be working

Available Badge

Dataset Availability: Posted on FigShare License: Creative Commons 4.0 (CC4.0)

gh-artifact-1763 commented 4 years ago

The file is easy to access and hosted on FigShare so "Available". The columns are reasonably easy to follow although it is a bit hard to see how the authors were capturing "agreement" - is it if at least one match occurred in the values selected? a final column that recorded "agreed/not agreed" and final resolution by the third arbiter would help to see if other raters were able to achieve the same high kappa score.

The main problem with the artifact is the curious choice to use URL instead of DOI. Many files are no longer accessible at the URL listed. DOI is in fact exactly for the purpose you are using URL for here - a permanent, (mostly) unchanging locator. Is it possible to replace this with DOI (i.e. you have a separate column somewhere ... ) - I'm guessing no, which diminishes the value of the artifact considerably.

timm commented 4 years ago

dear @ovislabmonash

is @arifn the human-level GH contact for this one?

if you add a DOI then this will be "available". please see Zenodo for easy addition of DOIs

ovislabmonash commented 4 years ago

Thank you @timm and anonymous reviewers. Below I report how we addressed the reviewers' comments.

All https://conferences.computer.org/ links appear to be broken. Now sure if it is possible to replace those links with working ones.

The main problem with the artifact is the curious choice to use URL instead of DOI. Many files are no longer accessible at the URL listed. DOI is in fact exactly for the purpose you are using URL for here - a permanent, (mostly) unchanging locator. Is it possible to replace this with DOI (i.e. you have a separate column somewhere ... ) - I'm guessing no, which diminishes the value of the artifact considerably.

Thank you for noticing the broken URL. We have corrected them to the publication links in the ACM website.

The file is easy to access and hosted on FigShare so "Available". The columns are reasonably easy to follow although it is a bit hard to see how the authors were capturing "agreement" - is it if at least one match occurred in the values selected? a final column that recorded "agreed/not agreed" and final resolution by the third arbiter would help to see if other raters were able to achieve the same high kappa score.

Thank you for the feedback. We have added a column noting the agreement/disagreement between raters. We also added explanations in the readme file about how we captured agreements.