search

researchart / fse18

artifacts track, FSE'18: https://2018.fseconference.org/home
2 stars 2 forks source link

DeFreez_function-embedding #22

Closed obaysal closed 6 years ago

obaysal commented 6 years ago

https://github.com/researchart/fse18/tree/master/submissions/function-embedding

anonymous-reviewer18 commented 6 years ago

Installation looks fine using docker. Tried some commands and it runs without errors. All of the files are downloadable online. I'm okay with the reusable and available status.

ReviewerArtFSE18 commented 6 years ago

I appreciated the provision of a Docker image with the artifact. Using the image I could perfectly run the tool without experiencing any issues.

The documentation is complete and describes clearly the different steps from installation to execution. It also provides scripts to reproduce the results of the paper totaly and parcially.

The source code of the artifact is publicly available.

The artifact meets the requirements to receive the Available bage.

I would like to clarify one point. The authors are asking for two badges, reusable and availabe. As far as I know, the available badge has higher grade than resuable. Awarding the available badge means that the artifact is also reusable. @timm could you please confirm this?

timm commented 6 years ago

Available implies reusable.

So given the above reviews, we would label this "available".

defreez-ucd commented 6 years ago

Many thanks to the reviewers for looking at our artifact and providing feedback. I am happy that everything went smoothly.

@timm Does Available implying Reusable mean that both badges would appear with the paper?

masud-technope commented 6 years ago

I have the same question @defreez-ucd. Currently, "Available" does not look much intuitive.

timm commented 6 years ago

we are stuck with the ACM badge names. when we announce all this, i suggest that you hit the twitter and facebook verse and announce that your reusable artifact has been judge by the FSE'18 review board to be available and ready for use by other researchers. in those posts, do not mention olga and me by name (that would be like naming the midwives when the real news is the baby).

defreez-ucd commented 6 years ago

@timm My question is whether a single Available badge will be applied or both badges will be applied.

For example:

Two badges were requested because our artifact meets the criteria for a Reusable badge, and it is publicly available.

My concern is that a reader will see the Available badge - without the corresponding Reusable badge - and come to the conclusion that the artifact was merely made public, and have no way of knowing that it also meets the Reusable criteria. The reader might even conclude that it was not reviewed at all, which would be justified by the ACM page, which states that

Artifacts do not need to have been formally evaluated in order for an article to receive [the Available] badge. In addition, they need not be complete in the sense described above. They simply need to be relevant to the study and add value beyond the text in the article. Such artifacts could be something as simple as the data from which the figures are drawn, or as complex as a complete software system under study.

Daniel

timm commented 6 years ago

i was unaware ACM was displaying multiple badges. but you are right

image

image

timm commented 6 years ago

i wonder what is ACM's policy for a top-of-the-line badge (reproduced). does the paper get 5 badges at top of page?

image

anyone know of such an example?

defreez-ucd commented 6 years ago

I'm not certain, but my interpretation of the policy page is that 3 would be the maximum. One badge from each category of Evaluation, Availability, and Validation. This goes along with the color scheme too, with Reusable (dark red) > Functional, and Reproduced (Dark Blue) > Replicated.

There are no papers in the ACM digital library with a Reproduced badge. This paper has 3 (Reusable, Available, Replicated).

Other papers have a validation badge and an evaluation badge, but are not available. I think this is further evidence that the three categories are independent.

defreez-ucd commented 6 years ago

@timm @obaysal

Now that the dust has settled, I just want to add my voice to the chorus of appreciation. This was an innovative and rewarding review process. I was impressed by the levels of openness, interaction, and responsiveness. Allowing authors to interact with the committee helped meet the unique demands of artifact review. I am delighted that I was able to participate.