Review of submission 24ferrari

data-track-reviewer commented 5 years ago

I believe I should be assigned to this issue, but appear to lack rights. @neilernst , could you please help?

neilernst commented 5 years ago

you will have to accept the collaborator invite, then can self-assign

data-track-reviewer commented 5 years ago

Summary

The authors present a data set for their RE submission Learning Requirements Elicitation Interviews with Role-playing, Self-assessment and Peer-review. The data set includes all material to rerun the training as well as the evaluation reported on in the paper. It therefore provides value in two aspects: 1. an educational resource for RE teachers, 2. an empirical resource for replication.

Evaluation

The authors apply here for availability. Thus, the data set must be checked against functionality and availability criteria.

Functional: I find the data set well documented and complete. The associated paper does provide "appropriate evidence of verification and validation", it is however not linked in the repository (likely because submission is pending). Perhaps a gist of this evidence and a placeholder for a link to the paper to be provided after publication could be added?
Available: The data set is available on github and zenodo, likely with zenodo being the leading repository. Thus, I find the criteria for this badge fulfilled.

In summary, I am in favor of the availability badge, but suggest to provide links to the paper as soon as these become available.

Further remarks

One could consider the reusable badge for this data set as well. For that, one of the following should be provided:

Resuable for empirical evaluation: Rational and plan for participant selection as well as experimental setup should be provided in form of guidelines. The goal of this would be to make it easy to repeat the setup in a different course setting, thus being "very carefully documented and well-structured to the extent that reuse and repurposing is facilitated".
Reusable for educational purposes: Relation to learning objectives and assessment should be provided as well as guidelines for teachers on how to include this module.

The authors however do not apply for this badge, thus this might be ideas for future work.

alessioferrari commented 5 years ago

@data-track-reviewer Thanks a lot for the very constructive feedback.

We provided a placeholder for the link to the paper in Zenodo, to be filled when the paper is published. To this end, we updated the Zenodo entry by adding the sentence:

More information is available in the publication "Learning Requirements Elicitation Interviews with Role-playing, Self-assessment and Peer-review", to appear in the proceedings of the 27th IEEE International Requirements Engineering Conference, 2019.

Please check: https://doi.org/10.5281/zenodo.2625705

Thank you for the suggestion about the reusable badge. At the moment, we would consider the option of "reusable for educational purposes", as, at this stage, we are more interested in this.

In your opinion, would it make sense to apply for the reusable badge, by including guidelines for teachers, and learning objectives, but without the guidelines for experimental evaluation? If @data-track-reviewer, @timm and @neilernst think it makes sense, we'll provide the material by tomorrow, end of the day.

Thank you again.

timm commented 5 years ago

@data-track-reviewer : please advise, when that new new material arrives from @alessioferrari , if this artifact now available and reusable

data-track-reviewer commented 5 years ago

Sure, I believe that would be a great addition. I will gladly check tomorrow and provide feedback/recommendation. I guess it is the decision of @timm and @neilernst , but feel encouraged by @timm 's comment...

One additional thought about the evidence: It is nicely summarized in the paper, so I am happy with the link. It could have been super cool to get a spreadsheet with the data from your evaluation. But, well, one can always wish for more. I do not see the availability badge in question.

alessioferrari commented 5 years ago

Thanks a million for the fast reply, we'll provide additional information on how to include the material in a course by today.

Concerning the spreadsheets: thanks a lot for the suggestion, we thought about that, but at the moment providing a full description of all the raw data in a way that can be easily reused would require a non negligible time, so at the moment we prefer to focus on the other material. As we're thinking about extending the work, we'll take this into account in the follow-up, thanks!

alessioferrari commented 5 years ago

@data-track-reviewer we have added the suggested information in the README file, and also updated the Zenodo record: https://zenodo.org/record/2625706#.XSXmRy17E60

We have also modified the status file, to notify that we apply also for the reusable badge.

We have also included a link to the YouTube video lectures corresponding to the slides.

Thanks again

neilernst commented 5 years ago

@data-track-reviewer please confirm the latest changes are acceptable

timm commented 5 years ago

@neilernst : i'm a little confused. i understand our criteria for "available" and "reusable" but i think @alessioferrari is proposing a new a badge ""reusable for educational purposes"

@data-track-reviewer : as i understand your comments, this can be available and you are debating resuable. is that correct? if so, when will you decide.

@alessioferrari , we cannot invent a badge ""reusable for educational purposes** since our goal is inclusion to ACMportal and IEEExplorer and they only support the badges as we have defined the,

neilernst commented 5 years ago

@timm I reached out to the secret identity of our reviewer by email as well.

alessioferrari commented 5 years ago

Hi Tim,

I originally applied for the available badge. After the feedback, I added some info to apply also for the standard reusable badge. So no special badge needed.

As the data is educational material, it should be reusable for educational purposes. But this is just an additional information, not a request for a special badge.

Hope this makes things clearer,

Thanks a lot!

On Mon, 15 Jul 2019, 06:04 Neil Ernst, notifications@github.com wrote:

@timm https://github.com/timm I reached out to the secret identity of our reviewer by email as well.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/researchart/re19/issues/19?email_source=notifications&email_token=AA6AGQK6QHJXBPT3662G3STP7PZMLA5CNFSM4H5CPMGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ4VMEA#issuecomment-511268368, or mute the thread https://github.com/notifications/unsubscribe-auth/AA6AGQLBOBQ3V4UXUWUOITTP7PZMLANCNFSM4H5CPMGA .

timm commented 5 years ago

I have reviewed this material and I agree with the authors that a standard reusable badge applies here. Therefore I state that badges of this artifact are

reusable
available

@neilernst : do you concur?

data-track-reviewer commented 5 years ago

Sorry, I was traveling without internet. But I see that the discussion has developed according to my view. Also, I like the additions.

The badges I recommend are reusable and available.

alessioferrari commented 5 years ago

Thanks a lot to everyone for the feedback and the badges.

On Wed, 17 Jul 2019, 10:48 data-track-reviewer, notifications@github.com wrote:

Sorry, I was traveling without internet. But I see that the discussion has developed according to my view. Also, I like the additions.

The badges I recommend are reusable and available.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/researchart/re19/issues/19?email_source=notifications&email_token=AA6AGQJPW2EHATH5O7YPYHTP73MGTA5CNFSM4H5CPMGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2DP5LA#issuecomment-512163500, or mute the thread https://github.com/notifications/unsubscribe-auth/AA6AGQPNQHCWAR54FGKBA7DP73MGTANCNFSM4H5CPMGA .

researchart / re19