Review of submission Daun_RE_AE

LloydThinks commented 3 years ago

Author: @tenbergen Reviewer 1: tba Reviewer 2: @johndoejohndoejohndoee

johndoejohndoejohndoee commented 3 years ago

Specific comments:

Number mismatch for every table: Table II - "Proposal of solution" - 69 in the excel, but 68 in the Paper. Same with "Type of learners". Table IV - many examples. Please carefully check your numbers again.
The URL is not immutable. As outlined in the Eligibility and Evaluation Criteria, the URL cannot be an institutional website
The README section should contain direct link to the artifact

Artifacts Available - Available Badge

Artifacts are hosted online - Yes
The URL to access the artifacts is immutable (cannot be altered by the author) - No
The organisation hosting the URL plans to maintain it for the foreseeable future. - Not sure
Artifacts have a Digital Object Identifier (DOI) redirecting to the immutable URL. - DOI is present
Anyone can access the artifacts, without the need for registration. - Yes

Artifacts Reusable - Reusable Badge

Artifacts can be obtained via the “Artifact Location” README section. - No
Artifacts are documented in the “Descriptions of Artifacts” README section. - Yes
Artifacts can be installed and run via the “Installation Instructions” README section. - N/A
Artifacts generate the results presented in the article following the “Steps to Reproduce” README section. - See comments
Artifacts have a proper open-source license attached as a LICENSE.md file - N/A
The artifacts are very carefully documented and well-structured to the extent that reuse and repurposing is facilitated. Norms and standards of the research community for artifacts of this type are strictly adhered to. - See comments

LloydThinks commented 3 years ago

Thank you @johndoejohndoejohndoee for your initial review. Upon internal review, it appears that the SmithScholarWorks "is an institutional repository that provides permanent online access to advance scholarship and encourage the growth of scholarly communities through open access, quick discovery, and wide dissemination of scholarly and creative content" [1]. For this reason, we believe it qualifies as an immutable and long-term solution to storing the artifact.

@tenbergen You have applied for all four badges and I hope to clear up some potential confusion regarding how you can qualify for these badges:

Available Badge: As reviewed above and clarified by the track Co-Chairs, we believe your artifact already qualifies for this badge.
Reusable: There does not appear to be a README.md or LICENSE.md attached to your data artifact available at the provided by the SmithScholarWorks link. These two files, as outlined in our submission guidelines [2], are very important and must be included. If you wish to qualify for the Reusable badge, please create these two files and put them in the same repository as your current submission (a single link must lead to the complete package).
You have also applied for the Reproduced and Replicated badges, however, you can only qualify for one of them, depending on the requirements met. I tried to read your pre-print, but it has been deleted from your GoogleDrive. This is why we recommend using services like arXiv that maintain pre-prints forever. Regardless, if YOU replicated someone else's work, then the OTHER article would receive the Reproduced or Replicated badge, not your article. This is because the OTHER work has now been Reproduced/Replicated. Additionally, only past RE submissions can receive the Reproduced or Replicated badge, so once you have provided a valid link to your pre-print, I would check the replicated article's submission venue to assess quantifiability.

I hope this clears some things up. Let me know if you have any additional questions. We are still waiting to hear from the second reviewer, who can help provide some final clarity on the Reusability of your work, should you choose to create the required documentation as outline in the submission guidelines [2].

[1] https://scholarworks.smith.edu/about.html [2] https://conf.researchr.org/getImage/RE-2021/orig/RE%2721+Artifact+Track+-+Submission+and+Reviewing+Guidelines.pdf

johndoejohndoejohndoee commented 3 years ago

SmithScholarWorks "is an institutional repository that provides permanent online access to advance scholarship and encourage the growth of scholarly communities through open access, quick discovery, and wide dissemination of scholarly and creative content"

Makes sense to me. So the qualification for Available Badge is met.

My only concern remains is the numbers mismatch between the dataset and the paper, since it's a requirement to be able to reproduce the results for Artifact Reusable, correct?

tenbergen commented 3 years ago

Dear all,

Thank you for your continued review of our submission. Regarding your request for clarification:

Available Badge: As reviewed above and clarified by the track Co-Chairs, we believe your artifact already qualifies for this badge.

Understood. Thank you.

Reusable: There does not appear to be a README.md or LICENSE.md attached to your data artifact available at the provided by the SmithScholarWorks link. These two files, as outlined in our submission guidelines [2], are very important and must be included. If you wish to qualify for the Reusable badge, please create these two files and put them in the same repository as your current submission (a single link must lead to the complete package).

We were extremely confused regarding the submission criteria for the artifact track and had trouble mapping github .md files and pull requests to the specific artifact we provided. We were particularly confused about where the LICSNESE and README needs to go and thought you mean that it should go with the github pull request. If you look therein, you can find the files there.

You are right that SmithScholarWorks provides permanent and immutable DOIs, but it’s possible for us to add those files, if you wish. Once we upload them, we can no longer delete them, however.

You have also applied for the Reproduced and Replicated badges, however, you can only qualify for one of them, depending on the requirements met. I tried to read your pre-print, but it has been deleted from your GoogleDrive. This is why we recommend using services like arXiv that maintain pre-prints forever. Regardless, if YOU replicated someone else's work, then the OTHER article would receive the Reproduced or Replicated badge, not your article. This is because the OTHER work has now been Reproduced/Replicated. Additionally, only past RE submissions can receive the Reproduced or Replicated badge, so once you have provided a valid link to your pre-print, I would check the replicated article's submission venue to assess quantifiability.

Thank you for the clarification. Like I stated above, we were very confused about what exactly you mean by these badges. It did not become clear to us that reproduced/replicated badges are mutually exclusive. We were furthermore confused what you mean by “preprint” and whether arXiv was mandatory or not – we looked at other submissions and they provided google doc links as well. Moreover, the preprint of our paper was a preliminary version prior to the camera ready copy. The most recent version is here: https://drive.google.com/file/d/1Ci7SeduhOgl-poxVxJhJ8QyJ6r-W1ThB/view?usp=sharing

Therein, you can see that we have in part replicated work by someone else. It was not an RE submission however.

Thanks for your continued consideration.

--Bastian

http://www.oswego.edu/

Dr. Bastian Tenbergen Asst. Professor of Software Engineering

Department of Computer Science 427 Shineman Center State University of New York at Oswego 7060 State Route 104 Oswego, NY, 13126, USA T +1 (315) 312-6605 E @.> @. http://www.tenbergen.org/ www.tenbergen.org

https://twitter.com/BTenbergen https://www.researchgate.net/profile/Bastian_Tenbergen https://www.linkedin.com/profile/view?id=24491293

“Understand well as I may, my comprehension can only be an infinitesimal fraction of all I want to understand.”

 – Ada Countess of Lovelace, Mathematician and “Mother of Computing.”

From: Lloyd Montgomery @.> Sent: Thursday, July 8, 2021 11:38 AM To: researchart/re21 @.> Cc: Bastian Tenbergen @.>; Mention @.> Subject: Re: [researchart/re21] Review of submission Daun_RE_AE (#6)

Thank you @johndoejohndoejohndoee https://www.google.com/url?q=https://github.com/johndoejohndoejohndoee&source=gmail-imap&ust=1626363497000000&usg=AOvVaw2eyxXhEJtU9pUZYrJr-Vpn for your initial review. Upon internal review, it appears that the SmithScholarWorks "is an institutional repository that provides permanent online access to advance scholarship and encourage the growth of scholarly communities through open access, quick discovery, and wide dissemination of scholarly and creative content" [1]. For this reason, we believe it qualifies as an immutable and long-term solution to storing the artifact.

@tenbergen https://www.google.com/url?q=https://github.com/tenbergen&source=gmail-imap&ust=1626363497000000&usg=AOvVaw2ev3336hQy4vY-OI8Qn6Zz You have applied for all four badges and I hope to clear up some potential confusion regarding how you can qualify for these badges:

Available Badge: As reviewed above and clarified by the track Co-Chairs, we believe your artifact already qualifies for this badge.
Reusable: There does not appear to be a README.md or LICENSE.md attached to your data artifact available at the provided by the SmithScholarWorks link. These two files, as outlined in our submission guidelines [2], are very important and must be included. If you wish to qualify for the Reusable badge, please create these two files and put them in the same repository as your current submission (a single link must lead to the complete package).
You have also applied for the Reproduced and Replicated badges, however, you can only qualify for one of them, depending on the requirements met. I tried to read your pre-print, but it has been deleted from your GoogleDrive. This is why we recommend using services like arXiv that maintain pre-prints forever. Regardless, if YOU replicated someone else's work, then the OTHER article would receive the Reproduced or Replicated badge, not your article. This is because the OTHER work has now been Reproduced/Replicated. Additionally, only past RE submissions can receive the Reproduced or Replicated badge, so once you have provided a valid link to your pre-print, I would check the replicated article's submission venue to assess quantifiability.

I hope this clears some things up. Let me know if you have any additional questions. We are still waiting to hear from the second reviewer, who can help provide some final clarity on the Reusability of your work, should you choose to create the required documentation as outline in the submission guidelines [2].

[1] https://scholarworks.smith.edu/about.html https://www.google.com/url?q=https://scholarworks.smith.edu/about.html&source=gmail-imap&ust=1626363497000000&usg=AOvVaw399VbNnA4X4jv7DPUS8UXH [2] https://conf.researchr.org/getImage/RE-2021/orig/RE%2721+Artifact+Track+-+Submission+and+Reviewing+Guidelines.pdf https://www.google.com/url?q=https://conf.researchr.org/getImage/RE-2021/orig/RE%252721%2BArtifact%2BTrack%2B-%2BSubmission%2Band%2BReviewing%2BGuidelines.pdf&source=gmail-imap&ust=1626363497000000&usg=AOvVaw1Jlhymjyo_n8lwMp0SEGTZ

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/researchart/re21/issues/6%23issuecomment-876542076&source=gmail-imap&ust=1626363497000000&usg=AOvVaw2G4vw8DQQuk8tbZ0p3_o9E , or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AEBDJBBC3I7AF3ITARTDRG3TWXA6RANCNFSM47OAZ2ZA&source=gmail-imap&ust=1626363497000000&usg=AOvVaw3gzoTI-DGaXt_VGGbapM4J . https://github.com/notifications/beacon/AEBDJBEBS5ELCDUZAP26F23TWXA6RA5CNFSM47OAZ2ZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOGQ7PQ7A.gif

tenbergen commented 3 years ago

Dear all,

Regarding the remaining concern:

My only concern remains is the numbers mismatch between the dataset and the paper, since it's a requirement to be able to reproduce the results for Artifact Reusable, correct?

Since the camera-ready deadline has not yet passed at the time of submission to the RE artifact track, we submitted a preliminary preprint (and did not use arXiv). The mismatch of numbers has been fixed in the camera-ready copy, which you can find here: https://drive.google.com/file/d/1Ci7SeduhOgl-poxVxJhJ8QyJ6r-W1ThB/view?usp=sharing

--Bastian

http://www.oswego.edu/

Dr. Bastian Tenbergen Asst. Professor of Software Engineering

Department of Computer Science 427 Shineman Center State University of New York at Oswego 7060 State Route 104 Oswego, NY, 13126, USA T +1 (315) 312-6605 E @.> @. http://www.tenbergen.org/ www.tenbergen.org

https://twitter.com/BTenbergen https://www.researchgate.net/profile/Bastian_Tenbergen https://www.linkedin.com/profile/view?id=24491293

“Understand well as I may, my comprehension can only be an infinitesimal fraction of all I want to understand.”

 – Ada Countess of Lovelace, Mathematician and “Mother of Computing.”

From: johndoejohndoejohndoee @.> Sent: Thursday, July 8, 2021 12:57 PM To: researchart/re21 @.> Cc: Bastian Tenbergen @.>; Mention @.> Subject: Re: [researchart/re21] Review of submission Daun_RE_AE (#6)

SmithScholarWorks "is an institutional repository that provides permanent online access to advance scholarship and encourage the growth of scholarly communities through open access, quick discovery, and wide dissemination of scholarly and creative content"

Makes sense to me. So the qualification for Available Badge is met.

My only concern remains is the numbers mismatch between the dataset and the paper, since it's a requirement to be able to reproduce the results for Artifact Reusable, correct?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/researchart/re21/issues/6%23issuecomment-876596416&source=gmail-imap&ust=1626368224000000&usg=AOvVaw0m7A0LjKP5ETy1Wv1nQpDL , or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AEBDJBHKBRFDUDMVHJW6NW3TWXKF7ANCNFSM47OAZ2ZA&source=gmail-imap&ust=1626368224000000&usg=AOvVaw0XOLcOrZhlI6hvyVhs7beK . https://github.com/notifications/beacon/AEBDJBESXOGM3VCHNGD3NWTTWXKF7A5CNFSM47OAZ2ZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOGQ74ZQA.gif

LloydThinks commented 3 years ago

Thank you for the clarification @tenbergen . The process is indeed complicated and we apologise for the confusion.

For the submission documents, we state in the submission guidelines "Required submission documents for the RE AE Track are listed below." and this includes the SUBMISSION.md file as well as the pre-print. We don't explicitly require that you upload your pre-print to arXiv, but when I tried to access your GoogleDrive file, it was in your trash. So please consider using arXiv in the future as it maintains your pre-print forever, which is desirable for Open Science.

As for the Reusable badge, we state in the submission guidelines "The Reusable badge requires additional documents to be created and stored with your artifact." The artifact being your hosted files in the immutable and long-term storage. These files are required for the reviewers to check your work, but more importantly, they are designed to live with your artifacts forever, and act as a guide for future researchers who want to use your work. For this reason, submitting them to GitHub only fulfils the first requirement.

I hope this clears up the purpose of the artifact track and your understanding of Open Science. Our goal is to educate and hopefully bring clarity to the process. We want to see your artifacts used many years into the future 😃

Lloyd

LloydThinks commented 3 years ago

@tenbergen @johndoejohndoejohndoee We outline in the submission guidelines that "All known deviations from results presented in the article must be explicitly outlined (E.g., when a table or figure is not produced, or the produced results are different from the results presented in the paper)." We understand that sometimes Reusable artifacts produce numbers slightly different than the paper. However, in such cases, we hope that you (the author, in your README.md) explain why the artifacts now produce different numbers or slightly off figures. This helps future researchers using your artifacts to understand how to interpret the different numbers.

In the best case, you will update your artifact when you make local changes to your paper. This way, the artifact stays relevant. Most archival services allow you to update documents, even though they don't let you remove documents.

I hope with the updated information we can assign you the Available and Reusable badges. This is up to @johndoejohndoejohndoee and the second reviewer who will be reviewing your artifact shortly.

Lloyd

tenbergen commented 3 years ago

@tenbergen https://www.google.com/url?q=https://github.com/tenbergen&source=gmail-imap&ust=1626378503000000&usg=AOvVaw1E7IDJ1TYJ0EEf7uVYhLc9 @johndoejohndoejohndoee https://www.google.com/url?q=https://github.com/johndoejohndoejohndoee&source=gmail-imap&ust=1626378503000000&usg=AOvVaw280g_kwzqnc7gCXMVLZ73F We outline in the submission guidelines that "All known deviations from results presented in the article must be explicitly outlined (E.g., when a table or figure is not produced, or the produced results are different from the results presented in the paper)."

Agreed. Like I said, in our case, the deviations were due to modifications requested by the reviewer for the CRC of the paper that, to the best of my knowledge are now all corrected.

We understand that sometimes Reusable artifacts produce numbers slightly different than the paper. However, in such cases, we hope that you (the author, in your README.md) explain why the artifacts now produce different numbers or slightly off figures. This helps future researchers using your artifacts to understand how to interpret the different numbers.

Understood. I must have overlooked the instructions where it said to explain issues pertaining to data in the README.md. Can you point me in the direction of where I can find that?

Best,

Bastian

http://www.oswego.edu/

Dr. Bastian Tenbergen Asst. Professor of Software Engineering

Department of Computer Science 427 Shineman Center State University of New York at Oswego 7060 State Route 104 Oswego, NY, 13126, USA T +1 (315) 312-6605 E @.> @. http://www.tenbergen.org/ www.tenbergen.org

https://twitter.com/BTenbergen https://www.researchgate.net/profile/Bastian_Tenbergen https://www.linkedin.com/profile/view?id=24491293

“Understand well as I may, my comprehension can only be an infinitesimal fraction of all I want to understand.”

 – Ada Countess of Lovelace, Mathematician and “Mother of Computing.”

From: Lloyd Montgomery @.> Sent: Thursday, July 8, 2021 3:48 PM To: researchart/re21 @.> Cc: Bastian Tenbergen @.>; Mention @.> Subject: Re: [researchart/re21] Review of submission Daun_RE_AE (#6)

@tenbergen https://www.google.com/url?q=https://github.com/tenbergen&source=gmail-imap&ust=1626378503000000&usg=AOvVaw1E7IDJ1TYJ0EEf7uVYhLc9 @johndoejohndoejohndoee https://www.google.com/url?q=https://github.com/johndoejohndoejohndoee&source=gmail-imap&ust=1626378503000000&usg=AOvVaw280g_kwzqnc7gCXMVLZ73F We outline in the submission guidelines that "All known deviations from results presented in the article must be explicitly outlined (E.g., when a table or figure is not produced, or the produced results are different from the results presented in the paper)." We understand that sometimes Reusable artifacts produce numbers slightly different than the paper. However, in such cases, we hope that you (the author, in your README.md) explain why the artifacts now produce different numbers or slightly off figures. This helps future researchers using your artifacts to understand how to interpret the different numbers.

In the best case, you will update your artifact when you make local changes to your paper. This way, the artifact stays relevant. Most archival services allow you to update documents, even though they don't let you remove documents.

I hope with the updated information we can assign you the Available and Reusable badges. This is up to @johndoejohndoejohndoee https://www.google.com/url?q=https://github.com/johndoejohndoejohndoee&source=gmail-imap&ust=1626378503000000&usg=AOvVaw280g_kwzqnc7gCXMVLZ73F and the second reviewer who will be reviewing your artifact shortly.

Lloyd

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/researchart/re21/issues/6%23issuecomment-876699839&source=gmail-imap&ust=1626378503000000&usg=AOvVaw3ZXeItKy9vzm1OO0nisc2l , or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AEBDJBHJMKBWWAL2KESD7JTTWX6INANCNFSM47OAZ2ZA&source=gmail-imap&ust=1626378503000000&usg=AOvVaw36flHx1uTjaQu_lST9bT1z . https://github.com/notifications/beacon/AEBDJBCKA7Y7NLEA4MLHEC3TWX6INA5CNFSM47OAZ2ZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOGRAWBPY.gif

LloydThinks commented 3 years ago

The RE website summarises some of the information [1], but the full submission guidelines are linked at the bottom of the page [2].

[1] https://conf.researchr.org/track/RE-2021/RE-2021-artifacts#Call-for-Artifacts [2] https://conf.researchr.org/getImage/RE-2021/orig/RE%2721+Artifact+Track+-+Submission+and+Reviewing+Guidelines.pdf

sc4585545 commented 3 years ago

Hi all, I agree with the previous concerns regarding the artifact, and mainly regarding the readme which does not have information regarding the artifact, and no link is provides. My main concern is lacking papers about educational experiences that foster knowledge management, creativity and innovation in RE, and there are many such as the ones provided below. However, the work is impressive and relevant to the conference.

Levy, M. (2020). "Emotional Requirements for Well-being Applications: The Customer Journey", In: Proceedings of REWBAH, The 1st International Workshop on Requirements Engineering for Well-Being, Aging and Health in conjunction with RE’20 – 31 August 2020, Zurich, Switzerland. Levy, M. (2018). "Educating for Empathy in Software Engineering Course", The 1st International Workshop on Facilitating Inclusive Requirements Engineering (FIRE), co-located with the 24th Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ), Utrecht, Netherlands. Levy, M. (2017). "Promoting the Elicitation of Usability and Accessibility Requirements in Design Thinking: Using a Designed Object as a Boundary Object", The 3rd International Workshop on Usability and Accessibility focused Requirements Engineering (UsARE), co-located with the 25th IEEE International Requirements Engineering Conference (RE), Lisbon, Portugal, Sept 4-8, 2017. Levy, M. (2017). "Design Thinking in Multidisciplinary Learning Teams: Insights from Multidisciplinary Teaching Events". The 1st international workshop on Teaching for Smart Information Systems - Smart Information Systems for Teaching (T4ST4T'17), co-located with the 29th International Conference on Advanced Information Systems Engineering, Essen, Germany, June 12-16, 2017. Levy, M. and Hadar, I. (2010) “Teaching MBA Students the Use of Web2.0: The Knowledge Management Perspective”, Journal of Information Systems Education. vol. 21, no. 1, pp.55-67.

tenbergen commented 3 years ago

Everyone,

I’m confused. What is the aim of the artifact review? Is it to evaluate the quality of the content of the artifact? Or availablility to other researchers? I thought the latter, not the former. If it’s the former, please be advised that the collected data was already reviewed by the conference program committee and accepted by the same. Moreover, many of the works cited below meet our exclusion criteria.

Regarding the README, with apologies, from the submission instructions it was not unambiguously clear where to put the README. Other submissions didn’t have a readme that I could easily see. Now, we offered to add the README to the DOI, but we have not received final instructions from you whether you would like us to do that.

We can also do the same with the license.md file, even though SmithScholarWorks publishes under CC-BY-4.0.

Please advise.

--Bastian

http://www.oswego.edu/

Dr. Bastian Tenbergen Asst. Professor of Software Engineering

Department of Computer Science 427 Shineman Center State University of New York at Oswego 7060 State Route 104 Oswego, NY, 13126, USA T +1 (315) 312-6605 E @.> @. http://www.tenbergen.org/ www.tenbergen.org

https://twitter.com/BTenbergen https://www.researchgate.net/profile/Bastian_Tenbergen https://www.linkedin.com/profile/view?id=24491293

“Understand well as I may, my comprehension can only be an infinitesimal fraction of all I want to understand.”

 – Ada Countess of Lovelace, Mathematician and “Mother of Computing.”

From: sc4585545 @.> Sent: Friday, July 9, 2021 12:21 PM To: researchart/re21 @.> Cc: Bastian Tenbergen @.>; Mention @.> Subject: Re: [researchart/re21] Review of submission Daun_RE_AE (#6)

Hi all, I agree with the previous concerns regarding the artifact, and mainly regarding the readme which does not have information regarding the artifact, and no link is provides. My main concern is lacking papers about educational experiences that foster knowledge management, creativity and innovation in RE, and there are many such as the ones provided below. However, the work is impressive and relevant to the conference.

Levy, M. (2020). "Emotional Requirements for Well-being Applications: The Customer Journey", In: Proceedings of REWBAH, The 1st International Workshop on Requirements Engineering for Well-Being, Aging and Health in conjunction with RE’20 – 31 August 2020, Zurich, Switzerland. Levy, M. (2018). "Educating for Empathy in Software Engineering Course", The 1st International Workshop on Facilitating Inclusive Requirements Engineering (FIRE), co-located with the 24th Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ), Utrecht, Netherlands. Levy, M. (2017). "Promoting the Elicitation of Usability and Accessibility Requirements in Design Thinking: Using a Designed Object as a Boundary Object", The 3rd International Workshop on Usability and Accessibility focused Requirements Engineering (UsARE), co-located with the 25th IEEE International Requirements Engineering Conference (RE), Lisbon, Portugal, Sept 4-8, 2017. Levy, M. (2017). "Design Thinking in Multidisciplinary Learning Teams: Insights from Multidisciplinary Teaching Events". The 1st international workshop on Teaching for Smart Information Systems - Smart Information Systems for Teaching (T4ST4T'17), co-located with the 29th International Conference on Advanced Information Systems Engineering, Essen, Germany, June 12-16, 2017. Levy, M. and Hadar, I. (2010) “Teaching MBA Students the Use of Web2.0: The Knowledge Management Perspective”, Journal of Information Systems Education. vol. 21, no. 1, pp.55-67.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/researchart/re21/issues/6%23issuecomment-877302938&source=gmail-imap&ust=1626452470000000&usg=AOvVaw2rP4CSeq1ADxUL6fUwwetA , or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AEBDJBALICV4OFBWC74NZDTTW4OXLANCNFSM47OAZ2ZA&source=gmail-imap&ust=1626452470000000&usg=AOvVaw2u9GsA_hG8aweksz498C1C . https://github.com/notifications/beacon/AEBDJBAV74ZMJ6EQFSE5PKDTW4OXLA5CNFSM47OAZ2ZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOGRFJJGQ.gif

LloydThinks commented 3 years ago

Hi @tenbergen , thank you for your patience as the reviews came in.

The purpose of the Artifact Track is to assess the availability and quality of your provided artifacts, combined with the requested documents. In traditional review processes, artifacts such as code are not assessed. In your case, there is a cross-over of assessment because your artifact is an Excel sheet, which, by the nature of your survey paper, was already assessed during the standard review process. I hope you understand the confusion of the situation for everyone involved.

Please attach the README.md and LICENSE.md to your artifact at SmithScholarWorks. While it appears that SmithScholarWorks has a nice landing page of information for your artifact and perhaps it is obvious to some what the licences attached to SmithScholarWorks are, if someone downloads your artifact and closes the webpage, that information is lost.

The Artifact Track Co-Chairs and I will discuss your submission soon and make a decision regarding the badges based on the feedback of the reviewers.

Thank you for your patience. We will get back to you soon regarding our summary judgement.

Lloyd & Nash

LloydThinks commented 3 years ago

Hi @tenbergen,

Nash and I have discussed your submission and here are our findings:

Available Badge: We find that your submission qualifies for the available badge.
Reproduced and Replicated Badges: We do not find that your submission qualifies for these badges. The criteria explicitly states "The main results of the article have been obtained in a subsequent study that was peer-reviewed and published by a person or team other than the authors, … using, in part, artifacts provided by the author. (Reproduced) … without the use of author-supplied artifacts. (Replicated)" We do not see evidence of someone else reproducing or replicating your work. As such, we do not find that your submission qualifies for either of these badges.
Reusable Badge: We believe that your submission could qualify for the Reusable badge if you upload the README.md and LICENSE.md next to your artifact. Their purpose is to facilitate the reuse of your artifact, so they do not serve their purpose unless they are attached to the artifact. As you have already created the README.md and LICENSE.md files, we believe this step should be easy. Once you have done so, please respond to this Github issue letting us know about your update, and we will assign your artifact the badge of Reusable. Please note that you have until Friday July 16th to do so.

We hope we have cleared up some of the confusion. Your feedback has led to some re-thinking about how to explain things already. We look forward to your response and hopefully assigning the Reusable badge as well.

Lloyd & Nash

researchart / re21