Open LloydThinks opened 3 years ago
Thank you @johndoejohndoejohndoee for your initial review. Upon internal review, it appears that the SmithScholarWorks "is an institutional repository that provides permanent online access to advance scholarship and encourage the growth of scholarly communities through open access, quick discovery, and wide dissemination of scholarly and creative content" [1]. For this reason, we believe it qualifies as an immutable and long-term solution to storing the artifact.
@tenbergen You have applied for all four badges and I hope to clear up some potential confusion regarding how you can qualify for these badges:
I hope this clears some things up. Let me know if you have any additional questions. We are still waiting to hear from the second reviewer, who can help provide some final clarity on the Reusability of your work, should you choose to create the required documentation as outline in the submission guidelines [2].
[1] https://scholarworks.smith.edu/about.html [2] https://conf.researchr.org/getImage/RE-2021/orig/RE%2721+Artifact+Track+-+Submission+and+Reviewing+Guidelines.pdf
SmithScholarWorks "is an institutional repository that provides permanent online access to advance scholarship and encourage the growth of scholarly communities through open access, quick discovery, and wide dissemination of scholarly and creative content"
Makes sense to me. So the qualification for Available Badge is met.
My only concern remains is the numbers mismatch between the dataset and the paper, since it's a requirement to be able to reproduce the results for Artifact Reusable, correct?
Dear all,
Thank you for your continued review of our submission. Regarding your request for clarification:
Understood. Thank you.
We were extremely confused regarding the submission criteria for the artifact track and had trouble mapping github .md files and pull requests to the specific artifact we provided. We were particularly confused about where the LICSNESE and README needs to go and thought you mean that it should go with the github pull request. If you look therein, you can find the files there.
You are right that SmithScholarWorks provides permanent and immutable DOIs, but it’s possible for us to add those files, if you wish. Once we upload them, we can no longer delete them, however.
Thank you for the clarification. Like I stated above, we were very confused about what exactly you mean by these badges. It did not become clear to us that reproduced/replicated badges are mutually exclusive. We were furthermore confused what you mean by “preprint” and whether arXiv was mandatory or not – we looked at other submissions and they provided google doc links as well. Moreover, the preprint of our paper was a preliminary version prior to the camera ready copy. The most recent version is here: https://drive.google.com/file/d/1Ci7SeduhOgl-poxVxJhJ8QyJ6r-W1ThB/view?usp=sharing
Therein, you can see that we have in part replicated work by someone else. It was not an RE submission however.
Thanks for your continued consideration.
--Bastian
Dr. Bastian Tenbergen Asst. Professor of Software Engineering
Department of Computer Science 427 Shineman Center State University of New York at Oswego 7060 State Route 104 Oswego, NY, 13126, USA T +1 (315) 312-6605 E @.> @. http://www.tenbergen.org/ www.tenbergen.org
https://twitter.com/BTenbergen https://www.researchgate.net/profile/Bastian_Tenbergen https://www.linkedin.com/profile/view?id=24491293
“Understand well as I may, my comprehension can only be an infinitesimal fraction of all I want to understand.”
– Ada Countess of Lovelace, Mathematician and “Mother of Computing.”
From: Lloyd Montgomery @.> Sent: Thursday, July 8, 2021 11:38 AM To: researchart/re21 @.> Cc: Bastian Tenbergen @.>; Mention @.> Subject: Re: [researchart/re21] Review of submission Daun_RE_AE (#6)
Thank you @johndoejohndoejohndoee https://www.google.com/url?q=https://github.com/johndoejohndoejohndoee&source=gmail-imap&ust=1626363497000000&usg=AOvVaw2eyxXhEJtU9pUZYrJr-Vpn for your initial review. Upon internal review, it appears that the SmithScholarWorks "is an institutional repository that provides permanent online access to advance scholarship and encourage the growth of scholarly communities through open access, quick discovery, and wide dissemination of scholarly and creative content" [1]. For this reason, we believe it qualifies as an immutable and long-term solution to storing the artifact.
@tenbergen https://www.google.com/url?q=https://github.com/tenbergen&source=gmail-imap&ust=1626363497000000&usg=AOvVaw2ev3336hQy4vY-OI8Qn6Zz You have applied for all four badges and I hope to clear up some potential confusion regarding how you can qualify for these badges:
I hope this clears some things up. Let me know if you have any additional questions. We are still waiting to hear from the second reviewer, who can help provide some final clarity on the Reusability of your work, should you choose to create the required documentation as outline in the submission guidelines [2].
[1] https://scholarworks.smith.edu/about.html https://www.google.com/url?q=https://scholarworks.smith.edu/about.html&source=gmail-imap&ust=1626363497000000&usg=AOvVaw399VbNnA4X4jv7DPUS8UXH [2] https://conf.researchr.org/getImage/RE-2021/orig/RE%2721+Artifact+Track+-+Submission+and+Reviewing+Guidelines.pdf https://www.google.com/url?q=https://conf.researchr.org/getImage/RE-2021/orig/RE%252721%2BArtifact%2BTrack%2B-%2BSubmission%2Band%2BReviewing%2BGuidelines.pdf&source=gmail-imap&ust=1626363497000000&usg=AOvVaw1Jlhymjyo_n8lwMp0SEGTZ
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/researchart/re21/issues/6%23issuecomment-876542076&source=gmail-imap&ust=1626363497000000&usg=AOvVaw2G4vw8DQQuk8tbZ0p3_o9E , or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AEBDJBBC3I7AF3ITARTDRG3TWXA6RANCNFSM47OAZ2ZA&source=gmail-imap&ust=1626363497000000&usg=AOvVaw3gzoTI-DGaXt_VGGbapM4J . https://github.com/notifications/beacon/AEBDJBEBS5ELCDUZAP26F23TWXA6RA5CNFSM47OAZ2ZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOGQ7PQ7A.gif
Dear all,
Regarding the remaining concern:
Since the camera-ready deadline has not yet passed at the time of submission to the RE artifact track, we submitted a preliminary preprint (and did not use arXiv). The mismatch of numbers has been fixed in the camera-ready copy, which you can find here: https://drive.google.com/file/d/1Ci7SeduhOgl-poxVxJhJ8QyJ6r-W1ThB/view?usp=sharing
--Bastian
Dr. Bastian Tenbergen Asst. Professor of Software Engineering
Department of Computer Science 427 Shineman Center State University of New York at Oswego 7060 State Route 104 Oswego, NY, 13126, USA T +1 (315) 312-6605 E @.> @. http://www.tenbergen.org/ www.tenbergen.org
https://twitter.com/BTenbergen https://www.researchgate.net/profile/Bastian_Tenbergen https://www.linkedin.com/profile/view?id=24491293
“Understand well as I may, my comprehension can only be an infinitesimal fraction of all I want to understand.”
– Ada Countess of Lovelace, Mathematician and “Mother of Computing.”
From: johndoejohndoejohndoee @.> Sent: Thursday, July 8, 2021 12:57 PM To: researchart/re21 @.> Cc: Bastian Tenbergen @.>; Mention @.> Subject: Re: [researchart/re21] Review of submission Daun_RE_AE (#6)
SmithScholarWorks "is an institutional repository that provides permanent online access to advance scholarship and encourage the growth of scholarly communities through open access, quick discovery, and wide dissemination of scholarly and creative content"
Makes sense to me. So the qualification for Available Badge is met.
My only concern remains is the numbers mismatch between the dataset and the paper, since it's a requirement to be able to reproduce the results for Artifact Reusable, correct?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/researchart/re21/issues/6%23issuecomment-876596416&source=gmail-imap&ust=1626368224000000&usg=AOvVaw0m7A0LjKP5ETy1Wv1nQpDL , or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AEBDJBHKBRFDUDMVHJW6NW3TWXKF7ANCNFSM47OAZ2ZA&source=gmail-imap&ust=1626368224000000&usg=AOvVaw0XOLcOrZhlI6hvyVhs7beK . https://github.com/notifications/beacon/AEBDJBESXOGM3VCHNGD3NWTTWXKF7A5CNFSM47OAZ2ZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOGQ74ZQA.gif
Thank you for the clarification @tenbergen . The process is indeed complicated and we apologise for the confusion.
For the submission documents, we state in the submission guidelines "Required submission documents for the RE AE Track are listed below." and this includes the SUBMISSION.md file as well as the pre-print. We don't explicitly require that you upload your pre-print to arXiv, but when I tried to access your GoogleDrive file, it was in your trash. So please consider using arXiv in the future as it maintains your pre-print forever, which is desirable for Open Science.
As for the Reusable badge, we state in the submission guidelines "The Reusable badge requires additional documents to be created and stored with your artifact." The artifact being your hosted files in the immutable and long-term storage. These files are required for the reviewers to check your work, but more importantly, they are designed to live with your artifacts forever, and act as a guide for future researchers who want to use your work. For this reason, submitting them to GitHub only fulfils the first requirement.
I hope this clears up the purpose of the artifact track and your understanding of Open Science. Our goal is to educate and hopefully bring clarity to the process. We want to see your artifacts used many years into the future 😃
Lloyd
@tenbergen @johndoejohndoejohndoee We outline in the submission guidelines that "All known deviations from results presented in the article must be explicitly outlined (E.g., when a table or figure is not produced, or the produced results are different from the results presented in the paper)." We understand that sometimes Reusable artifacts produce numbers slightly different than the paper. However, in such cases, we hope that you (the author, in your README.md) explain why the artifacts now produce different numbers or slightly off figures. This helps future researchers using your artifacts to understand how to interpret the different numbers.
In the best case, you will update your artifact when you make local changes to your paper. This way, the artifact stays relevant. Most archival services allow you to update documents, even though they don't let you remove documents.
I hope with the updated information we can assign you the Available and Reusable badges. This is up to @johndoejohndoejohndoee and the second reviewer who will be reviewing your artifact shortly.
Lloyd
Agreed. Like I said, in our case, the deviations were due to modifications requested by the reviewer for the CRC of the paper that, to the best of my knowledge are now all corrected.
Understood. I must have overlooked the instructions where it said to explain issues pertaining to data in the README.md. Can you point me in the direction of where I can find that?
Best,
Bastian
Dr. Bastian Tenbergen Asst. Professor of Software Engineering
Department of Computer Science 427 Shineman Center State University of New York at Oswego 7060 State Route 104 Oswego, NY, 13126, USA T +1 (315) 312-6605 E @.> @. http://www.tenbergen.org/ www.tenbergen.org
https://twitter.com/BTenbergen https://www.researchgate.net/profile/Bastian_Tenbergen https://www.linkedin.com/profile/view?id=24491293
“Understand well as I may, my comprehension can only be an infinitesimal fraction of all I want to understand.”
– Ada Countess of Lovelace, Mathematician and “Mother of Computing.”
From: Lloyd Montgomery @.> Sent: Thursday, July 8, 2021 3:48 PM To: researchart/re21 @.> Cc: Bastian Tenbergen @.>; Mention @.> Subject: Re: [researchart/re21] Review of submission Daun_RE_AE (#6)
@tenbergen https://www.google.com/url?q=https://github.com/tenbergen&source=gmail-imap&ust=1626378503000000&usg=AOvVaw1E7IDJ1TYJ0EEf7uVYhLc9 @johndoejohndoejohndoee https://www.google.com/url?q=https://github.com/johndoejohndoejohndoee&source=gmail-imap&ust=1626378503000000&usg=AOvVaw280g_kwzqnc7gCXMVLZ73F We outline in the submission guidelines that "All known deviations from results presented in the article must be explicitly outlined (E.g., when a table or figure is not produced, or the produced results are different from the results presented in the paper)." We understand that sometimes Reusable artifacts produce numbers slightly different than the paper. However, in such cases, we hope that you (the author, in your README.md) explain why the artifacts now produce different numbers or slightly off figures. This helps future researchers using your artifacts to understand how to interpret the different numbers.
In the best case, you will update your artifact when you make local changes to your paper. This way, the artifact stays relevant. Most archival services allow you to update documents, even though they don't let you remove documents.
I hope with the updated information we can assign you the Available and Reusable badges. This is up to @johndoejohndoejohndoee https://www.google.com/url?q=https://github.com/johndoejohndoejohndoee&source=gmail-imap&ust=1626378503000000&usg=AOvVaw280g_kwzqnc7gCXMVLZ73F and the second reviewer who will be reviewing your artifact shortly.
Lloyd
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/researchart/re21/issues/6%23issuecomment-876699839&source=gmail-imap&ust=1626378503000000&usg=AOvVaw3ZXeItKy9vzm1OO0nisc2l , or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AEBDJBHJMKBWWAL2KESD7JTTWX6INANCNFSM47OAZ2ZA&source=gmail-imap&ust=1626378503000000&usg=AOvVaw36flHx1uTjaQu_lST9bT1z . https://github.com/notifications/beacon/AEBDJBCKA7Y7NLEA4MLHEC3TWX6INA5CNFSM47OAZ2ZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOGRAWBPY.gif
The RE website summarises some of the information [1], but the full submission guidelines are linked at the bottom of the page [2].
[1] https://conf.researchr.org/track/RE-2021/RE-2021-artifacts#Call-for-Artifacts [2] https://conf.researchr.org/getImage/RE-2021/orig/RE%2721+Artifact+Track+-+Submission+and+Reviewing+Guidelines.pdf
Hi all, I agree with the previous concerns regarding the artifact, and mainly regarding the readme which does not have information regarding the artifact, and no link is provides. My main concern is lacking papers about educational experiences that foster knowledge management, creativity and innovation in RE, and there are many such as the ones provided below. However, the work is impressive and relevant to the conference.
Levy, M. (2020). "Emotional Requirements for Well-being Applications: The Customer Journey", In: Proceedings of REWBAH, The 1st International Workshop on Requirements Engineering for Well-Being, Aging and Health in conjunction with RE’20 – 31 August 2020, Zurich, Switzerland. Levy, M. (2018). "Educating for Empathy in Software Engineering Course", The 1st International Workshop on Facilitating Inclusive Requirements Engineering (FIRE), co-located with the 24th Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ), Utrecht, Netherlands. Levy, M. (2017). "Promoting the Elicitation of Usability and Accessibility Requirements in Design Thinking: Using a Designed Object as a Boundary Object", The 3rd International Workshop on Usability and Accessibility focused Requirements Engineering (UsARE), co-located with the 25th IEEE International Requirements Engineering Conference (RE), Lisbon, Portugal, Sept 4-8, 2017. Levy, M. (2017). "Design Thinking in Multidisciplinary Learning Teams: Insights from Multidisciplinary Teaching Events". The 1st international workshop on Teaching for Smart Information Systems - Smart Information Systems for Teaching (T4ST4T'17), co-located with the 29th International Conference on Advanced Information Systems Engineering, Essen, Germany, June 12-16, 2017. Levy, M. and Hadar, I. (2010) “Teaching MBA Students the Use of Web2.0: The Knowledge Management Perspective”, Journal of Information Systems Education. vol. 21, no. 1, pp.55-67.
Everyone,
I’m confused. What is the aim of the artifact review? Is it to evaluate the quality of the content of the artifact? Or availablility to other researchers? I thought the latter, not the former. If it’s the former, please be advised that the collected data was already reviewed by the conference program committee and accepted by the same. Moreover, many of the works cited below meet our exclusion criteria.
Regarding the README, with apologies, from the submission instructions it was not unambiguously clear where to put the README. Other submissions didn’t have a readme that I could easily see. Now, we offered to add the README to the DOI, but we have not received final instructions from you whether you would like us to do that.
We can also do the same with the license.md file, even though SmithScholarWorks publishes under CC-BY-4.0.
Please advise.
--Bastian
Dr. Bastian Tenbergen Asst. Professor of Software Engineering
Department of Computer Science 427 Shineman Center State University of New York at Oswego 7060 State Route 104 Oswego, NY, 13126, USA T +1 (315) 312-6605 E @.> @. http://www.tenbergen.org/ www.tenbergen.org
https://twitter.com/BTenbergen https://www.researchgate.net/profile/Bastian_Tenbergen https://www.linkedin.com/profile/view?id=24491293
“Understand well as I may, my comprehension can only be an infinitesimal fraction of all I want to understand.”
– Ada Countess of Lovelace, Mathematician and “Mother of Computing.”
From: sc4585545 @.> Sent: Friday, July 9, 2021 12:21 PM To: researchart/re21 @.> Cc: Bastian Tenbergen @.>; Mention @.> Subject: Re: [researchart/re21] Review of submission Daun_RE_AE (#6)
Hi all, I agree with the previous concerns regarding the artifact, and mainly regarding the readme which does not have information regarding the artifact, and no link is provides. My main concern is lacking papers about educational experiences that foster knowledge management, creativity and innovation in RE, and there are many such as the ones provided below. However, the work is impressive and relevant to the conference.
Levy, M. (2020). "Emotional Requirements for Well-being Applications: The Customer Journey", In: Proceedings of REWBAH, The 1st International Workshop on Requirements Engineering for Well-Being, Aging and Health in conjunction with RE’20 – 31 August 2020, Zurich, Switzerland. Levy, M. (2018). "Educating for Empathy in Software Engineering Course", The 1st International Workshop on Facilitating Inclusive Requirements Engineering (FIRE), co-located with the 24th Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ), Utrecht, Netherlands. Levy, M. (2017). "Promoting the Elicitation of Usability and Accessibility Requirements in Design Thinking: Using a Designed Object as a Boundary Object", The 3rd International Workshop on Usability and Accessibility focused Requirements Engineering (UsARE), co-located with the 25th IEEE International Requirements Engineering Conference (RE), Lisbon, Portugal, Sept 4-8, 2017. Levy, M. (2017). "Design Thinking in Multidisciplinary Learning Teams: Insights from Multidisciplinary Teaching Events". The 1st international workshop on Teaching for Smart Information Systems - Smart Information Systems for Teaching (T4ST4T'17), co-located with the 29th International Conference on Advanced Information Systems Engineering, Essen, Germany, June 12-16, 2017. Levy, M. and Hadar, I. (2010) “Teaching MBA Students the Use of Web2.0: The Knowledge Management Perspective”, Journal of Information Systems Education. vol. 21, no. 1, pp.55-67.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/researchart/re21/issues/6%23issuecomment-877302938&source=gmail-imap&ust=1626452470000000&usg=AOvVaw2rP4CSeq1ADxUL6fUwwetA , or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AEBDJBALICV4OFBWC74NZDTTW4OXLANCNFSM47OAZ2ZA&source=gmail-imap&ust=1626452470000000&usg=AOvVaw2u9GsA_hG8aweksz498C1C . https://github.com/notifications/beacon/AEBDJBAV74ZMJ6EQFSE5PKDTW4OXLA5CNFSM47OAZ2ZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOGRFJJGQ.gif
Hi @tenbergen , thank you for your patience as the reviews came in.
The purpose of the Artifact Track is to assess the availability and quality of your provided artifacts, combined with the requested documents. In traditional review processes, artifacts such as code are not assessed. In your case, there is a cross-over of assessment because your artifact is an Excel sheet, which, by the nature of your survey paper, was already assessed during the standard review process. I hope you understand the confusion of the situation for everyone involved.
Please attach the README.md and LICENSE.md to your artifact at SmithScholarWorks. While it appears that SmithScholarWorks has a nice landing page of information for your artifact and perhaps it is obvious to some what the licences attached to SmithScholarWorks are, if someone downloads your artifact and closes the webpage, that information is lost.
The Artifact Track Co-Chairs and I will discuss your submission soon and make a decision regarding the badges based on the feedback of the reviewers.
Thank you for your patience. We will get back to you soon regarding our summary judgement.
Lloyd & Nash
Hi @tenbergen,
Nash and I have discussed your submission and here are our findings:
We hope we have cleared up some of the confusion. Your feedback has led to some re-thinking about how to explain things already. We look forward to your response and hopefully assigning the Reusable badge as well.
Lloyd & Nash
Author: @tenbergen Reviewer 1: tba Reviewer 2: @johndoejohndoejohndoee