Closed whedon closed 2 years ago
Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @rasbt, @Atcold it looks like you're currently assigned to review this paper :tada:.
:warning: JOSS reduced service mode :warning:
Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.
:star: Important :star:
If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/jose-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews ๐ฟ
To fix this do the following two things:
For a list of things I can do to help you, just type:
@whedon commands
For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:
@whedon generate pdf
Wordcount for paper.md
is 1305
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):
OK DOIs
- None
MISSING DOIs
- 10.1109/5.58337 may be a valid DOI for title: Backpropagation through time: what it does and how to do it
INVALID DOIs
- None
Software report (experimental):
github.com/AlDanial/cloc v 1.88 T=0.43 s (161.2 files/s, 104633.3 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
JavaScript 15 6243 5772 22772
HTML 23 425 5 3486
Python 5 310 444 1423
CSS 6 375 83 1192
reStructuredText 13 392 573 710
Markdown 3 24 0 117
Jupyter Notebook 1 0 891 99
TeX 1 3 0 36
DOS Batch 1 8 1 27
make 1 4 6 10
Bourne Shell 1 0 0 3
-------------------------------------------------------------------------------
SUM: 70 7784 7775 29875
-------------------------------------------------------------------------------
Statistical information for the repository '12b028f6f1a1c86180d2f1bd' was
gathered on 2022/02/25.
The following historical commit information, by author, was found:
Author Commits Insertions Deletions % of changes
David Sondak 17 11196 965 19.61
Rachel Moon 38 11962 2524 23.36
Xinyue(Cynthia) Wang 5 948 2 1.53
lindseysbrown 115 25380 9031 55.49
Below are the number of rows from each author that have survived and are still
intact in the current revision:
Author Rows Stability Age % in comments
David Sondak 10904 97.4 7.1 16.81
Rachel Moon 11544 96.5 0.3 16.48
lindseysbrown 14516 57.2 11.2 16.33
:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
@rasbt, @Atcold โ Thank you for agreeing to review for JOSE! This is where the action happens: work your way through the review checklist, feel free to ask questions or post comments here, and also open issues in the submission repository as needed. Godspeed! ๐
It's really hard to understand what the repo / project offers. There seem to be two components: a Python library and a web app. The web app is hard to use without a tutorial of sorts. The documentation seems more like a lecture (hence, its content should go in the paper, instead), and only later there is a demo of how to use the web app, actually documenting it.
The README.md
or the PDF paper (or both) should clearly state how the project is organised and how one should consume the material. Right now, I count 3 places where to read text (paper, developer manual, documentation) and 2 places where to use code (web app, Python library).
If the project is about providing a tool for understanding automatic differentiation, then the repo should display a tutorial as the main component. Or, even better, the web app should walk you through a teaching experience, but I understand this would be harder to put together.
@labarba General question about opening issues during the review. Should we be using the "convert to Issue" function here and tagging the author in that issue?
Or should we be creating the issues in the project's GitHub repo? Just wondering what the best practices are for JOSE.
EDIT: NVM, I think it is the latter. Not sure why I thought otherwise.
@Atcold I have a similar experience.
Looking at the GitHub repository, I think most of the files (except ./docs) are the ones that are uploaded to Heroku for creating the website? Ideally, it should be more clear what we are looking at in this repo. This could be addressed by clarifying the code structure in the Readme file.
@Atcold One of us should probably create an issue for that. Since you commented first and said it better, do you want to go ahead with that?
The other issue is that it's not quite clear what we are reviewing here. Maybe @labarba could also help clarifying this?
Yeah, I can open the issue, @rasbt. And I agree with your sentiment of not knowing what we're asked to review.
Hi folks! If you open any issues, be sure that you do so in the target repository, and mention this issue so we get a cross-link between the two. Thanks!
The author submitted this under the "software" article type, not the "learning module," so I suppose the author intends this application to be used as software to support teaching and learning in machine learning courses. If the author does not make clear how they intend their contribution to be re-used, that is useful input for improvement.
@labarba Thanks for the clarification. Our submission spanned both so we weren't sure how to submit.
The main contribution is the web app to visualize automatic differentiation that could be used to support any class in which that is taught. We have additionally provided a learning module that is complemented by the use of the web app as an example of how the app could be incorporated. For advanced users with programming experience, we have also provided the underlying software package which can be run independently of the web app.
We can certainly edit the README/paper to make it more clear that you can pick which of these tools are most appropriate based on your level of familiarity with automatic differentiation and with programming:
:wave: @Atcold, please update us on how your review is going (this is an automated reminder).
:wave: @rasbt, please update us on how your review is going (this is an automated reminder).
@Atcold I think you said you were planning on opening an issue, but I didn't see it in the target repo. I have now added text to both the README and paper (second paragraph in Content) detailing the three different levels of interaction with the software we've provided.
@rasbt @labarba For your reference that this change has been made. (Still not completely clear on if this was more appropriately classified as software or learning module since we did both.)
Thanks for making those updates @lindseysbrown !!
Regarding the scope of the review. I am still unsure what to do here and would appreciate your feedback and clarification @labarba . I recently read through the learning modules and had some thoughts, but I am not sure if this is appropriate/within the scope of this review. When I am looking through the reviewer checklist at the top of this thread, the review is basically just about making sure the software works (i.e., installs correctly and works as promised). This is very different from a regular journal review, and I am happy to stick to that, but I want to make sure that this is indeed what you want us to do?
๐ hi! When we started JOSE, we planned to have two submission types: software papers, and papers about learning modules (teaching content of some size bigger than a lecture and smaller than a course). When submitting, authors choose one article type, and according to that, our editorial system posts the correct review checklist. On a couple of occasions, we've had submissions that bridge the two article types, but we don't have an editorial process for these edge cases.
I request that you use the review checklist for the software (as that is the submission type chosen by the authors) but comment here on your assessment of the other relevant parts of this project.
Hi @rasbt, @Atcold ๐ โ how is it going with your reviews? I see you've checked off a bunch of items on your checklists. Are we waiting for the authors to make some revisions? Give us a quick update when you can!
Hey @labarba , I have yet another question about the review process ๐ . When I reviewed papers for JOSS and JOSE in the past, as far as I remember, there have always been unit tests (and even a CI) as it is best practice for open source.
In the reviewer guidelines it says
Tests: Are there automated tests or manual steps described so that the function of the software can be verified?
So, it does sound like unit tests (or a CI setup) are not required. Technically, one could walk through the examples in (DeveloperDocumentation.ipynb) and check that everything is correct and then rely on jupyter notebook file-diffs to check whether the results can be reproduced on one's own machine. However, this would be quite a burden on the reviewer/user/developer, and in the spirit of open source best-practices, I am wondering if it is okay to suggest to the package maintainers to add unit tests to test the software and verify the functionality on one's own machine?
Our submission guidelines are not exhaustive, because they cannot be, and we do rely on community standards. We should make reference to those community standards when our review guidelines are insufficient.
In regards to testing, the key is that to be eligible for publication, the software needs to adhere to minimum standards that enable reusability: the code should be immediately useful to others. For JOSE, the software should be useful in teaching and learning settings. If the learners do not need to look at or modify the code, as the tool is used more like an app, any tests and documentation target potential contributors, who need to have a way to check correctness. This may exclude unit tests, but at least should have some ability to do system or regression testing. Relying on Jupyter-notebook diffs is surely inadequate for that?
It's sometimes tricky to strike the balance between being a critical reviewer and trying to improve the submission, but not coming across as asking for unreasonable things.
Thanks for the response on that. I do think that in this case unit tests not only help me as a reviewer, but it could also help users who want to run it locally and use their package verify that everything works as intended in their environment. And, of course it makes contributing and updating this package in the future easier.
I think in this case it doesn't have to be a comprehensive new suite of unit tests hooked up to a CI. A reasonable balance between extra work and utility would be to convert the current examples in the developer doc into unit tests so that someone can run them from their computer and verify the actual results match the expected results.
@labarba I just wanted to check in on this submission. The only open issue from the reviewers was with regards to testing which I responded to over 2 weeks ago.
The authors just addressed the last items on my checklist, and everything looks good to me now from my end.
๐ @Atcold โ You have some unchecked items in the review checklist for this JOSE submission. Can you give us an update on your review? Have you been able to complete a full review of the submission? Are there any changes you recommended to the authors that remain pending? Thank you! ๐
We can certainly edit the README/paper to make it more clear that you can pick which of these tools are most appropriate based on your level of familiarity with automatic differentiation and with programming:
Hi @lindseysbrown, where do I find the updated article?
@whedon generate pdf
:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
Hi @labarba, where do I see the previous version and/or a diff?
Hi @Atcold โ we don't have the ability to compile a diff. The changes are of course tracked by commits on the target repository: https://github.com/lindseysbrown/Auto-eD/commits/master
@lindseysbrown โ could you summarize the changes, to facilitate the review?
Hi @Atcold, the main clarifying change in the paper is shown in this commit to outline the three different levels. I think this should address your main concern about the lack of clarity in how someone should interact with the tool/learning module/software here: https://github.com/lindseysbrown/Auto-eD/commit/2d32cb2120ee9d471d25c3966a77d5ea501fedd8
Other changes in response to reviewer comments are on the issues tab (adding versioning, tests, and some edits to a citation).
Awesome. All good for me now!
We have two happy reviewers and we're ready to move to the last stage and publish your work, @Atcold. ๐
@whedon recommend-accept
No archive DOI set. Exiting...
Oops. I still get the order of things mixed up. A few steps needed:
@whedon generate pdf
:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
@whedon generate pdf
:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
@whedon generate pdf
:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
@labarba My coauthors and I have reviewed the proof and made a couple minor edits to correct typos.
I have created a tagged release: v1.0.2 (https://github.com/lindseysbrown/Auto-eD/tree/v1.0.2)
And archived that release on Zenodo: 10.5281/zenodo.6800009
Please let me know if you need any further information.
@whedon set v1.0.2 as version
OK. v1.0.2 is the version.
@whedon set 10.5281/zenodo.6800009 as archive
OK. 10.5281/zenodo.6800009 is the archive.
@lindseysbrown โ We request that authors edit the metadata of the Zenodo deposit so title and author list match the JOSE paper. It's just cleaner that way as readers see these as part of the "same scholarly object." Could you do that? (Note that you may not want Zenodo to do automatic updates of versions with each release. Note also that Zenodo pulls every committer into the author list.)
@labarba I've edited the title and author list on Zenodo.
@whedon recommend-accept
Submitting author: !--author-handle-->@lindseysbrown<!--end-author-handle-- (Lindsey Brown) Repository: https://github.com/lindseysbrown/Auto-eD Branch with paper.md (empty if default branch): master Version: v1.0.2 Editor: !--editor-->@labarba<!--end-editor-- Reviewer: @rasbt, @Atcold Archive: 10.5281/zenodo.6800009 Paper kind: software
:warning: JOSE reduced service mode :warning:
Due to the challenges of the COVID-19 pandemic, JOSE is currently operating in a "reduced service mode". You can read more about what that means in our blog post.
Status
Status badge code:
Reviewers and authors:
Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)
Reviewer instructions & questions
@rasbt & @Atcold, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:
The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @labarba know.
โจ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest โจ
Review checklist for @rasbt
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper
paper.md
file include a list of authors with their affiliations?Review checklist for @Atcold
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper
paper.md
file include a list of authors with their affiliations?