search

openjournals / jose-reviews

Reviews for the Journal of Open Source Education (JOSE)
http://jose.theoj.org
Creative Commons Zero v1.0 Universal
33 stars 4 forks source link

[REVIEW]: Auto-eD: A visual learning tool for automatic differentiation #161

Closed whedon closed 2 years ago

whedon commented 2 years ago

Submitting author: !--author-handle-->@lindseysbrown<!--end-author-handle-- (Lindsey Brown) Repository: https://github.com/lindseysbrown/Auto-eD Branch with paper.md (empty if default branch): master Version: v1.0.2 Editor: !--editor-->@labarba<!--end-editor-- Reviewer: @rasbt, @Atcold Archive: 10.5281/zenodo.6800009 Paper kind: software

:warning: JOSE reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSE is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://jose.theoj.org/papers/e485c9012c66d61d25f9a2c8008d441d"><img src="https://jose.theoj.org/papers/e485c9012c66d61d25f9a2c8008d441d/status.svg"></a>
Markdown: [![status](https://jose.theoj.org/papers/e485c9012c66d61d25f9a2c8008d441d/status.svg)](https://jose.theoj.org/papers/e485c9012c66d61d25f9a2c8008d441d)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@rasbt & @Atcold, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/jose-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @labarba know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @rasbt

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @Atcold

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 2 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @rasbt, @Atcold it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/jose-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/jose-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 2 years ago

Wordcount for paper.md is 1305

whedon commented 2 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- None

MISSING DOIs

- 10.1109/5.58337 may be a valid DOI for title: Backpropagation through time: what it does and how to do it

INVALID DOIs

- None
whedon commented 2 years ago 
Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.43 s (161.2 files/s, 104633.3 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
JavaScript                      15           6243           5772          22772
HTML                            23            425              5           3486
Python                           5            310            444           1423
CSS                              6            375             83           1192
reStructuredText                13            392            573            710
Markdown                         3             24              0            117
Jupyter Notebook                 1              0            891             99
TeX                              1              3              0             36
DOS Batch                        1              8              1             27
make                             1              4              6             10
Bourne Shell                     1              0              0              3
-------------------------------------------------------------------------------
SUM:                            70           7784           7775          29875
-------------------------------------------------------------------------------

Statistical information for the repository '12b028f6f1a1c86180d2f1bd' was
gathered on 2022/02/25.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
David Sondak                    17         11196            965           19.61
Rachel Moon                     38         11962           2524           23.36
Xinyue(Cynthia) Wang             5           948              2            1.53
lindseysbrown                  115         25380           9031           55.49

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
David Sondak              10904           97.4          7.1               16.81
Rachel Moon               11544           96.5          0.3               16.48
lindseysbrown             14516           57.2         11.2               16.33
whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

labarba commented 2 years ago

@rasbt, @Atcold — Thank you for agreeing to review for JOSE! This is where the action happens: work your way through the review checklist, feel free to ask questions or post comments here, and also open issues in the submission repository as needed. Godspeed! 🚀

Atcold commented 2 years ago

Drafting here my review

It's really hard to understand what the repo / project offers. There seem to be two components: a Python library and a web app. The web app is hard to use without a tutorial of sorts. The documentation seems more like a lecture (hence, its content should go in the paper, instead), and only later there is a demo of how to use the web app, actually documenting it.

The README.md or the PDF paper (or both) should clearly state how the project is organised and how one should consume the material. Right now, I count 3 places where to read text (paper, developer manual, documentation) and 2 places where to use code (web app, Python library).

If the project is about providing a tool for understanding automatic differentiation, then the repo should display a tutorial as the main component. Or, even better, the web app should walk you through a teaching experience, but I understand this would be harder to put together.

rasbt commented 2 years ago

@labarba General question about opening issues during the review. Should we be using the "convert to Issue" function here and tagging the author in that issue? Screen Shot 2022-03-02 at 3 41 08 PM

Or should we be creating the issues in the project's GitHub repo? Just wondering what the best practices are for JOSE.

EDIT: NVM, I think it is the latter. Not sure why I thought otherwise.

rasbt commented 2 years ago

@Atcold I have a similar experience.

Looking at the GitHub repository, I think most of the files (except ./docs) are the ones that are uploaded to Heroku for creating the website? Ideally, it should be more clear what we are looking at in this repo. This could be addressed by clarifying the code structure in the Readme file.

@Atcold One of us should probably create an issue for that. Since you commented first and said it better, do you want to go ahead with that?

The other issue is that it's not quite clear what we are reviewing here. Maybe @labarba could also help clarifying this?

Atcold commented 2 years ago

Yeah, I can open the issue, @rasbt. And I agree with your sentiment of not knowing what we're asked to review.

labarba commented 2 years ago

Hi folks! If you open any issues, be sure that you do so in the target repository, and mention this issue so we get a cross-link between the two. Thanks!

labarba commented 2 years ago

The author submitted this under the "software" article type, not the "learning module," so I suppose the author intends this application to be used as software to support teaching and learning in machine learning courses. If the author does not make clear how they intend their contribution to be re-used, that is useful input for improvement.

lindseysbrown commented 2 years ago

@labarba Thanks for the clarification. Our submission spanned both so we weren't sure how to submit.

The main contribution is the web app to visualize automatic differentiation that could be used to support any class in which that is taught. We have additionally provided a learning module that is complemented by the use of the web app as an example of how the app could be incorporated. For advanced users with programming experience, we have also provided the underlying software package which can be run independently of the web app.

We can certainly edit the README/paper to make it more clear that you can pick which of these tools are most appropriate based on your level of familiarity with automatic differentiation and with programming:

whedon commented 2 years ago

:wave: @Atcold, please update us on how your review is going (this is an automated reminder).

whedon commented 2 years ago

:wave: @rasbt, please update us on how your review is going (this is an automated reminder).

lindseysbrown commented 2 years ago

@Atcold I think you said you were planning on opening an issue, but I didn't see it in the target repo. I have now added text to both the README and paper (second paragraph in Content) detailing the three different levels of interaction with the software we've provided.

@rasbt @labarba For your reference that this change has been made. (Still not completely clear on if this was more appropriately classified as software or learning module since we did both.)

rasbt commented 2 years ago

Thanks for making those updates @lindseysbrown !!

Regarding the scope of the review. I am still unsure what to do here and would appreciate your feedback and clarification @labarba . I recently read through the learning modules and had some thoughts, but I am not sure if this is appropriate/within the scope of this review. When I am looking through the reviewer checklist at the top of this thread, the review is basically just about making sure the software works (i.e., installs correctly and works as promised). This is very different from a regular journal review, and I am happy to stick to that, but I want to make sure that this is indeed what you want us to do?

labarba commented 2 years ago

👋 hi! When we started JOSE, we planned to have two submission types: software papers, and papers about learning modules (teaching content of some size bigger than a lecture and smaller than a course). When submitting, authors choose one article type, and according to that, our editorial system posts the correct review checklist. On a couple of occasions, we've had submissions that bridge the two article types, but we don't have an editorial process for these edge cases.

I request that you use the review checklist for the software (as that is the submission type chosen by the authors) but comment here on your assessment of the other relevant parts of this project.

labarba commented 2 years ago

Hi @rasbt, @Atcold 👋 – how is it going with your reviews? I see you've checked off a bunch of items on your checklists. Are we waiting for the authors to make some revisions? Give us a quick update when you can!

rasbt commented 2 years ago

Hey @labarba , I have yet another question about the review process 😅. When I reviewed papers for JOSS and JOSE in the past, as far as I remember, there have always been unit tests (and even a CI) as it is best practice for open source.

In the reviewer guidelines it says

Tests: Are there automated tests or manual steps described so that the function of the software can be verified?

So, it does sound like unit tests (or a CI setup) are not required. Technically, one could walk through the examples in (DeveloperDocumentation.ipynb) and check that everything is correct and then rely on jupyter notebook file-diffs to check whether the results can be reproduced on one's own machine. However, this would be quite a burden on the reviewer/user/developer, and in the spirit of open source best-practices, I am wondering if it is okay to suggest to the package maintainers to add unit tests to test the software and verify the functionality on one's own machine?

labarba commented 2 years ago

Our submission guidelines are not exhaustive, because they cannot be, and we do rely on community standards. We should make reference to those community standards when our review guidelines are insufficient.

In regards to testing, the key is that to be eligible for publication, the software needs to adhere to minimum standards that enable reusability: the code should be immediately useful to others. For JOSE, the software should be useful in teaching and learning settings. If the learners do not need to look at or modify the code, as the tool is used more like an app, any tests and documentation target potential contributors, who need to have a way to check correctness. This may exclude unit tests, but at least should have some ability to do system or regression testing. Relying on Jupyter-notebook diffs is surely inadequate for that?

rasbt commented 2 years ago

It's sometimes tricky to strike the balance between being a critical reviewer and trying to improve the submission, but not coming across as asking for unreasonable things.

Thanks for the response on that. I do think that in this case unit tests not only help me as a reviewer, but it could also help users who want to run it locally and use their package verify that everything works as intended in their environment. And, of course it makes contributing and updating this package in the future easier.

I think in this case it doesn't have to be a comprehensive new suite of unit tests hooked up to a CI. A reasonable balance between extra work and utility would be to convert the current examples in the developer doc into unit tests so that someone can run them from their computer and verify the actual results match the expected results.

lindseysbrown commented 2 years ago

@labarba I just wanted to check in on this submission. The only open issue from the reviewers was with regards to testing which I responded to over 2 weeks ago.

rasbt commented 2 years ago

The authors just addressed the last items on my checklist, and everything looks good to me now from my end.

labarba commented 2 years ago

👋 @Atcold — You have some unchecked items in the review checklist for this JOSE submission. Can you give us an update on your review? Have you been able to complete a full review of the submission? Are there any changes you recommended to the authors that remain pending? Thank you! 🙏

Atcold commented 2 years ago

We can certainly edit the README/paper to make it more clear that you can pick which of these tools are most appropriate based on your level of familiarity with automatic differentiation and with programming:

Hi @lindseysbrown, where do I find the updated article?

labarba commented 2 years ago

@whedon generate pdf

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

Atcold commented 2 years ago

Hi @labarba, where do I see the previous version and/or a diff?

labarba commented 2 years ago

Hi @Atcold — we don't have the ability to compile a diff. The changes are of course tracked by commits on the target repository: https://github.com/lindseysbrown/Auto-eD/commits/master

@lindseysbrown — could you summarize the changes, to facilitate the review?

lindseysbrown commented 2 years ago

Hi @Atcold, the main clarifying change in the paper is shown in this commit to outline the three different levels. I think this should address your main concern about the lack of clarity in how someone should interact with the tool/learning module/software here: https://github.com/lindseysbrown/Auto-eD/commit/2d32cb2120ee9d471d25c3966a77d5ea501fedd8

Other changes in response to reviewer comments are on the issues tab (adding versioning, tests, and some edits to a citation).

Atcold commented 2 years ago

Awesome. All good for me now!

labarba commented 2 years ago

We have two happy reviewers and we're ready to move to the last stage and publish your work, @Atcold. 👏

labarba commented 2 years ago

@whedon recommend-accept

whedon commented 2 years ago

No archive DOI set. Exiting...

labarba commented 2 years ago

Oops. I still get the order of things mixed up. A few steps needed:

  1. Please give the article proof one last read-through to find any remaining typos, and approve the proof for publication.
  2. Make a tagged release on the target repository, and report the version number here.
  3. Make an archive of the repository on Zenodo or an equivalent service, and report the archive DOI here.
lindseysbrown commented 2 years ago

@whedon generate pdf

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

lindseysbrown commented 2 years ago

@whedon generate pdf

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

lindseysbrown commented 2 years ago

@whedon generate pdf

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

lindseysbrown commented 2 years ago

@labarba My coauthors and I have reviewed the proof and made a couple minor edits to correct typos.

I have created a tagged release: v1.0.2 (https://github.com/lindseysbrown/Auto-eD/tree/v1.0.2)

And archived that release on Zenodo: 10.5281/zenodo.6800009

Please let me know if you need any further information.

labarba commented 2 years ago

@whedon set v1.0.2 as version

whedon commented 2 years ago

OK. v1.0.2 is the version.

labarba commented 2 years ago

@whedon set 10.5281/zenodo.6800009 as archive

whedon commented 2 years ago

OK. 10.5281/zenodo.6800009 is the archive.

labarba commented 2 years ago

@lindseysbrown — We request that authors edit the metadata of the Zenodo deposit so title and author list match the JOSE paper. It's just cleaner that way as readers see these as part of the "same scholarly object." Could you do that? (Note that you may not want Zenodo to do automatic updates of versions with each release. Note also that Zenodo pulls every committer into the author list.)

lindseysbrown commented 2 years ago

@labarba I've edited the title and author list on Zenodo.

labarba commented 2 years ago

@whedon recommend-accept