search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
710 stars 38 forks source link

[REVIEW]: TX2: Transformer eXplainability and eXploration #3652

Closed whedon closed 2 years ago

whedon commented 3 years ago

Submitting author: @WildfireXIII (Nathan Martindale) Repository: https://github.com/ORNL/tx2 Version: v1.0.0 Editor: @fabian-s Reviewers: @assenmacher-mat, @deniederhut Archive: 10.5281/zenodo.5796089

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/b7c161917e5a31af052a597bf98f0e94"><img src="https://joss.theoj.org/papers/b7c161917e5a31af052a597bf98f0e94/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/b7c161917e5a31af052a597bf98f0e94/status.svg)](https://joss.theoj.org/papers/b7c161917e5a31af052a597bf98f0e94)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@assenmacher-mat & @sara-02, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @fabian-s know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @assenmacher-mat

✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @deniederhut

✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 3 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @assenmacher-mat, @sara-02 it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 3 years ago

Wordcount for paper.md is 1341

whedon commented 3 years ago 
Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.20 s (299.4 files/s, 146974.3 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
JavaScript                      15           2516           2689           9868
HTML                            12           1495             36           2909
SVG                              1              0              0           2671
Python                           9            402            512           1630
CSS                              5            183             36            734
Jupyter Notebook                 2              0           2621            302
Markdown                         5             69              0            177
reStructuredText                 8            121            152            164
TeX                              1             17              0             96
DOS Batch                        1              8              1             26
make                             1              4              7              9
-------------------------------------------------------------------------------
SUM:                            60           4815           6054          18586
-------------------------------------------------------------------------------

Statistical information for the repository 'faa95a716c4f24951fd7338d' was
gathered on 2021/08/24.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Martindale, Nathan               7          2146             68           12.25
Nathan Martindale               16         15697            158           87.75

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Martindale, Nathan        17617          820.9          0.5               17.03
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.18653/v1/P19-3007 is OK
- 10.1145/3366424.3383542 is OK

MISSING DOIs

- 10.18653/v1/2020.emnlp-demos.15 may be a valid DOI for title: The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models

INVALID DOIs

- None
fabian-s commented 3 years ago

👋🏼 @WildfireXIII @assenmacher-mat @sara-02, this is the review thread for the paper. All of our communications will happen here from now on.

Both reviewers have checklists at the top of this thread with the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#REVIEW_NUMBER so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 2-4 weeks. Please let me know if any of you require some more time. We can also use Whedon (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@fabian-s) if you have any questions/concerns.

whedon commented 3 years ago

:wave: @assenmacher-mat, please update us on how your review is going (this is an automated reminder).

whedon commented 3 years ago

:wave: @sara-02, please update us on how your review is going (this is an automated reminder).

assenmacher-mat commented 3 years ago

👋 @assenmacher-mat, please update us on how your review is going (this is an automated reminder).

still struggling a little bit with the runtime on my cpu machine, but everythin seems to work. Should be done by the end of this week.

assenmacher-mat commented 3 years ago

Two small issues:

fabian-s commented 3 years ago

@WildfireXIII : please add a CONTRIBUTING.md file or similar to your repo, as pointed out by @assenmacher-mat

fabian-s commented 3 years ago

@sara-02 please update us on how your review is going, as you know JOSS aims for quick turnaround times.

stewartsl commented 3 years ago

@WildfireXIII : please add a CONTRIBUTING.md file or similar to your repo, as pointed out by @assenmacher-mat

@fabian-s please note that there is currently a contributing.MD file in master tx2 contributing. does this sufficiently address your concern?

fabian-s commented 3 years ago

@fabian-s please note that there is currently a contributing.MD file in master tx2 contributing. does this sufficiently address your concern?

sure, that seems sufficient.

WarmCyan commented 2 years ago

Hey, I haven't heard anything in a while, so I just wanted to check what the status on this review was, and if there's anything else needed from me so far.

fabian-s commented 2 years ago

@sara-02 please do update us on how this review is going, as you know JOSS aims for quick turnaround times and we are waiting for your review to proceed.

fabian-s commented 2 years ago

@sara-02 ping

fabian-s commented 2 years ago

@WildfireXIII very sorry for the delay, @sara-02 is not reacting to my e-mails or messages. I'll find someone else for the 2nd review.

fabian-s commented 2 years ago

hi @ahurriyetoglu @richardlitt @deniederhut @linuxscout -- would one of you be willing and able to review this submission for JOSS within the next 2 weeks?

RichardLitt commented 2 years ago

Bit out of my wheelhouse. Sorry!

deniederhut commented 2 years ago

Hey @fabian-s yeah I should have some time to take a look at this late next week. Would you mind asking whedon to remind me? Thanks

fabian-s commented 2 years ago

thanks, @deniederhut!

fabian-s commented 2 years ago

@whedon add @deniederhut as reviewer

whedon commented 2 years ago

OK, @deniederhut is now a reviewer

fabian-s commented 2 years ago

@whedon remind @deniederhut in eight days

whedon commented 2 years ago

Reminder set for @deniederhut in eight days

deniederhut commented 2 years ago

Couple of initial questions:

deniederhut commented 2 years ago

Paper suggestions: https://github.com/ORNL/tx2/issues/1 Documentation suggestion: https://github.com/ORNL/tx2/issues/2 Questions about API/usage: https://github.com/ORNL/tx2/issues/3 I didn't find any tests for this software: https://github.com/ORNL/tx2/issues/4

fabian-s commented 2 years ago

thx @deniederhut!

WarmCyan commented 2 years ago

Great, thank you @deniederhut for all the feedback!

is this just an archival reference to the repository? Or a separate publication?

Yeah, it's just a DOI to point to more details about the project, not a publication

I see a lot of commits by Nathan Martindale, but this does not seem to be associated with the github account listed above

Oops, yeah that's me! I made the commits from a work computer and I'm guessing this is because I have a different email in my global git config than normal, I'll look into fixing it. Sorry about that!

@fabian-s what's the protocol for me, once the linked issues are addressed and eventually closed, do I throw a comment up for it here?

fabian-s commented 2 years ago

@fabian-s what's the protocol for me, once the linked issues are addressed and eventually closed, do I throw a comment up for it here?

sure, that will work. if you reference openjournals/joss-reviews#3652 in your comments in the linked issues (or commit messages), notifications will pop up here as well.

deniederhut commented 2 years ago

Hey @fabian-s , could I ask you to weigh in on https://github.com/ORNL/tx2/issues/4? I'm not super clear on the testing requirements for JOSS submissions.

fabian-s commented 2 years ago

@WildfireXIII please update us on your progress in adressing these issues.

stewartsl commented 2 years ago

@WildfireXIII please update us on your progress in adressing these issues.

We have made some progress and will continue to work on these next week after the holiday break.

WarmCyan commented 2 years ago

@deniederhut Hey! Just wanted to leave a note that I've made an update to each of the issues, so it's ready if you want to take another pass at it.

deniederhut commented 2 years ago

Thanks for doing that! Also thanks for the ping. I'll take a look now.

deniederhut commented 2 years ago

@whedon generate pdf

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

WarmCyan commented 2 years ago

@fabian-s hey if you get a chance, could you drop by https://github.com/ORNL/tx2/issues/1? There's some questions out of my scope.

deniederhut commented 2 years ago

Okay confirmed that the notebooks run as expected. @fabian-s I think we're good to go here!

fabian-s commented 2 years ago

@whedon generate pdf

fabian-s commented 2 years ago

@whedon check references

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 2 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.18653/v1/P19-3007 is OK
- 10.1145/3366424.3383542 is OK

MISSING DOIs

- 10.18653/v1/2020.emnlp-demos.15 may be a valid DOI for title: The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models
- 10.21105/joss.00861 may be a valid DOI for title: UMAP: Uniform Manifold Approximation and Projection

INVALID DOIs

- None
fabian-s commented 2 years ago

Wonderful, we might be able to get this done before the holidays. Thank you all for working on this submission.

@WildfireXIII - final requested changes for style & spelling:

WarmCyan commented 2 years ago

@fabian-s requested changes made and pushed!

fabian-s commented 2 years ago

@whedon generate pdf

fabian-s commented 2 years ago

@whedon check references

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 2 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.18653/v1/2020.emnlp-demos.15 is OK
- 10.21105/joss.00861 is OK
- 10.18653/v1/P19-3007 is OK
- 10.1145/3366424.3383542 is OK

MISSING DOIs

- None

INVALID DOIs

- None