[REVIEW]: Automated Sleep Stage Scoring Using k-Nearest Neighbors Classifier

whedon commented 4 years ago

Submitting author: @teamPSG (Tamas Kiss) Repository: https://github.com/teamPSG/kNN_Sleep_Scorer_kNNSS Version: v1.0.0 Editor: @oliviaguest Reviewers: @Emma-k-ward, @sbuergers, @Trisbek, @samikakumar Archive: 10.5281/zenodo.4009178

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/5726bdd4b144467c8093e90df0169c25"><img src="https://joss.theoj.org/papers/5726bdd4b144467c8093e90df0169c25/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/5726bdd4b144467c8093e90df0169c25/status.svg)](https://joss.theoj.org/papers/5726bdd4b144467c8093e90df0169c25)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@Emma-k-ward & @sbuergers, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

Make sure you're logged in to your GitHub account
Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @oliviaguest know.

✨ Please try and complete your review in the next six weeks ✨

Review checklist for @Emma-k-ward

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@teamPSG) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @sbuergers

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@teamPSG) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @Trisbek

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@teamPSG) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @samikakumar

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@teamPSG) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

oliviaguest commented 4 years ago

@teamPSG do you want to have both previous and current? Or just change to current?

teamPSG commented 4 years ago

@oliviaguest I was thinking to have both.

teamPSG commented 4 years ago

@whedon generate pdf

whedon commented 4 years ago

:point_right: Check article proof :page_facing_up: :point_left:

teamPSG commented 4 years ago

@oliviaguest I uploaded the revised version implementing the "Current affiliation" part the simple way so that you can see what I mean exactly. Pls. let me know if this should be done differently.

oliviaguest commented 4 years ago

@teamPSG ah, thanks! I will check and get back to you. In the meantime (no pressure, asking to get organised), are you working on what @Emma-k-ward has pending (the items not checked off yet)?

teamPSG commented 4 years ago

@oliviaguest Yes, all modifications are done, uploaded, typing response to reviewers right now :mage:

teamPSG commented 4 years ago

Hi @oliviaguest and Reviewers,

Authors would like to thank all Reviewers for their insightful comments and suggestions. I included here a list of answers and modifications describing accompanying commit: https://github.com/teamPSG/kNN_Sleep_Scorer_kNNSS/commit/3f268a639a482fc635a3c0fff5324adcb6deff6e

@trisbek and @samikakumar: Re: Functionality/Performance -- we included a second dataset recorded from rats in control condition. The data files and manual scores are uploaded to GitHub and Readme.md files in the Software_Verification and the Example_Data folders were updated to reflect this. Furthermore, the three example scripts in Software_Verification were slightly modified to allow for a comfortable processing of the newly added data. Similarly, the JOSS paper was updated and now includes a figure about prediction accuracy of the rat data. Data from the non-human primate (NHP) recordings have not been published yet so we would not like to disclose that data or results at this time.

Re: Script to find model maximizing overall accuracy -- Thank you for this idea, it might be a good addition to the package on GitHub at a later time. In fact, in practical situations the experimenter might not want to maximize an average prediction accuracy. Indeed, the reason for adding the "deflated model" was to increase goodness of REM prediction at the expense of decreasing Wake and NREM prediction accuracy. Since Wake and NREM are much more numerous than REM a slight decrease of W/NR prediction brings overall scores (like Mathew's correlation coefficient) down. In short, each real-world research has its very specific questions and narrowing down which model to use depends very much on this question. So yes, it will be a good idea to collect a number of scripts that look for optimal models from a number of point of views.

Re: Using the tool for human EEG -- the package was initially developed to help pre-clinical drug research by enabling the uniform scoring of data from mice, rats and NHPs as the drug candidate advances through the discovery pipeline. While we recognize the importance of developing tools for the analysis of human EEG, the focus of our work was to enable robust translatable automated sleep analysis for animal studies.

@Emma-k-ward: Re: Code | path -- sorry for the confusion with setting up the path. I typically like to code the way I did it here, where I have a main par structure and use its fields later on in the script. I am wondering if combining the content of the setup.m file with your main script (in the fashion you like it) would help you get this better organized.

Re: Code | comments -- I tried my best and added more comments in the most important functions :bulb:

Re: Documentation | features -- this is a very helpful comment, thank you Emma! I wrote the whole thing with electrophysiologists -- particularly those engaged with sleep research -- in mind. But you're right, people with a different background might (I hope they will) read into the paper and have the same questions. I added a few sentences in the paper to be more precise about what the features mean. Indeed, as you say, the features are literally human-readable: if I was to plot the EEG_theta, the EEG_delta and the EMG_high features I would be able to roughly point out when the animal is awake, when he is in non-REM or REM sleep. In rats and mice, in wakefulness there is typically high muscle (EMG) activity and the power in the EEG is low, except for the theta (4-12Hz) band when the animal is actively exploring its environment. Contrary to this, during sleep, EMG is low. During REM sleep there is a gigantic theta peak, while during non-REM sleep delta is high. The point clouds in this 3 dimensional space are not clearly separate so the need for a classifier. Furthermore, other states such as cataplexy, might require other but similar features to be well classified. I also removed the reference to XAI Wiki page from the paper since as you pointed out it was not appropriately placed .

Re: Documentation | transitions & states -- the three "basic" stages are the stages that are most likely present in rodents. In human (and even in non-human primates) more stages can be classified. Drug manipulations might evoke still further states (my paper on this will be submitted immediately when this present methods paper is published and can be cited; I will link it to GitHub when published). Similarly, genetic modifications might evoke different sleep stages, like cataplexy in narcolepsy. We did not want to go into all these details in the paper because we felt that it would lead way too far but I added a few sentences in the manuscript clarifying these points a little more. Anyways, an important point that you might have missed is that the method allows for the classification of any number of states! It only depends on the training set: if an exotic state (like cataplexy) is first visually identified, then manually scored, then it will be trained and the algorithm will predict those scores as well. I added a comment on this in the paper. Also, I added references to the fourth sentence before "Brief Software Description" section to clarify the transition / state question.

We hope these modifications address Reviewers' comments and suggestions, however, let us know if further clarifications are needed.

Best regards, Tamás

oliviaguest commented 4 years ago

@Emma-k-ward if you get a moment to check and tick off the final few boxes that would be amazing. 😊

trisbek commented 4 years ago

I am OK and happy with the responses. I hope the tool is discovered, recognised and used by animal sleep studies and associated questions. I hope that some of the code and criteria also matches human studies in the future as it would be useful for translational purposes. best to all!

El lun., 27 jul. 2020 a las 15:51, Olivia Guest (notifications@github.com) escribió:

@Emma-k-ward https://github.com/Emma-k-ward if you get a moment to check and tick off the final few boxes that would be amazing. 😊

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/2377#issuecomment-664443736, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQCEF2GBFEILSU4MX6IPFG3R5WH6VANCNFSM4OETIXNQ .

oliviaguest commented 4 years ago

@Emma-k-ward, have you had a chance to check the above? 😊

teamPSG commented 4 years ago

Thank you very much @trisbek! And thanks @oliviaguest for keeping an eye on things!

teamPSG commented 4 years ago

@oliviaguest in the meanwhile: is there a solution for how to properly format "Current affiliation" of authors (pls. see comments from 11 days ago)?

oliviaguest commented 4 years ago

@teamPSG ah, sorry, I must have thought I replied but did not — thanks for reminding me. AFAIK what you have done is totally fine. 😺

teamPSG commented 4 years ago

Thanks @oliviaguest then I'll go with the solution I have for current affiliation.

@Emma-k-ward please let me know if I can do anything further to help you proceed with the review process. Looking back at the comments I submitted my responses almost 3 weeks ago...

oliviaguest commented 4 years ago

@teamPSG if @Emma-k-ward doesn't get back to us by next week, I'll email her. Thank you for your patience.

oliviaguest commented 4 years ago

@whedon remind @Emma-k-ward in 1 week

whedon commented 4 years ago

Reminder set for @Emma-k-ward in 1 week

teamPSG commented 4 years ago

Thank you very much @oliviaguest! I was actually planning to ask you tomorrow is there is any deadline or targeted maximal length of a review. Also, what happens if a reviewer becomes silent during a review (anything might happen these days :mask:) -- like, can you make an editorial decision on publishing | rejecting a paper if one the reviewers doesn't finalize their review for a long time? But OK, let's wait for @Emma-k-ward for one more week and then I'll ask you the above questions :innocent:

whedon commented 4 years ago

:wave: @Emma-k-ward, please update us on how your review is going.

teamPSG commented 4 years ago

Hello @oliviaguest! I'd return to my questions from last week: I would like to ask you to please consider making a decision about acceptance of our paper as editor (or as board of editors). I am asking this because I feel the review of our paper got stuck. A brief summary of where we are: I submitted the paper on May 20 (https://joss.theoj.org/papers/5726bdd4b144467c8093e90df0169c25), the review began June 22, I finished replying to comments July 24 and by today three of our four reviewers fully endorsed the paper, we are still waiting for an update from @Emma-k-ward who also mostly accepted our work. I am particularly asking for an editorial decision on our paper because the comments and suggestions only required minor modifications and additions that we have all completed more than a month ago. Thank you very much for considering my request.

Emma-k-ward commented 4 years ago

Hi everyone, sorry I disappeared, I moved to a new country and transitioned from PhD to Postdoc and it was a lot.

I am happy with the changes and think the paper looks good! I will tick the remaining boxes now.

Best wishes, Emma

teamPSG commented 4 years ago

Hi @Emma-k-ward! Glad you're back and thanks for finishing the review.

Hope you'll enjoy living in a new country and you can stay safe there. Good luck with your Postdoc! :woman_scientist:

Best, Tamás

oliviaguest commented 4 years ago

@whedon generate pdf

whedon commented 4 years ago

:point_right: Check article proof :page_facing_up: :point_left:

oliviaguest commented 4 years ago

@teamPSG can you please deposit the code in an appropriate place like zenodo or figshare and then post the DOI here?

teamPSG commented 4 years ago

Thank you @oliviaguest! The DOI for Zenodo is: 10.5281/zenodo.4009178 :rocket:

teamPSG commented 4 years ago

Hello @oliviaguest! Will this Zenodo archive work for the JOSS publication: https://zenodo.org/record/4009178#.X09YJ4uxX0O or with DOI https://doi.org/10.5281/zenodo.4009178

Thanks, Tamás

oliviaguest commented 4 years ago

@whedon set 10.5281/zenodo.4009178 as archive

whedon commented 4 years ago

OK. 10.5281/zenodo.4009178 is the archive.

oliviaguest commented 4 years ago

@whedon check references

whedon commented 4 years ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.5664/jcsm.6576 is OK
- 10.1371/journal.pcbi.1006968 is OK

MISSING DOIs

- None

INVALID DOIs

- https://doi.org/10.1016/S0165-0270(99)00027-8 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1053/smrv.1999.0087 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1016/j.cmpb.2016.12.004 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1016/j.jneumeth.2009.08.014 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1016/j.jneumeth.2014.07.018 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.3389/fnins.2019.00207 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1016/j.jneumeth.2020.108668 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1111/jsr.12991 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1007/s11325-019-02008-w is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1016/j.smrv.2019.07.007 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1016/j.cmpb.2019.04.032 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.3389/fphar.2018.00035 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1371/journal.pone.0224642 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1016/j.biopsych.2012.05.014 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.3109/0954898x.2013.880003 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1016/j.neuropharm.2013.06.020 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1016/j.cmpb.2020.105448 is INVALID because of 'https://doi.org/' prefix

oliviaguest commented 4 years ago

@teamPSG can you fix these DOIs above, please?

teamPSG commented 4 years ago

@whedon check references

whedon commented 4 years ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.5664/jcsm.6576 is OK
- 10.1016/S0165-0270(99)00027-8 is OK
- 10.1053/smrv.1999.0087 is OK
- 10.1016/j.cmpb.2016.12.004 is OK
- 10.1371/journal.pcbi.1006968 is OK
- 10.1016/j.jneumeth.2009.08.014 is OK
- 10.1016/j.jneumeth.2014.07.018 is OK
- 10.3389/fnins.2019.00207 is OK
- 10.1016/j.jneumeth.2020.108668 is OK
- 10.1111/jsr.12991 is OK
- 10.1007/s11325-019-02008-w is OK
- 10.1016/j.smrv.2019.07.007 is OK
- 10.1016/j.cmpb.2019.04.032 is OK
- 10.3389/fphar.2018.00035 is OK
- 10.1371/journal.pone.0224642 is OK
- 10.1016/j.biopsych.2012.05.014 is OK
- 10.3109/0954898x.2013.880003 is OK
- 10.1016/j.neuropharm.2013.06.020 is OK
- 10.1016/j.cmpb.2020.105448 is OK

MISSING DOIs

- None

INVALID DOIs

- None

teamPSG commented 4 years ago

@whedon generate pdf

whedon commented 4 years ago

:point_right: Check article proof :page_facing_up: :point_left:

teamPSG commented 4 years ago

@oliviaguest: DOIs fixed, proof looks good to me. Thanks.

oliviaguest commented 4 years ago

@whedon accept

whedon commented 4 years ago

Attempting dry run of processing paper acceptance...

whedon commented 4 years ago

:wave: @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof :point_right: https://github.com/openjournals/joss-papers/pull/1706

If the paper PDF and Crossref deposit XML look good in https://github.com/openjournals/joss-papers/pull/1706, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

teamPSG commented 4 years ago

Thanks @oliviaguest, both files look good to me.

oliviaguest commented 4 years ago

Cool! Somebody from @openjournals/joss-eics will take over this final step — so just sit tight. 😊

teamPSG commented 4 years ago

Great! Thanks for the adventure :volcano:

kthyng commented 4 years ago

@teamPSG Hi! I will take over from here. Can you verify the version number for your software? I also encourage you to tag a release to mark this occasion if you want.

kthyng commented 4 years ago

@teamPSG Your paper looks great!

teamPSG commented 4 years ago

Hello @kthyng! Thanks for guiding us through the last steps :world_map:

Version number is v1.0 and I created a release with this tag here: https://github.com/teamPSG/kNN_Sleep_Scorer_kNNSS/releases/tag/v1.0

kthyng commented 4 years ago

Ok, looks good.

kthyng commented 4 years ago

@whedon accept deposit=true

whedon commented 4 years ago

Doing it live! Attempting automated processing of paper acceptance...

openjournals / joss-reviews