editorialbot commented 1 year ago

Submitting author: !--author-handle-->@matsvanes@crsl4<!--end-editor-- Reviewers: @gflofst, @ashahide Archive: 10.5281/zenodo.10495308

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/7f09b30f96082cab883d4dad60195e97"><img src="https://joss.theoj.org/papers/7f09b30f96082cab883d4dad60195e97/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/7f09b30f96082cab883d4dad60195e97/status.svg)](https://joss.theoj.org/papers/7f09b30f96082cab883d4dad60195e97)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@gflofst & @ashahide, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @crsl4 know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @gflofst

📝 Checklist for @ashahide

editorialbot commented 1 year ago

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

editorialbot commented 1 year ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1093/nar/gky379 is OK
- 10.3389/fnins.2018.00261 is OK
- 10.3389/fninf.2012.00007 is OK
- 10.1038/nrn3475 is OK
- 10.1111/nyas.13325 is OK
- 10.1016/j.neuron.2017.10.013 is OK
- 10.3389/fninf.2011.00013 is OK
- 10.1093/bioinformatics/bth361 is OK
- 10.1155/2011/156869 is OK
- 10.1126/science.aac4716 is OK
- 10.1016/j.neuroimage.2019.06.046 is OK
- 10.1016/S1053-8119(03)00185-X is OK
- 10.1177/0956797611417632 is OK
- 10.1371/journal.pbio.2000797 is OK
- 10.1017/S0140525X17001972 is OK

MISSING DOIs

- 10.1007/3-211-27183-x_1 may be a valid DOI for title: MATLAB

INVALID DOIs

- None

editorialbot commented 1 year ago

Software report:

github.com/AlDanial/cloc v 1.88  T=17.42 s (418.7 files/s, 69514.6 lines/s)
---------------------------------------------------------------------------------------
Language                             files          blank        comment           code
---------------------------------------------------------------------------------------
MATLAB                                5647          95785         230136         480408
C                                      829          33375          46894         162300
C++                                    109           6363           6643          25657
C/C++ Header                           283           8799          18127          22648
make                                   115           2891           1412           9397
TeX                                     19           1676            623           9097
Fortran 77                              36           1743           3894           7722
Java                                    39           1069           1661           5194
HTML                                    40            598           1060           5020
Bourne Shell                            20            961           1044           2710
Windows Module Definition                8            122              0           2437
Arduino Sketch                           7            536            904           2303
Markdown                                27            499              0           1150
XML                                     14             18             43           1005
TypeScript                               1              0              0           1001
C Shell                                 29            148            127            784
Expect                                   6             17              0            783
Ruby                                     4              0              0            603
Python                                   4            173             94            532
MSBuild script                           2              0              0            455
Mathematica                              3              9              0            220
Perl                                     1             40             81            220
XSLT                                     1             47             21            217
sed                                      5              8              0            164
ProGuard                                 7             50             25            141
CMake                                    3             58             16            107
Ant                                      2             16              2            100
Scheme                                   1              0              0             78
CSS                                      1             14              2             66
INI                                      9              6              5             59
Bourne Again Shell                       3              8              0             40
DOS Batch                                3             11              1             39
reStructuredText                         1             18              0             33
awk                                      6             15              0             29
Qt Project                               2              6              0             18
MUMPS                                    1              4              0             14
YAML                                     2              0              0             11
Solidity                                 2              0              0             10
---------------------------------------------------------------------------------------
SUM:                                  7292         155083         312815         742772
---------------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository

editorialbot commented 1 year ago

Wordcount for paper.md is 1579

editorialbot commented 1 year ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

gflofst commented 1 year ago

Review checklist for @gflofst

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the https://github.com/fieldtrip/fieldtrip?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@matsvanes) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
[x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
[x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
[x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

ashahide commented 1 year ago

Review checklist for @ashahide

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the https://github.com/fieldtrip/fieldtrip?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@matsvanes) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
[x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
[x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
[x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

gflofst commented 1 year ago

@matsvanes does not appear to have made any contributions to this code base. Am I looking at this wrong?

crsl4 commented 1 year ago

You seem to be right! Not sure of the expectations for first authorship, but let me check!

crsl4 commented 1 year ago

Hi @matsvanes, in response to @gflofst 's comment, do you mind sharing your author's contributions to FieldTrip? Thanks!

matsvanes commented 1 year ago

@crsl4 Thanks for reaching out. I can see how my contributions are not clear from FieldTrip's github directly. I hope the following clears things up. Unfortunately, much of the work submitted here has somewhat dispersed origins. My contributions were mostly in design, testing, and documentation, and haven't ended up in the main branch of the FieldTrip toolbox directly. See below for a trail of my involvement in the various stages.

Design The toolbox addition was inspired by this early implementation.

Development and testing Mostly through comments on issues and pulls on the FieldTrip Github, for example: #1617, #1362, #1330, #1317, #973

Documentation A trail of developing the tests and documentation is found here, and their implementation on the FieldTrip website here.

gflofst commented 1 year ago

@matsvanes JOSS will have to judge if this is sufficient. I don't have adequate guidance except to see if "significant" effort was involved on the submitter's part. If you have any design notes or otherwise that can be added to this thread, that would make it more easily pass the bar. I appreciate tests and documentation rather than just code. I would certainly count those. Similar software is hard to judge attributions without something explicit.

matsvanes commented 1 year ago

Thank you for the suggestion @gflofst. Of importance would definitely be the two links above under Documentation. In case it's relevant, we also have a pre-print available.

matsvanes commented 1 year ago

Hi @crsl4 - is there any update on the review progress? Thanks!

crsl4 commented 1 year ago

Thanks @matsvanes for your message! I am still waiting to hear back from reviewers. The pre-print opens up a new set of questions. Can you explain the overlap? Is that preprint a journal submission?

gflofst commented 1 year ago

@crsl4, We have a question as to whether the submission meets the required significant effort bar. In the submitted materials, the author is not in the code repo at all, but was responsible for writing some of the associated papers, provided after the fact. If this is sufficient, I think we need to have the repo updated with links/copies of the additional materials to be able to check that review criteria as satisfied. I am waiting for your guidance on whether what is in the repo is sufficient, if the additional materials existing are sufficient, if the existing materials need to be added/linked in the repo to be sufficient, or if it is insufficient given the lack of code contributions.

ashahide commented 1 year ago

@crsl4, I was wondering the same thing about the preprint. Should the review comments be focused only on the material in the JOSS article, or should we also consider the additional preprint?

Thanks for your help.

crsl4 commented 1 year ago

@ashahide We are supposed to only review the JOSS article, but we should understand what is the gap to the pre-print, and whether the pre-print has significant overlap (bc then, why do we have the JOSS article?).

crsl4 commented 1 year ago

@gflofst Thanks for your message! I think I want to discuss this with other editors. I am not sure what the standard expectations are compared to other articles. I will reach back, thanks!

ashahide commented 1 year ago

Hi @matsvanes , is there a list of the required Matlab toolboxes somewhere that I may be missing? I see that there is an external folder for additional non-Matlab dependencies and that some of the tests check for necessary toolboxes, but I'm unable to find a list in the documentation.

matsvanes commented 1 year ago

Hi @matsvanes , is there a list of the required Matlab toolboxes somewhere that I may be missing? I see that there is an external folder for additional non-Matlab dependencies and that some of the tests check for necessary toolboxes, but I'm unable to find a list in the documentation.

Hi @ashahide, the list of external toolboxes can be found here. Most of these are supplied alongside FieldTrip (i.e., in /external), and are added to the path when running ft_defaults (see here for more info on installation)

matsvanes commented 1 year ago

@crsl4, We have a question as to whether the submission meets the required significant effort bar. In the submitted materials, the author is not in the code repo at all, but was responsible for writing some of the associated papers, provided after the fact. If this is sufficient, I think we need to have the repo updated with links/copies of the additional materials to be able to check that review criteria as satisfied. I am waiting for your guidance on whether what is in the repo is sufficient, if the additional materials existing are sufficient, if the existing materials need to be added/linked in the repo to be sufficient, or if it is insufficient given the lack of code contributions.

I hope I can clarify these questions with some historic perspective/context. This project was part of my PhD - a time when I unfortunately was rather new to git(hub). Because of this, my FieldTrip branch was messy, and I used the quick route through my collaborators (co-authors on this paper) to get the code on the master branch. As mentioned before, I have also done most of the testing and documentation of the code. Regarding the pre-print: this is essentially one chapter out of my PhD thesis. We were interested in submitting to JOSS because we believe its aims align well with our paper, and we wanted to change the narrative from a tutorial-style opinion/recommendation paper to put the focus on the software implementation itself (and get credit for it). Consequently, after opening a pre-submission enquiry, we adapted the pre-print to the JOSS format, and moved the tutorials to the website documentation: 1, 2, 3.

robertoostenveld commented 1 year ago

@matsvanes does not appear to have made any contributions to this code base. Am I looking at this wrong?

Let me as the head of the FieldTrip project and senior author on this submission confirm Mats' comment: Mats did indeed make significant contributions to the conceptual design, the implementation, the testing, and the the documentation (initially in the form of a manuscript that was rejected elsewhere, later also on the website). He fully deserves credit for this as first author.

My own involvement was mainly to supervise the process, to ensure consistency and future maintainability, and to help with the technical aspects of the coding and merging process. Since we are neuroscientists and neither of us are git and GitHub experts, I realize that the representation on github is not as ideal as it could have been. We continue to constantly learn how to make use of version control using git, and also how to pass those skills on to students that are working on concrete projects which involve coding (like this reproducescript one).

crsl4 commented 1 year ago

Thanks @robertoostenveld ! I talked with other JOSS editors, and it seems that @matsvanes satisfies the criteria for first authorship.

crsl4 commented 1 year ago

One thing we need to determine is the existing overlap between the pre-print and the JOSS article. @matsvanes Is the pre-print to be published in another journal? If the pre-print will be published elsewhere, then we need to make sure that there is no overlap with the JOSS manuscript. That is, it is ok to use the software in a dataset for another publication, but not for the JOSS manuscript (or the introduction of the software) to be a subset of the pre-print to be published elsewhere.

@ashahide @gflofst Can you add this as part of your review? Thanks!

matsvanes commented 1 year ago

We have no intentions to publish the pre-print in another journal. After releasing the pre-print we changed course and decided to focus more on the software for publication. For this reason, we moved most of the content of the pre-print (e.g., the examples) to the FieldTrip Website. If the software were to be published at JOSS, the pre-print would thus become obsolete, just like it would when a pre-print is finally published in a peer-reviewed journal.

matsvanes commented 1 year ago

Hi @crsl4, Has there been any progress in the review process?

crsl4 commented 1 year ago

Hi! We are waiting for reviewers to finish their review 👍.

gflofst commented 1 year ago

I've had two bad weeks including technology failure. Hopefully this week I'll be able to get it done.

Best,

Jay

From: Claudia Solis-Lemus @.> Sent: Monday, September 18, 2023 10:36:30 AM To: openjournals/joss-reviews @.> Cc: Lofstead, Jay F @.>; Mention @.> Subject: [EXTERNAL] Re: [openjournals/joss-reviews] [REVIEW]: Reducing the efforts to create reproducible analysis code with FieldTrip (Issue #5566)

Hi! We are waiting for reviewers to finish their review 👍.

— Reply to this email directly, view it on GitHubhttps://github.com/openjournals/joss-reviews/issues/5566#issuecomment-1723913550, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAUR4WULPLMAYX725IVQVTLX3B2A5ANCNFSM6AAAAAAZJN6RPQ. You are receiving this because you were mentioned.Message ID: @.***>

crsl4 commented 1 year ago

Thank you, Jay! No worries! We are experiencing delays on several fronts.

ashahide commented 1 year ago

Hi @crsl4, I have completed my review. I think the article meets all of the JOSS criteria, and the work is interesting and addresses an important topic. Let me know if there is anything else I should do on my end. Thanks!

Kevin-Mattheus-Moerman commented 1 year ago

@gflofst thanks again for your help as a reviewer here. Do you think you can pick up the review again at this point? Let us know if you have any updates.

Kevin-Mattheus-Moerman commented 1 year ago

@gflofst :wave:

Kevin-Mattheus-Moerman commented 1 year ago

@crsl4 can you please keep on top of this review and remind the reviewers when needed.

gflofst commented 1 year ago

I'm back in. Reviewing now.

gflofst commented 1 year ago

I can't find installation instructions.

I also do not have a desktop version of MATLAB so I'm trying the free online version. There are no notes as to whether or not that should work.

matsvanes commented 1 year ago

Hi Jay,

The installation instructions can be found here: https://www.fieldtriptoolbox.org/download/ In short, after downloading the toolbox, you should add it to your path with addpath <path_to_fieldtrip>, and then run ft_defaults, which will setup the rest.

For requirements, see https://www.fieldtriptoolbox.org/faq/requirements/. Depending on the fieldtrip functionality you want to use, you may be required to have (licensed) toolboxes installed.

gflofst commented 12 months ago

I'm at SC23 this week. I doubt I'll have time to do this before Saturday.

From: Mats @.> Sent: Thursday, November 9, 2023 7:12 AM To: openjournals/joss-reviews @.> Cc: Lofstead, Jay F @.>; Mention @.> Subject: [EXTERNAL] Re: [openjournals/joss-reviews] [REVIEW]: Reducing the efforts to create reproducible analysis code with FieldTrip (Issue #5566)

Hi Jay,

The installation instructions can be found here: https://www.fieldtriptoolbox.org/download/ In short, after downloadinghttps://www.fieldtriptoolbox.org/download.php the toolbox, you should add it to your path with addpath , and then run ft_defaults, which will setup the rest.

For requirements, see https://www.fieldtriptoolbox.org/faq/requirements/. Depending on the fieldtrip functionality you want to use, you may be required to have (licensed) toolboxes installed.

- Reply to this email directly, view it on GitHubhttps://github.com/openjournals/joss-reviews/issues/5566#issuecomment-1803908443, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAUR4WWYJNBMCOYV2TUBWVLYDTQEHAVCNFSM6AAAAAAZJN6RPSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBTHEYDQNBUGM. You are receiving this because you were mentioned.Message ID: @.**@.>>

gflofst commented 11 months ago

I got a MATLAB license and will be able to test everything. Installing now.

gflofst commented 11 months ago

It looks good and technically meets the requirements. I'd prefer if there was more in the github rather than their own website (e.g., installation instructions), but it is all finable without too much effort. I vote to accept.

robertoostenveld commented 11 months ago

I'd prefer if there was more in the github rather than their own website (e.g., installation instructions), but it is all finable without too much effort. I vote to accept.

Note that the website itself is also on github (separate from the code), including documentation on reproduce script here, here and here. It is just that the rendering as html on our own server looks better and it makes us independent from github/microsoft as commercial service provider.

Having independent installation instructions in the git repository, i.e., also on github, bitbucket and gitlab, separate from those on the website, is a good idea and simple to implement. I will add that to the README file.

matsvanes commented 11 months ago

@crsl4 Are there any updates/conclusions following the reviews?

crsl4 commented 11 months ago

Yes! I feel both reviewers have approved the manuscript and we can move forward. Tagging them in case I missed something: @gflofst @ashahide

So, @Kevin-Mattheus-Moerman , is this the time to do create post-review checklist? Thanks all for your hard work!

Kevin-Mattheus-Moerman commented 10 months ago

@crsl4 yep, please do create that checklist.

crsl4 commented 10 months ago

Post-Review Checklist for Editor and Authors

Additional Author Tasks After Review is Complete

[ ] Double check authors and affiliations (including ORCIDs)
[ ] Make a release of the software with the latest changes from the review and post the version number here. This is the version that will be used in the JOSS paper.
[ ] Archive the release on Zenodo/figshare/etc and post the DOI here.
[ ] Make sure that the title and author list (including ORCIDs) in the archive match those in the JOSS paper.
[ ] Make sure that the license listed for the archive is the same as the software license.

Editor Tasks Prior to Acceptance

[x] Read the text of the paper and offer comments/corrections (as either a list or a PR)
[x] Check the references in the paper for corrections (e.g. capitalization)
[x] Check that the archive title, author list, version tag, and the license are correct
[x] Set archive DOI with @editorialbot set <DOI here> as archive
[x] Set version with @editorialbot set <version here> as version
[x] Double check rendering of paper with @editorialbot generate pdf
[x] Specifically check the references with @editorialbot check references and ask author(s) to update as needed
[x] Recommend acceptance with @editorialbot recommend-accept

matsvanes commented 10 months ago

Hi @crsl4, what is the status on this?

crsl4 commented 10 months ago

Hi @matsvanes ! Working on the post-review checklist for editors. You need to go over the ones for authors (see message above).

matsvanes commented 9 months ago

Post-Review Checklist for Editor and Authors

Additional Author Tasks After Review is Complete

[x] Double check authors and affiliations (including ORCIDs)

[x] Make a release of the software with the latest changes from the review and post the version number here. This is the version that will be used in the JOSS paper. - Version 20231220 (Note that it doesn't appear to be the last version on Zenodo, but that's incorrect and Zenodo is aware of the issue - this version/DOI can be used however).
[x] Archive the release on Zenodo/figshare/etc and post the DOI here. - https://doi.org/10.5281/zenodo.10495308
[x] Make sure that the title and author list (including ORCIDs) in the archive match those in the JOSS paper. - @crsl4 please advise what to do here, since the software is an extension the author list is not the same as in the original. The authors of the original do match the ones in the archive
[x] Make sure that the license listed for the archive is the same as the software license.

crsl4 commented 9 months ago

@editorialbot generate pdf

editorialbot commented 9 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

openjournals / joss-reviews

[REVIEW]: Reducing the efforts to create reproducible analysis code with FieldTrip #5566

Status

Reviewer instructions & questions

Checklists

Review checklist for @gflofst

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @ashahide

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Post-Review Checklist for Editor and Authors

Additional Author Tasks After Review is Complete

Editor Tasks Prior to Acceptance

Post-Review Checklist for Editor and Authors

Additional Author Tasks After Review is Complete