[REVIEW]: xesn: Echo state networks powered by xarray and dask

Submitting author: !--author-handle-->@timothyas@sneakers-the-rat<!--end-editor-- Reviewers: @Arcomano1234, @wiljnich Archive: Pending

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/84a7fddace8425c95603b4f690a69896"><img src="https://joss.theoj.org/papers/84a7fddace8425c95603b4f690a69896/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/84a7fddace8425c95603b4f690a69896/status.svg)](https://joss.theoj.org/papers/84a7fddace8425c95603b4f690a69896)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@Arcomano1234 & @wiljnich, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @sneakers-the-rat know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @wiljnich

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

Software report:

github.com/AlDanial/cloc v 1.90  T=0.06 s (1033.4 files/s, 258542.1 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          24           1054            726           2964
TeX                              1            163              0           1699
Jupyter Notebook                 6              0           6019            840
YAML                            12             50             66            476
reStructuredText                 9            159            146            408
Markdown                         3             67              0            298
TOML                             2              4              0             34
DOS Batch                        1              8              1             26
JSON                             1              1              0             16
make                             1              5              7             11
Bourne Shell                     1              2              8              4
-------------------------------------------------------------------------------
SUM:                            61           1513           6973           6776
-------------------------------------------------------------------------------

Commit count by author:

    42  Timothy Smith
    21  timothyas

Paper file info:

📄 Wordcount for paper.md is 1699

✅ The paper includes a Statement of need section

License info:

🟡 License found: Other (Check here for OSI approval)

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.1029/2020GL087776 is OK
- 10.1029/2020MS002290 is OK
- 10.1029/2021MS002537 is OK
- 10.1002/essoar.10509867.1 is OK
- 10.1103/PhysRevFluids.5.113506 is OK
- 10.1098/rspa.2021.0135 is OK
- 10.1016/j.ocemod.2022.102023 is OK
- 10.1175/AIES-D-21-0002.1 is OK
- 10.1146/annurev-fluid-010719-060214 is OK
- 10.1017/eds.2022.10 is OK
- 10.1029/2021MS002502 is OK
- 10.1029/2020MS002109 is OK
- 10.5194/gmd-12-2797-2019 is OK
- 10.1029/2019MS001705 is OK
- 10.1029/2020MS002405 is OK
- 10.1002/qj.4180 is OK
- 10.5194/npg-28-423-2021 is OK
- 10.5194/gmd-15-3433-2022 is OK
- 10.1029/2021MS002554 is OK
- 10.1098/rsta.2020.0097 is OK
- 10.1029/2018GL080704 is OK
- 10.1002/qj.3803 is OK
- 10.5194/gmd-9-3655-2016 is OK
- 10.1007/978-3-030-96709-3 is OK
- 10.1137/070694855 is OK
- 10.1146/annurev.fluid.010908.165248 is OK
- 10.1016/j.jcp.2012.08.013 is OK
- 10.1029/JD093iD09p11015 is OK
- 10.1175/1520-0493(1989)117<2165:RFAFUP>2.0.CO;2 is OK
- 10.1016/j.neunet.2022.06.025 is OK
- 10.1103/PhysRevLett.120.024102 is OK
- 10.1063/1.5039508 is OK
- 10.1063/1.4979665 is OK
- 10.1038/s41467-021-25801-2 is OK
- 10.1063/5.0024890 is OK
- 10.1063/1.5120710 is OK
- 10.1016/j.neunet.2007.04.003 is OK
- 10.1016/j.neunet.2012.07.005 is OK
- 10.1063/1.5120733 is OK
- 10.1063/5.0066013 is OK
- 10.1103/PhysRevE.99.042203 is OK
- 10.1016/j.neunet.2020.02.016 is OK
- 10.1029/2021MS002843 is OK
- 10.1038/s41592-019-0686-2 is OK
- 10.1063/1.5010300 is OK
- 10.1162/089976602760407955 is OK
- 10.1109/IJCNN.2004.1380039 is OK
- 10.48550/arXiv.2201.05193 is OK
- 10.1175/2008JAS2921.1 is OK
- 10.1175/1520-0469(1978)035<0774:UPVFPI>2.0.CO;2 is OK
- 10.1175/1520-0469(1978)035<0784:UPVFPI>2.0.CO;2 is OK
- 10.1111/j.2153-3490.1949.tb01265.x is OK
- 10.1023/A:1008306431147 is OK
- 10.1016/j.advengsoft.2019.03.005 is OK
- 10.1007/3-540-07165-2_55 is OK
- 10.1007/978-3-642-35289-8_36 is OK
- 10.1109/IJCNN.2010.5596884 is OK
- 10.48550/arXiv.2212.12794 is OK
- 10.48550/arXiv.2210.12504 is OK
- 10.1109/ISDA.2011.6121637 is OK
- 10.1103/PhysRevE.106.055303 is OK
- 10.1063/5.0071926 is OK
- 10.1016/j.ins.2019.09.049 is OK
- 10.1088/2634-4386/ac1b75 is OK
- 10.1016/j.neunet.2018.08.002 is OK
- 10.1016/j.neucom.2016.12.089 is OK
- 10.1109/TCYB.2016.2533545 is OK
- 10.1007/978-3-540-77655-0_16 is OK
- 10.1017/S0022112095000012 is OK
- 10.2514/1.J057129 is OK
- 10.1007/s00158-020-02488-5 is OK
- 10.48550/arXiv.2304.12865 is OK
- 10.5281/zenodo.8368225 is OK
- 10.1038/s41586-023-06185-3 is OK
- 10.1029/2023MS003792 is OK
- 10.5334/jors.148 is OK
- 10.1126/science.1091277 is OK
- 10.1038/s41586-020-2649-2 is OK
- 10.1007/978-3-030-61616-8_40 is OK
- 10.1029/2021MS002712 is OK
- 10.1029/2022GL102649 is OK
- 10.5281/zenodo.7508156 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: Forecasting Global Weather with Graph Neural Netwo...
- No DOI given, and none found for title: FourCastNet: A Global Data-driven High-resolution ...
- No DOI given, and none found for title: Convolutional LSTM Network: A Machine Learning App...
- No DOI given, and none found for title: Coupled Data Assimilation for Integrated Earth Sys...
- No DOI given, and none found for title: Statistics for spatial data
- No DOI given, and none found for title: Sequence Modeling: Recurrent and Recursive Nets
- No DOI given, and none found for title: The "echo state” approach to analysing and trainin...
- No DOI given, and none found for title: Solution of Incorrectly Formulated Problems and th...
- No DOI given, and none found for title: A Catch-22 of Reservoir Computing
- No DOI given, and none found for title: Dask: Library for dynamic task scheduling
- No DOI given, and none found for title: Pangu-Weather: A 3D High-Resolution Model for Fast...
- No DOI given, and none found for title: An Image is Worth 16x16 Words: Transformers for Im...
- No DOI given, and none found for title: Attention is All you Need
- No DOI given, and none found for title: Investigation of Proper Orthogonal Decomposition f...
- No DOI given, and none found for title: CuPy: A NumPy-Compatible Library for NVIDIA GPU Ca...
- No DOI given, and none found for title: HYCOM + NCODA Gulf of Mexico 1/25° Reanalysis, (GO...
- No DOI given, and none found for title: A Rational Subdivision of Scales for Atmospheric P...
- No DOI given, and none found for title: Long-term instabilities of deep learning-based dig...
- No DOI given, and none found for title: Implicit Neural Representations with Periodic Acti...
- No DOI given, and none found for title: Making a Science of Model Search: Hyperparameter O...
- No DOI given, and none found for title: Tune: A Research Platform for Distributed Model Se...
- No DOI given, and none found for title: JAX: composable transformations of Python+NumPy pr...

❌ MISSING DOIs

- 10.1007/s13137-022-00210-9 may be a valid DOI for title: Predicting Shallow Water Dynamics using Echo-State...
- 10.31223/osf.io/cqmb2 may be a valid DOI for title: Deep spatial transformers for autoregressive data-...
- 10.1145/3592979.3593412 may be a valid DOI for title: FourCastNet: Accelerating Global High-Resolution W...
- 10.1063/5.0098707 may be a valid DOI for title: Learning Spatiotemporal Chaos Using Next-Generatio...
- 10.1017/cbo9780511617652.004 may be a valid DOI for title: Predictability - a problem partly solved
- 10.1007/s42967-024-00398-7 may be a valid DOI for title: Overview frequency principle/spectral bias in deep...
- 10.1029/2023ms003681 may be a valid DOI for title: Generative data-driven approaches for stochastic s...
- 10.1017/cbo9780511617652.008 may be a valid DOI for title: Ensemble forecasting and data assimilation: two pr...

❌ INVALID DOIs

- https://doi.org/10.5194/gmd-11-3999-2018 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1029/2020MS002203 is INVALID because of 'https://doi.org/' prefix

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

Hello everyone! Thanks for your patience with the pre-review, and thanks to our reviewers @Arcomano1234 and @wiljnich for agreeing to review!

Here's my usual pre-review spiel:

As the top issue says, you'll be working from a checklist, a set of guidelines, and some review criteria.

Goals of Review

The goal here is to collaboratively help this package reach a basic standard for usability, maintainability, and correctness - we are all friends here, there are no gates to be kept, we're just trying to learn from and help each other out. Feel free to talk amongst each other as peers in the issues related to this review. This is an open and iterative review, so you don't need to wait until you've reviewed everything before raising an issue or commenting - in fact its preferable to raise a comment early so that the authors have time to respond and make any requested changes while you're completing the rest of your review.

The checklist is a set of minimum standards that the package has to meet, but you are also free to read, comment, raise issues and pull requests, and do whatever else falls under your purview as a reviewer. Think about what you would want to have if you were a potential new user of the package - is the documentation clear? do the tests cover what they need to? etc. That kind of neutral third-party perspective is invaluable for making software accessible and maintainable.

Process

As you review, please

Raise issues in the repository and
link back to this issue so that we can track issues related to this review.

You might also want to link to issues from your checklist comment to keep track of what is blocking each check.

Since github issues don't have threading, this is how we can keep discussion organized, because commenting in this issue can be a bit messy. Don't be shy commenting here to ask questions about the review itself, though!

It's up to you all and the maintainers if you want to follow any sort of naming convention for issues/PRs - authors please comment here about any preferences you have for raised issues. I would recommend at least tagging issues by general topic areas - [docs], [tests], [bug], etc. and also if the reviewers would indicate if a review is a [question], [suggestion] or a [blocker] for the review so the authors know if a given issue needs some fix in order for the review to be completed.

As a reviewer, it's up to you to decide how detailed you'd like your review to be - completing the checklist without further comment is totally acceptable, but I would also encourage reviewers to raise issues as described above and include some closing ~1-few paragraph summary that highlights the strengths and opportunities for future development in the package.

Other than that, i'll be here so feel free to ask questions! The first thing you'll want to do is generate your checklists using the commands in the OP comment. have fun everyone <3

Thanks @sneakers-the-rat! I don't have any preferences regarding issue naming conventions. Also, I just want to let everyone know that I'll be out on leave next week, so I may be slow to respond. Additionally, I'm a federal employee at NOAA, and so if the government does shutdown, then I'm legally forbidden from working. So in such an event, I will have to wait until the government opens back up to reply. All that said, please know that I'm very interested in working with you all on this review.

Finally, I updated the paper draft to address DOI issues in this PR. A lot of the citations in the references.bib file are not necessary for the paper, so many don't matter. But I added the ones I could find that are relevant.

I updated the paper draft to address DOI issues in https://github.com/timothyas/xesn/pull/77.

Thanks for this, i was going to say something when i saw the DOI report - I will double check with EiC but i believe only the citations that are used in the paper need to have valid DOIs, it looks like you might be using that bibliography in your docs as well, which is totally fine (and imo a good practice to single source them).

Please take your time. I think ideally we don't like to let reviews drag on too long mostly so that they are finished in a reasonable amount of time for authors and to not keep reviewers on the line indefinitely, but if anyone needs extra time, this is all volunteer labor so i certainly won't be upset - key thing to me is just communication (which i modeled poorly at the beginning, and you are modeling excellently now) <3

Review checklist for @wiljnich

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[ ] Repository: Is the source code for this software available at the https://github.com/timothyas/xesn?
[ ] License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
[ ] Contribution and authorship: Has the submitting author (@timothyas) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[ ] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
[ ] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
[ ] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
[ ] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

[ ] Installation: Does installation proceed as outlined in the documentation?
[ ] Functionality: Have the functional claims of the software been confirmed?
[ ] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[ ] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[ ] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[ ] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[ ] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[ ] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[ ] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
[ ] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[ ] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[ ] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

openjournals / joss-reviews