[REVIEW]: metrica: an R package to evaluate prediction performance of regression and classification point-forecast models

editorialbot commented 2 years ago

Submitting author: !--author-handle-->@adriancorrendo@osorensen<!--end-editor-- Reviewers: @wiljnich, @simonpcouch Archive: 10.5281/zenodo.7291776

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/c3113e0e3fe8c8f9f49d43fde5f4125f"><img src="https://joss.theoj.org/papers/c3113e0e3fe8c8f9f49d43fde5f4125f/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/c3113e0e3fe8c8f9f49d43fde5f4125f/status.svg)](https://joss.theoj.org/papers/c3113e0e3fe8c8f9f49d43fde5f4125f)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@neerajdhanraj & @kauedesousa, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @osorensen know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @wiljnich

📝 Checklist for @simonpcouch

editorialbot commented 2 years ago

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

editorialbot commented 2 years ago

Software report:

github.com/AlDanial/cloc v 1.88  T=0.10 s (1042.4 files/s, 113261.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
R                               88            803           3699           3479
Markdown                         5            267              0           1040
TeX                              1             51              0            465
Rmd                              6            401            759            379
YAML                             6             33             17            124
-------------------------------------------------------------------------------
SUM:                           106           1555           4475           5487
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository

editorialbot commented 2 years ago

Wordcount for paper.md is 1614

editorialbot commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

editorialbot commented 2 years ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.agsy.2021.103194 is OK
- 10.21105/joss.01686 is OK
- 10.5281/zenodo.839854 is OK
- 10.21105/joss.03139 is OK
- 10.1111/j.1740-9713.2018.01169.x is OK
- 10.3325/cmj.2020.61.66 is OK
- 10.1016/B978-0-12-811756-9.00009-5 is OK
- 10.1016/j.agsy.2005.11.004 is OK
- 10.1016/j.envsoft.2018.02.002 is OK
- 10.1016/j.envsoft.2014.07.009 is OK
- 10.1371/journal.pone.0269047 is OK
- 10.5281/zenodo.4618017 is OK
- 10.21105/joss.01903 is OK
- 10.2134/agronj2000.922345x is OK
- 10.1017/S1464793106007007 is OK
- 10.1038/srep19401 is OK
- 10.1126/science.1205438 is OK
- 10.1214/009053607000000505 is OK
- 10.1016/0304-3800(93)E0074-D is OK
- 10.1016/0022-1694(70)90255-6 is OK
- 10.1016/j.agsy.2014.01.008 is OK
- 10.1080/02723646.1981.10642213 is OK
- 10.1029/JC090iC05p08995 is OK
- 10.1002/joc.2419 is OK
- 10.1029/1998WR900018 is OK
- 10.5194/adgeo-5-89-2005 is OK
- 10.1016/j.jhydrol.2012.01.011 is OK
- 10.2307/2088760 is OK
- 10.2307/2089382 is OK
- 10.1016/j.patrec.2008.08.010 is OK
- 10.1016/j.engappai.2007.01.001 is OK

MISSING DOIs

- None

INVALID DOIs

- doi:10.14358/PERS.72.7.823 is INVALID (failed connection)

adriancorrendo commented 2 years ago

Thank you! I just updated the branch containing the paper https://github.com/adriancorrendo/metrica/tree/JOSS , fixing the INVALID DOI within the "paper.bib" under the following commit: https://github.com/adriancorrendo/metrica/commit/1a71b1cd1ee96a9cb7610725861e5bc8d52a9bed

Best regards,

ADRIAN

osorensen commented 2 years ago

@editorialbot generate pdf

editorialbot commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

osorensen commented 2 years ago

👋 @neerajdhanraj, @kauedesousa, could you please update us on how it's going with your reviews? You're welcome to reach out to me if you have any questions.

osorensen commented 2 years ago

👋 @neerajdhanraj, @kauedesousa, could you please update us on how it's going with your reviews? You're welcome to reach out to me if you have any questions.

osorensen commented 2 years ago

👋 @neerajdhanraj, @kauedesousa, could you please update us on how it's going with your reviews?

kauedesousa commented 2 years ago

Hi @osorensen I am so sorry for the long silence. I was in the middle of several duty travels. I will deliver my report in 1 week. Is that ok? Best regards

osorensen commented 2 years ago

Thanks @kauedesousa, that sounds great 👍

osorensen commented 2 years ago

👋 @neerajdhanraj, @kauedesousa, could you please update us on how it's going with your reviews?

osorensen commented 2 years ago

@adriancorrendo, for your information: since I'm not able to get in touch with the reviewers, I'll try to find new reviewers for this submission. Sorry for the delay.

osorensen commented 2 years ago

👋 @wiljnich @simonpcouch @Emma-k-ward, would any of you be willing to review this submission for JOSS? We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/review_criteria.html

wiljnich commented 2 years ago

@osorensen yes, I am able and willing to review this submission.

adriancorrendo commented 2 years ago

@adriancorrendo, for your information: since I'm not able to get in touch with the reviewers, I'll try to find new reviewers for this submission. Sorry for the delay.

Thank you very much, Dr. Sorensen, No worries at all. Finding good and pragmatic reviewers it's a big deal everywhere. I sincerely appreciate your proactiveness to find a solution.

Best regards,

ADRIAN

osorensen commented 2 years ago

@editorialbot add @wiljnich as reviewer

editorialbot commented 2 years ago

@wiljnich added to the reviewers list!

osorensen commented 2 years ago

Perfect @wiljnich! You can get started by issuing the command @editorialbot generate my checklist.

wiljnich commented 2 years ago

Review checklist for @wiljnich

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the https://github.com/adriancorrendo/metrica/?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@adriancorrendo) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
[x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
[x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
[x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

simonpcouch commented 2 years ago

@osorensen I'd be happy to review this submission!👍

osorensen commented 2 years ago

@editorialbot add @simonpcouch as reviewer

editorialbot commented 2 years ago

@simonpcouch added to the reviewers list!

osorensen commented 2 years ago

Thanks @simonpcouch :-) You can get started by issuing the command @editorialbot generate my checklist.

simonpcouch commented 2 years ago

Review checklist for @simonpcouch

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the https://github.com/adriancorrendo/metrica/?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@adriancorrendo) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
[x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
[x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
[x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

simonpcouch commented 2 years ago

@editorialbot generate pdf

editorialbot commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

simonpcouch commented 2 years ago

I've left some review comments with minor revisions for the unchecked boxes in my checklist!

One comment I'll bring up here rather than on the repository issue tracker.

Contribution and authorship: Has the submitting author (@adriancorrendo) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

The submitting author has surely made major contributions to the software! The extent of contributions from the remaining six authors are not self-evident given the GitHub commit history, though I recognize that indeed "authorship is a complex topic with different practices in different communities."

wiljnich commented 2 years ago

@adriancorrendo I've completed my initial review and request only minor revisions, all of which I feel are sufficiently addressed by @simonpcouch's excellent and thorough issue #28. Thank you for your hard work and forthcoming updates.

osorensen commented 2 years ago

@adriancorrendo, could you please give us an update on how long you expect it will take to address the issues raised by the reviewers? It's absolutely fine if things take time, but it would be nice to have an estimate.

adriancorrendo commented 2 years ago

@adriancorrendo, could you please give us an update on how long you expect it will take to address the issues raised by the reviewers? It's absolutely fine if things take time, but it would be nice to have an estimate.

Good morning, Dr. Sorensen!

How are you?

Thank you for reaching out. My apologies for the delay. Actually, I am planning to submit the response today by COB. It just took me longer than expected to work on the changes to paper, updating the code and documentation (latest commit to the JOSS-branch here), and putting things in common with the coauthors.

I also have a quick question regarding the response: most of reviews are provided on this issue by @simonpcouch .

But where I am supposed to provide the point by point response: i) here, ii) on the same issue, or iii) create a new issue with the answers?

Thank you very much for your time and patience!

Best regards, ADRIAN

osorensen commented 2 years ago

@adriancorrendo, please respond in the issue in the source repository https://github.com/adriancorrendo/metrica/issues/28

The additional comment by @simonpcouch can be answered in this thread.

adriancorrendo commented 2 years ago

Dear @simonpcouch and @wiljnich First of all, thank you very much for your compelling revision of the package! It certainly served to improve important aspects of this project.

I will proceed to answer the most of the revisions on the corresponding issue#28 opened by @simonpcouch on the source repository.

Please, you may read below the answer related to the Contribution and authorship:

Contribution and authorship: Has the submitting author (@adriancorrendo) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

The submitting author has surely made major contributions to the software! The extent of contributions from the remaining six authors are not self-evident given the GitHub commit history, though I recognize that indeed "authorship is a complex topic with different practices in different communities."

Answer: Thank you very much for this comment. You are correct with the observation, most of the code has been uploaded by myself (@adriancorrrendo) but it is simply the manner we have found suitable in our case. There has been a considerable amount of discussions between all the coauthors about key concepts, metrics, examples, data, etc. to give the necessary structure to the package. Many times, coauthors worked developing code for specific functions, writing vignettes, etc. However, for our simplicity, we decided to maintain a “centralized” management of the repository. Maybe not the best practice but since we were all relatively new using GitHub, PR, forking repositories, etc., we found this way as reasonable and straightforward for everyone. I hope this helps to clarify how we have managed the authorship for the package.

adriancorrendo commented 2 years ago

@adriancorrendo, please respond in the issue in the source repository adriancorrendo/metrica#28

The additional comment by @simonpcouch can be answered in this thread.

Dear Dr. @osorensen ,

I have just submitted the point-by-point response to revisions at the corresponding issue https://github.com/adriancorrendo/metrica/issues/28.

Please, let me know if any further step is required from my side.

Thank you for your time and patience!

Best regards,

ADRIAN

osorensen commented 2 years ago

Thanks @adriancorrendo, no further steps needed at this point

simonpcouch commented 2 years ago

I've checked off the remaining unchecked boxes for my review, and recommend acceptance of the paper! Thanks for the invitation to review—I really enjoyed spending time with this package.

wiljnich commented 2 years ago

I am now satisfied with this paper and its package, and happily recommend acceptance. Congratulations on your hard work!

osorensen commented 2 years ago

Thank you very much for your reviews @simonpcouch and @wiljnich!

osorensen commented 2 years ago

@adriancorrendo, I will now read through the paper once more, and let you know if I have any suggested changes.

In the meantime could you:

[ ] Make a tagged release of your software, and list the version tag of the archived version here.
[ ] Archive the reviewed software in Zenodo or a similar service (e.g., figshare, an institutional repository)
[ ] Check the archival deposit (e.g., in Zenodo) has the correct metadata. This includes the title (should match the paper title) and author list (make sure the list is correct and people who only made a small fix are not on it). You may also add the authors' ORCID.
[ ] Please list the DOI of the archived version here.

osorensen commented 2 years ago

@editorialbot generate pdf

osorensen commented 2 years ago

@editorialbot check references

editorialbot commented 2 years ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.agsy.2021.103194 is OK
- 10.21105/joss.01686 is OK
- 10.5281/zenodo.839854 is OK
- 10.21105/joss.03139 is OK
- 10.1111/j.1740-9713.2018.01169.x is OK
- 10.3325/cmj.2020.61.66 is OK
- 10.1016/B978-0-12-811756-9.00009-5 is OK
- 10.1016/j.agsy.2005.11.004 is OK
- 10.1016/j.envsoft.2018.02.002 is OK
- 10.1016/j.envsoft.2014.07.009 is OK
- 10.1371/journal.pone.0269047 is OK
- 10.5281/zenodo.4618017 is OK
- 10.21105/joss.01903 is OK
- 10.2134/agronj2000.922345x is OK
- 10.1017/S1464793106007007 is OK
- 10.14358/PERS.72.7.823 is OK
- 10.1038/srep19401 is OK
- 10.1126/science.1205438 is OK
- 10.1214/009053607000000505 is OK
- 10.1016/0304-3800(93)E0074-D is OK
- 10.1016/0022-1694(70)90255-6 is OK
- 10.1016/j.agsy.2014.01.008 is OK
- 10.1080/02723646.1981.10642213 is OK
- 10.1029/JC090iC05p08995 is OK
- 10.1002/joc.2419 is OK
- 10.1029/1998WR900018 is OK
- 10.5194/adgeo-5-89-2005 is OK
- 10.1016/j.jhydrol.2012.01.011 is OK
- 10.2307/2088760 is OK
- 10.2307/2089382 is OK
- 10.1016/j.patrec.2008.08.010 is OK
- 10.1016/j.engappai.2007.01.001 is OK

MISSING DOIs

- None

INVALID DOIs

- None

editorialbot commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

adriancorrendo commented 2 years ago

Dear @osorensen ,

Thank you very much for your copy edits. Paper edits:

Within the latest commit to the JOSS branch, I have made the modifications requested issues 29 to 31.

Release:

New release tag v2.0.1.999
- Question: Please, note the release was made on the "stable" master-branch, not on the temporary JOSS. Is it correct? Or do I need to create the new release using the JOSS-branch (containing the "paper.md" file) as the target??
Zenodo archived version
Archived version DOI: https://doi.org/10.5281/zenodo.7291776

Please, let me know if any further modification is needed from my end.

Thank you for your time!

Best regards,

ADRIAN

osorensen commented 2 years ago

Thanks @adriancorrendo, and to your question, the answer is yes, what you've done is correct.

osorensen commented 2 years ago

@editorialbot generate pdf

editorialbot commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

osorensen commented 2 years ago

@editorialbot set v2.0.1.999 as version

openjournals / joss-reviews