[REVIEW]: scores: A Python package for verifying and evaluating models and predictions with xarray

editorialbot commented 3 months ago

Submitting author: !--author-handle-->@tennlee@crvernon<!--end-editor-- Reviewers: @calebweinreb, @savente93 Archive: 10.5281/zenodo.12697242

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/c51b4a6def315f7b9789983663677468"><img src="https://joss.theoj.org/papers/c51b4a6def315f7b9789983663677468/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/c51b4a6def315f7b9789983663677468/status.svg)](https://joss.theoj.org/papers/c51b4a6def315f7b9789983663677468)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@calebweinreb & @savente93, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @crvernon know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @savente93

📝 Checklist for @calebweinreb

tennlee commented 2 months ago

Fantastic! @crvernon please note, we have (slightly) updated the title of the paper during the review process. I don't know whether that has any implication i.e. if the issue names of the review and pre-review needs to be made consistent to match or anything like that.

crvernon commented 2 months ago

No problem @tennlee, I'll take care of it. I am on travel today but will wrap up my part of this in the morning and provide feedback.

crvernon commented 2 months ago

@editorialbot generate pdf

editorialbot commented 2 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

crvernon commented 2 months ago

@editorialbot check references

editorialbot commented 2 months ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.21105/joss.02781 is OK
- 10.1002/met.1732 is OK
- 10.5281/zenodo.5173153 is OK
- 10.1111/rssb.12154 is OK
- 10.1002/qj.4266 is OK
- 10.1198/tech.2011.10136 is OK
- 10.1016/S0169-2070(96)00719-4 is OK
- 10.5334/jors.148 is OK
- 10.1080/07350015.1995.10524599 is OK
- 10.1175/BAMS-D-19-0093.1 is OK
- 10.22499/4.0021 is OK
- 10.1175/waf-d-19-0248.1 is OK
- 10.1002/qj.4206 is OK
- 10.1214/21-ejs1957 is OK
- 10.1198/jbes.2010.08110 is OK
- 10.1175/waf-d-23-0201.1 is OK
- 10.1175/bams-d-22-0253.1 is OK
- 10.1073/pnas.2016191118 is OK
- 10.1071/es21010 is OK
- 10.48550/arXiv.2404.18429 is OK
- 10.1002/qj.2270 is OK
- 10.1198/016214506000001437 is OK
- 10.5194/gmd-12-4185-2019 is OK
- 10.1002/qj.4461 is OK
- 10.5281/zenodo.10957263 is OK
- 10.25080/Majora-92bf1922-00a is OK
- 10.5281/zenodo.3764117 is OK
- 10.5065/D6H70CW6 is OK
- 10.11578/dc.20180330.1 is OK
- 10.5281/zenodo.3773450 is OK
- 10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2 is OK
- 10.1126/science.ns-4.93.453.b is OK
- 10.1175/WAF-D-10-05030.1 is OK
- 10.1175/2007MWR2123.1 is OK

MISSING DOIs

- No DOI given, and none found for title: Assessing calibration when predictive distribution...
- No DOI given, and none found for title: Dask: Library for dynamic task scheduling
- No DOI given, and none found for title: WMO No. 306 FM 92 GRIB (edition 2)
- No DOI given, and none found for title: Jupyter Interactive Notebook
- No DOI given, and none found for title: Finley’s tornado predictions

INVALID DOIs

- None

crvernon commented 2 months ago

👋 @tennlee - In your paper's references, please ensure that capitalization has been maintained where it is necessary. You can do this using curly brackets around the formatting you want to maintain in your bib file. For example, in LINE 219 in the paper, "...with generalized huber loss..." the "h" in "huber" should be capitalized. Please check other refs for this as well.

Next is just setting up the archive for your new release.

We want to make sure the archival has the correct metadata that JOSS requires. This includes a title that matches the paper title and a correct author list.

So here is what we have left to do:

[x] Conduct a GitHub release of the current reviewed version of the software and archive the reviewed software in Zenodo or a similar service (e.g., figshare, an institutional repository). Please ensure that the software archive uses the same license as the license you have posted on GitHub.
[x] Check the archival deposit (e.g., in Zenodo) to ensure it has the correct metadata. This includes the title (should match the paper title) and author list (make sure the list is correct and people who only made a small fix are not on it). You may also add the authors' ORCID.
[x] Please respond with the DOI of the archived version here

I can then move forward with accepting the submission.

Steph-Chong commented 2 months ago

@editorialbot generate pdf

editorialbot commented 2 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

Steph-Chong commented 2 months ago

@crvernon we have capitalized the "H" in "Huber" in our references section. We have also done another check of the references section. Please let us know if there is anything else that needs our attention.

We still need to make the GitHub release and archival deposit. Once that is done, we will post here with the version number and archive DOI.

tennlee commented 2 months ago

We have proceeded to release version 0.9.3. I have also updated the branch containing paper.md to match. Release 0.9.3 has been lodged with Zenodo. The DOI link is https://doi.org/10.5281/zenodo.12697242 . I have gone through the checklist you posted, but I was unsure if I should tick that list off myself or whether you would like to double-check.

crvernon commented 2 months ago

@editorialbot set 10.5281/zenodo.12697242 as archive

editorialbot commented 2 months ago

Done! archive is now 10.5281/zenodo.12697242

crvernon commented 2 months ago

@editorialbot set 0.9.3 as version

editorialbot commented 2 months ago

Done! version is now 0.9.3

crvernon commented 2 months ago

🔍 checking out the following:

[x] reviewer checklists are completed or addressed
[x] version set
[x] archive set
[x] archive names (including order) and title in archive matches those specified in the paper
[x] archive uses the same license as the repo and is OSI approved as open source
[x] archive DOI and version match or redirect to those set by editor in review thread
[x] paper is error free - grammar and typos
[x] paper is error free - test links in the paper and bib
[x] paper is error free - refs preserve capitalization where necessary
[x] paper is error free - no invalid refs without justification

crvernon commented 2 months ago

@editorialbot recommend-accept

editorialbot commented 2 months ago

Attempting dry run of processing paper acceptance...

editorialbot commented 2 months ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.21105/joss.02781 is OK
- 10.1002/met.1732 is OK
- 10.5281/zenodo.5173153 is OK
- 10.1111/rssb.12154 is OK
- 10.1002/qj.4266 is OK
- 10.1198/tech.2011.10136 is OK
- 10.1016/S0169-2070(96)00719-4 is OK
- 10.5334/jors.148 is OK
- 10.1080/07350015.1995.10524599 is OK
- 10.1175/BAMS-D-19-0093.1 is OK
- 10.22499/4.0021 is OK
- 10.1175/waf-d-19-0248.1 is OK
- 10.1002/qj.4206 is OK
- 10.1214/21-ejs1957 is OK
- 10.1198/jbes.2010.08110 is OK
- 10.1175/waf-d-23-0201.1 is OK
- 10.1175/bams-d-22-0253.1 is OK
- 10.1073/pnas.2016191118 is OK
- 10.1071/es21010 is OK
- 10.48550/arXiv.2404.18429 is OK
- 10.1002/qj.2270 is OK
- 10.1198/016214506000001437 is OK
- 10.5194/gmd-12-4185-2019 is OK
- 10.1002/qj.4461 is OK
- 10.5281/zenodo.10957263 is OK
- 10.25080/Majora-92bf1922-00a is OK
- 10.5281/zenodo.3764117 is OK
- 10.5065/D6H70CW6 is OK
- 10.11578/dc.20180330.1 is OK
- 10.5281/zenodo.11320255 is OK
- 10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2 is OK
- 10.1126/science.ns-4.93.453.b is OK
- 10.1175/WAF-D-10-05030.1 is OK
- 10.1175/2007MWR2123.1 is OK

MISSING DOIs

- No DOI given, and none found for title: Assessing calibration when predictive distribution...
- No DOI given, and none found for title: Dask: Library for dynamic task scheduling
- No DOI given, and none found for title: WMO No. 306 FM 92 GRIB (edition 2)
- No DOI given, and none found for title: Jupyter Interactive Notebook
- No DOI given, and none found for title: Finley’s tornado predictions

INVALID DOIs

- None

editorialbot commented 2 months ago

:wave: @openjournals/dsais-eics, this paper is ready to be accepted and published.

Check final proof :point_right::page_facing_up: Download article

If the paper PDF and the deposit XML files look good in https://github.com/openjournals/joss-papers/pull/5590, then you can now move forward with accepting the submission by compiling again with the command @editorialbot accept

crvernon commented 2 months ago

@editorialbot accept

editorialbot commented 2 months ago

Doing it live! Attempting automated processing of paper acceptance...

editorialbot commented 2 months ago

Ensure proper citation by uploading a plain text CITATION.cff file to the default branch of your repository.

If using GitHub, a Cite this repository menu will appear in the About section, containing both APA and BibTeX formats. When exported to Zotero using a browser plugin, Zotero will automatically create an entry using the information contained in the .cff file.

You can copy the contents for your CITATION.cff file here:

CITATION.cff

``` cff-version: "1.2.0" authors: - family-names: Leeuwenburg given-names: Tennessee orcid: "https://orcid.org/0009-0008-2024-1967" - family-names: Loveday given-names: Nicholas orcid: "https://orcid.org/0009-0000-5796-7069" - family-names: Ebert given-names: Elizabeth E. - family-names: Cook given-names: Harrison orcid: "https://orcid.org/0009-0009-3207-4876" - family-names: Khanarmuei given-names: Mohammadreza orcid: "https://orcid.org/0000-0002-5017-9622" - family-names: Taggart given-names: Robert J. orcid: "https://orcid.org/0000-0002-0067-5687" - family-names: Ramanathan given-names: Nikeeth orcid: "https://orcid.org/0009-0002-7406-7438" - family-names: Carroll given-names: Maree orcid: "https://orcid.org/0009-0008-6830-8251" - family-names: Chong given-names: Stephanie orcid: "https://orcid.org/0009-0007-0796-4127" - family-names: Griffiths given-names: Aidan - family-names: Sharples given-names: John contact: - family-names: Leeuwenburg given-names: Tennessee orcid: "https://orcid.org/0009-0008-2024-1967" doi: 10.5281/zenodo.12697242 message: If you use this software, please cite our article in the Journal of Open Source Software. preferred-citation: authors: - family-names: Leeuwenburg given-names: Tennessee orcid: "https://orcid.org/0009-0008-2024-1967" - family-names: Loveday given-names: Nicholas orcid: "https://orcid.org/0009-0000-5796-7069" - family-names: Ebert given-names: Elizabeth E. - family-names: Cook given-names: Harrison orcid: "https://orcid.org/0009-0009-3207-4876" - family-names: Khanarmuei given-names: Mohammadreza orcid: "https://orcid.org/0000-0002-5017-9622" - family-names: Taggart given-names: Robert J. orcid: "https://orcid.org/0000-0002-0067-5687" - family-names: Ramanathan given-names: Nikeeth orcid: "https://orcid.org/0009-0002-7406-7438" - family-names: Carroll given-names: Maree orcid: "https://orcid.org/0009-0008-6830-8251" - family-names: Chong given-names: Stephanie orcid: "https://orcid.org/0009-0007-0796-4127" - family-names: Griffiths given-names: Aidan - family-names: Sharples given-names: John date-published: 2024-07-09 doi: 10.21105/joss.06889 issn: 2475-9066 issue: 99 journal: Journal of Open Source Software publisher: name: Open Journals start: 6889 title: "scores: A Python package for verifying and evaluating models and predictions with xarray" type: article url: "https://joss.theoj.org/papers/10.21105/joss.06889" volume: 9 title: "scores: A Python package for verifying and evaluating models and predictions with xarray" ```

If the repository is not hosted on GitHub, a .cff file can still be uploaded to set your preferred citation. Users will be able to manually copy and paste the citation.

Find more information on .cff files here and here.

editorialbot commented 2 months ago

🐘🐘🐘 👉 Toot for this paper 👈 🐘🐘🐘

editorialbot commented 2 months ago

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

Check final PDF and Crossref metadata that was deposited :point_right: https://github.com/openjournals/joss-papers/pull/5592
Wait five minutes, then verify that the paper DOI resolves https://doi.org/10.21105/joss.06889
If everything looks good, then close this review issue.
Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

crvernon commented 2 months ago

🥳 Congratulations on your new publication @tennlee! Many thanks to @calebweinreb and @savente93 for your time, hard work, and expertise!! JOSS wouldn't be able to function nor succeed without your efforts.

Please consider becoming a reviewer for JOSS if you are not already: https://reviewers.joss.theoj.org/join

editorialbot commented 2 months ago

:tada::tada::tada: Congratulations on your paper acceptance! :tada::tada::tada:

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.06889/status.svg)](https://doi.org/10.21105/joss.06889)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.06889">
  <img src="https://joss.theoj.org/papers/10.21105/joss.06889/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.06889/status.svg
   :target: https://doi.org/10.21105/joss.06889

This is how it will look in your documentation:

We need your help!

The Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://reviewers.joss.theoj.org/join
Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss

tennlee commented 2 months ago

Thanks so much @savente93 @calebweinreb and @crvernon! This was a really constructive process and your efforts are all hugely appreciated.

savente93 commented 2 months ago

You're welcome! I'm very glad to hear you found it a constructive process. Congrats, and well done on the publication 🎉

tennlee commented 2 months ago

@crvernon I have noticed a possible error in the Crossref metadata. I am looking at https://api.crossref.org/works/10.21105/joss.06889 . The title has an errant newline instead of a space.

There seem to be newlines in the metadata in other places too, so perhaps it's just how the metadata is - perhaps Crossref have a line width requirement and perhaps it's just their formatting and not an error after all.

The "cite" tool in Crossref is generating citations that look correct - no sign of a newline in there.

What caused me to go looking was when I added the record to my ORCID, there was no space between "evaluating" and "models". This is the same location as the newline appears in the JSON from Crossref.

I'm not sure whether this is significant or not but I wanted to mention it.

crvernon commented 2 months ago

Thanks @tennlee, I'll see if we can get someone to check this out. I'll let you know when I hear back.

danielskatz commented 2 months ago

The metadata we submitted to Crossref is visible in https://github.com/openjournals/joss-papers/pull/5592/files, and it indeed has some strange newlines in it. I'm unsure about what happened here, and am pinging @openjournals/dev about this

Steph-Chong commented 3 weeks ago

@crvernon @danielskatz I was just wondering if there has been any update on the Crossref metadata? The Crossref metadata is still showing the newlines.

danielskatz commented 2 weeks ago

@openjournals/dev - any thoughts on this?

openjournals / joss-reviews