search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
721 stars 38 forks source link

[REVIEW]: pyhf: pure-Python implementation of HistFactory statistical models #2823

Closed whedon closed 3 years ago

whedon commented 3 years ago

Submitting author: @matthewfeickert (Matthew Feickert) Repository: https://github.com/scikit-hep/pyhf Version: v0.5.4 Editor: @eloisabentivegna Reviewer: @suchitakulkarni, @bradkav Archive: 10.5281/zenodo.4318533

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/e26720fa85f0ff3720df588acf968f80"><img src="https://joss.theoj.org/papers/e26720fa85f0ff3720df588acf968f80/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/e26720fa85f0ff3720df588acf968f80/status.svg)](https://joss.theoj.org/papers/e26720fa85f0ff3720df588acf968f80)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@suchitakulkarni & @bradkav, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @eloisabentivegna know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @suchitakulkarni

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @bradkav

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

eloisabentivegna commented 3 years ago

@whedon check references from branch docs/add-JOSS-paper

whedon commented 3 years ago 
Attempting to check references... from custom branch docs/add-JOSS-paper
whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.physletb.2014.05.011 is OK
- 10.1088/1742-6596/898/10/102006 is OK
- 10.5281/zenodo.1169739 is OK
- 10.31526/lhep.2020.158 is OK
- 10.1140/epjc/s10052-011-1554-0 is OK

MISSING DOIs

- 10.2172/1614734 may be a valid DOI for title: Reinterpretation of LHC Results for New Physics: Status and Recommendations after Run 2

INVALID DOIs

- None
matthewfeickert commented 3 years ago

@bradkav, @suchitakulkarni, many thanks for your help and comments. Based on your recommendation, I will start the prepublication phase for this submission.

Yes, many thanks to @bradkav and @suchitakulkarni not only for their review but also for helping to make pyhf better! We truly do appreciate it and your time and effort. :bow: 

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

MISSING DOIs

- 10.2172/1614734 may be a valid DOI for title: Reinterpretation of LHC Results for New Physics: Status and Recommendations after Run 2

Ah it seems that Reinterpretation of LHC Results for New Physics: Status and Recommendations after Run 2 now has an updated DOI of 0.21468/SciPostPhys.9.2.022. I'll update this and trigger whedon again.

matthewfeickert commented 3 years ago

@whedon generate pdf from branch docs/add-JOSS-paper

whedon commented 3 years ago 
Attempting PDF compilation from custom branch docs/add-JOSS-paper. Reticulating splines etc...
eloisabentivegna commented 3 years ago

The proof reads fine to me!

@matthewfeickert, @kratsg, we'll also need a tagged release and a code archive for your submission (see https://joss.readthedocs.io/en/latest/submitting.html#the-review-process). I see that a Zenodo archive already exists, but I believe this does not match the current version. The archive title should also match the submission title.

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

matthewfeickert commented 3 years ago

@whedon check references from branch docs/add-JOSS-paper

whedon commented 3 years ago 
Attempting to check references... from custom branch docs/add-JOSS-paper
whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.physletb.2014.05.011 is OK
- 10.1088/1742-6596/898/10/102006 is OK
- 10.5281/zenodo.1169739 is OK
- 10.21468/SciPostPhys.9.2.022 is OK
- 10.31526/lhep.2020.158 is OK
- 10.1140/epjc/s10052-011-1554-0 is OK

MISSING DOIs

- None

INVALID DOIs

- None
matthewfeickert commented 3 years ago

we'll also need a tagged release and a code archive for your submission (see https://joss.readthedocs.io/en/latest/submitting.html#the-review-process). I see that a Zenodo archive already exists, but I believe this does not match the current version. The archive title should also match the submission title.

Sounds good :+1: . I'll get on that in the morning my time (US) and then update here as soon as that's taken care of.

matthewfeickert commented 3 years ago

@whedon generate pdf from branch master

whedon commented 3 years ago 
Attempting PDF compilation from custom branch master. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

matthewfeickert commented 3 years ago

@eloisabentivegna We have merged the branch and have cut a release on GitHub with tag JOSS-paper which now exists on Zenodo as DOI 10.5281/zenodo.4472736. DOI

The Zenodo title is that of the paper, "pyhf: pure-Python implementation of HistFactory statistical models", and I have set the version to be JOSS-paper instead of v0.5.4 to help distinguish it from the v0.5.4 release Zenodo archive.

Please let me know if you have any requested changes or if there are more standard ways that people prepare JOSS papers on Zenodo.

eloisabentivegna commented 3 years ago

Thanks, @matthewfeickert. Do you want to change the reference in the main text to point to this new archive?

eloisabentivegna commented 3 years ago

@whedon set JOSS-paper as version

whedon commented 3 years ago

OK. JOSS-paper is the version.

eloisabentivegna commented 3 years ago

@whedon set 10.5281/zenodo.4472736 as archive

whedon commented 3 years ago

OK. 10.5281/zenodo.4472736 is the archive.

matthewfeickert commented 3 years ago

Do you want to change the reference in the main text to point to this new archive?

@eloisabentivegna Does this show up anywhere other than this Issue? If so, and it shows up in the paper (seems like no from looking at the draft) then if we could have version v0.5.4, which is the release that the code reviewed corresponds to, that would be better. If it is just for bookkeeping then it is fine to keep it as JOSS-paper.

Ah sorry I think I misunderstood. You're referring to the citation in the paper itself to https://doi.org/10.5281/zenodo.1169739? We generally use the top level Zenodo archive DOI for pyhf, 10.5281/zenodo.1169739. As the JOSS-paper release is a pre-release as we're getting close to releasing v0.6.0 with some API changes, it would probably be better to keep the reference to v0.5.4 and the top level Zenodo archive for stability and consistency. So if you're fine with it I think setting the version to v0.5.4 would be best.

eloisabentivegna commented 3 years ago

@matthewfeickert, thanks for the explanation, I didn't realize the top-level DOI now points to the JOSS version. Does this mean that once new versions are added to the archive, the DOI in the paper will point to them instead of the version corresponding to the JOSS submission? I think this would be acceptable, but I would perhaps recommend making it clearer in the text (e.g. adding "The version corresponding to this article is XXX" on line 28).

As for the version, are 0.5.4 and JOSS-paper identical but for the paper itself?

matthewfeickert commented 3 years ago

I didn't realize the top-level DOI now points to the JOSS version. Does this mean that once new versions are added to the archive, the DOI in the paper will point to them instead of the version corresponding to the JOSS submission?

Yes, the preferred BibTeX entry for citation that we include in our README and docs gets bumped automatically by our workflow when we make new releases to PyPI. The version number gets bumped, and the DOI stays the same. We use what Zenodo refers to as the "cite all versions" DOI for pyhf 10.5281/zenodo.1169739 to keep people from grabbing the citation and putting it in their citation manager and citing only older versions in the future, which sadly does happen. You can see the version release history in the Zenodo versions:

Zenodo_versions

I think this would be acceptable, but I would perhaps recommend making it clearer in the text (e.g. adding "The version corresponding to this article is XXX" on line 28).

Okay, the version number is already explicitly given in the citation

draft_reference

but I can revise the paper to mention it in the text.

I'm a bit surprised given the metadata that exists in the left column of the first page of the paper that version number isn't listed there too. Is there a place that I can suggest revisions to the JOSS paper formattting?

As for the version, are 0.5.4 and JOSS-paper identical but for the paper itself?

No, they are significantly different as we've been preparing for release v0.6.0.

matthewfeickert commented 3 years ago

@whedon generate pdf from branch docs/add-version-number-to-JOSS-paper

whedon commented 3 years ago 
Attempting PDF compilation from custom branch docs/add-version-number-to-JOSS-paper. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

matthewfeickert commented 3 years ago

@eloisabentivegna I've incorporated the release number into the paper text now. :+1: If the preview that whedon built (above) looks okay I'll merge pyhf PR 1280 to bring the changes in and then have whedon rebuild off pyhf's master branch.

As we've changed the paper, after I merge the PR do you want me to additionally cut another release to re-trigger Zenodo to make another archive?

eloisabentivegna commented 3 years ago

Thanks, @matthewfeickert! The spirit of having a tagger release quoted with the paper is that it will allow readers to track down which version exactly was reviewed as part of the JOSS procedure. If JOSS-paper is significantly different from what @bradkav and @suchitakulkarni have reviewed, then it cannot be the release associated with the submission. Could you (and @bradkav and @suchitakulkarni) confirm that 0.5.4 is what the reviewers worked with? If so, I am happy with the new proof and ready to issue the accept command.

Regarding suggestions, you're more than welcome to open a PR on the JOSS repo (https://github.com/openjournals/joss). Thanks!

matthewfeickert commented 3 years ago

The spirit of having a tagger release quoted with the paper is that it will allow readers to track down which version exactly was reviewed as part of the JOSS procedure. If JOSS-paper is significantly different from what @bradkav and @suchitakulkarni have reviewed, then it cannot be the release associated with the submission. Could you (and @bradkav and @suchitakulkarni) confirm that 0.5.4 is what the reviewers worked with? If so, I am happy with the new proof and ready to issue the accept command.

@eloisabentivegna Thanks for the clarification. This should be fine then to use v0.5.4. In this case then the only thing that matters is the public API and the documentation. We only support officially released versions of pyhf, so as long as the public API is stable for the checks used for the review then continued development work shouldn't matter. There will be additions to the public API for v0.6.0 that is shortly forthcoming, but nothing in the changes from the public API for v0.5.4 would be relevant to checks in the "Functionality" section of the review (Installation, Functionality, Performance).

This is demonstrable as we have automated nightly tests that run the test suite for the public API using the latest release from PyPI so we know for sure anytime there is a breaking change in our public API as this test breaks (c.f. the GitHub Actions workflow here).

open a PR on the JOSS repo

Awesome! :+1: Thanks for the direction.

matthewfeickert commented 3 years ago

@whedon generate pdf from branch master

whedon commented 3 years ago 
Attempting PDF compilation from custom branch master. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

eloisabentivegna commented 3 years ago

@matthewfeickert, it's all clear now. Thanks!

I will switch the version to v0.5.4 and proceed.

Thanks for your submission and interactions, and thanks @bradkav and @suchitakulkarni for finding the time for the review in this extraordinarily challenging moment.

Congratulations! 🎉

eloisabentivegna commented 3 years ago

@whedon set v0.5.4 as version

whedon commented 3 years ago

OK. v0.5.4 is the version.

eloisabentivegna commented 3 years ago

@whedon generate pdf

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

eloisabentivegna commented 3 years ago

@whedon check references

whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.physletb.2014.05.011 is OK
- 10.1088/1742-6596/898/10/102006 is OK
- 10.5281/zenodo.1169739 is OK
- 10.21468/SciPostPhys.9.2.022 is OK
- 10.31526/lhep.2020.158 is OK
- 10.1140/epjc/s10052-011-1554-0 is OK

MISSING DOIs

- None

INVALID DOIs

- None
eloisabentivegna commented 3 years ago

@whedon accept

whedon commented 3 years ago 
Attempting dry run of processing paper acceptance...
whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.physletb.2014.05.011 is OK
- 10.1088/1742-6596/898/10/102006 is OK
- 10.5281/zenodo.1169739 is OK
- 10.21468/SciPostPhys.9.2.022 is OK
- 10.31526/lhep.2020.158 is OK
- 10.1140/epjc/s10052-011-1554-0 is OK

MISSING DOIs

- None

INVALID DOIs

- None
whedon commented 3 years ago

:wave: @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof :point_right: https://github.com/openjournals/joss-papers/pull/2067

If the paper PDF and Crossref deposit XML look good in https://github.com/openjournals/joss-papers/pull/2067, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true
matthewfeickert commented 3 years ago

it's all clear now. Thanks!

I will switch the version to v0.5.4 and proceed.

Thanks for your submission and interactions, and thanks @bradkav and @suchitakulkarni for finding the time for the review in this extraordinarily challenging moment.

Congratulations!

Thank you so much @eloisabentivegna! You have been a fantastic editor and we very much appreciate getting to work with you on this and for how helpful you have been the whole way. :smile:

One final question: At the moment the archive is:

Archive: 10.5281/zenodo.4472736

which is the JOSS-paper tag of pyhf before I put in the clarifying sentence RE: the v0.5.4 version number. Would it be better to have the archive have the exact text of the paper? If so, I can delete the current GitHub tag, and request the Zenodo team delete the current archive and then simply make a new JOSS-paper release on GitHub and ask whedon (or maybe you have to given permissions?) to update the archive URL. We're fine either way, but I in now way mind updating things and I think the Zenodo team could probably delete the current archive withing 24 hours.

I've already done the above and cut a new GitHub release named JOSS-paper which is now also archived on Zenodo under the same name at https://doi.org/10.5281/zenodo.4484948. I've also requested the Zenodo team delete the old archive to avoid potential confusion in the future. Can you please have whedon set 10.5281/zenodo.4484948 as the archive, as I've tried below but lack the correct permissions?

Update: The Zenodo team is very fast and has already deleted the archive 10.5281/zenodo.4472736, so there will be no confusion with the new archive 10.5281/zenodo.4484948. :+1:

matthewfeickert commented 3 years ago

@whedon set 10.5281/zenodo.4484948 as archive

whedon commented 3 years ago

I'm sorry @matthewfeickert, I'm afraid I can't do that. That's something only editors are allowed to do.

arfon commented 3 years ago

@whedon set 10.5281/zenodo.4484948 as archive

whedon commented 3 years ago

OK. 10.5281/zenodo.4484948 is the archive.

arfon commented 3 years ago

@whedon accept