Closed whedon closed 3 years ago
Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @grburgess, @benjaminpope it looks like you're currently assigned to review this paper :tada:.
:warning: JOSS reduced service mode :warning:
Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.
:star: Important :star:
If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿
To fix this do the following two things:
For a list of things I can do to help you, just type:
@whedon commands
For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:
@whedon generate pdf
@grburgess, @benjaminpope – This is the review thread for the paper. All of our communications will happen here from now on.
Please read the "Reviewer instructions & questions" in the first comment above.
Both reviewers have checklists at the top of this thread (in that first comment) with the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.
The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention https://github.com/openjournals/joss-reviews/issues/3285
so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.
We aim for the review process to be completed within about 4-6 weeks but please make a start well ahead of this as JOSS reviews are by their nature iterative and any early feedback you may be able to provide to the author will be very helpful in meeting this schedule.
@arfon I have had the same issue on other reviews... I am not able to check the boxes. It seems independent of the browser. Is this a common issue?
Software report (experimental):
github.com/AlDanial/cloc v 1.88 T=0.57 s (116.2 files/s, 15678.9 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
Python 39 1170 1566 3968
TeX 1 43 0 646
Markdown 6 68 0 372
reStructuredText 6 190 93 301
YAML 9 33 0 272
CSS 1 16 0 78
make 1 8 5 30
TOML 1 2 0 28
INI 1 0 0 8
HTML 1 0 0 6
-------------------------------------------------------------------------------
SUM: 66 1530 1664 5709
-------------------------------------------------------------------------------
Statistical information for the repository '58c76c3075310f0d22a5303f' was
gathered on 2021/05/14.
The following historical commit information, by author, was found:
Author Commits Insertions Deletions % of changes
Adrian Price-Whelan 5 33 18 0.01
Arjun Savel 18 220 24 0.03
Christina Hedges 5 91 82 0.02
Dan F-M 560 165043 316130 58.16
Dan Foreman-Mackey 36 3538 176542 21.77
Eric Agol 1 1 1 0.00
Ian Czekala 5 115 23 0.02
Luke Bouma 1 13 9 0.00
Rodrigo Luger 10 164894 273 19.96
Timothy D Brandt 1 254 0 0.03
Tom Barclay 1 1 1 0.00
Below are the number of rows from each author that have survived and are still
intact in the current revision:
Author Rows Stability Age % in comments
Adrian Price-Whelan 1 3.0 12.4 0.00
Arjun Savel 52 23.6 5.2 5.77
Christina Hedges 38 41.8 2.1 7.89
Dan F-M 135 0.1 8.5 74.07
Dan Foreman-Mackey 6173 174.5 18.2 11.45
Eric Agol 1 100.0 0.2 100.00
Ian Czekala 70 60.9 23.8 14.29
Luke Bouma 13 100.0 10.0 0.00
Rodrigo Luger 221 0.1 19.7 9.95
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):
OK DOIs
- 10.5281/zenodo.1998447 is OK
- 10.5281/zenodo.4695331 is OK
- 10.7717/peerj-cs.55 is OK
- 10.21105/joss.01143 is OK
- 10.1093/mnras/stt1435 is OK
- 10.1093/mnrasl/slt075 is OK
- 10.3847/1538-3881/aaf22f is OK
- 10.1051/0004-6361/201322068 is OK
- 10.3847/1538-3881/aabc4f is OK
- 10.3847/1538-3881/aae8e5 is OK
- 10.3847/1538-3881/ab4fee is OK
- 10.3847/1538-3881/aa9332 is OK
- 10.3847/2515-5172/aaaf6c is OK
- 10.1051/0004-6361/201118085 is OK
- 10.1093/mnras/stz2870 is OK
- 10.1086/669497 is OK
- 10.1093/mnras/stz2688 is OK
- 10.3847/1538-4365/abe70e is OK
- 10.1086/683602 is OK
- 10.1088/1538-3873/aaaaa8 is OK
- 10.3847/1538-3881/ab6663 is OK
- 10.1051/0004-6361/201628579 is OK
- 10.1093/mnras/sty2472 is OK
- 10.1093/mnras/stv894 is OK
- 10.1093/mnras/stz3251 is OK
- 10.3847/1538-4357/abc686 is OK
- 10.3847/1538-3881/abbc16 is OK
- 10.18637/jss.v076.i01 is OK
- 10.1038/s41586-020-2649-2 is OK
- 10.3847/1538-4357/abebe3 is OK
- 10.3847/1538-3881/aba4b2 is OK
- 10.1038/s41586-020-2400-z is OK
- 10.1093/mnras/stx138 is OK
MISSING DOIs
- None
INVALID DOIs
- None
:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
Intro Comments:
Installation:
python setup.py
Unit tests:
test_small_star
and test_sky_coords
. This was because I hadn't yet installed batman - if these are going to be in unit tests perhaps they should also be in dependencies.Case Studies:
WARNING (theano.tensor.opt): Cannot construct a scalar test value from a test value with no size: InplaceDimShuffle{x,0}.0
WARNING (theano.tensor.opt): Cannot construct a scalar test value from a test value with no size: Elemwise{cos,no_inplace}.0
General paper comments:
exoplanet
. The multi-instrument light curve and RV fits from "Fitting light curves from multiple instruments" and "RVs with multiple instruments" respectively could go well as a multi-panel figure. (This is by no means a requirement for the paper to be accepted, just an opportunity). ldtk
(I list these because they are the ones I have used myself; wide citation is to be encouraged, and perhaps the authors can think of others). Given the detached EB model, perhaps a citation to Irwin's eb
or the PHOEBE
or JKTEBOP
projects?Thanks @benjaminpope for your detailed and constructive review - all of these suggestions are very helpful!! As you've seen, I've created a series of issue threads to track these changes and I'll report back here once I've gone through them.
This is a very well written package from a software point of view. It follows all the modern standards of CI and maintainable python code. The documentation appears complete and testing is well done.
I will second comments of @benjaminpope in order to not duplicate with an emphasis on expanding the autodiff explanation. Even more, why do you need HMC to do this? I wholeheartedly agree, but its a teaching opportunity to the field that these types of integrators are more robust for heavy-tailed and correlated posteriors and basically the only way to deal with high-dimensions. This seems to be a core benefit of this tool to the field, and could use a little more promotion.
I recommend publication as well, after addressing the issues I have linked above.
pet peeve: In the documentation there are several references to "finding the best parameters." I know what is heuristically meant, but perhaps one could use "estimating/computing the posterior" or "conditioning the model on data"? There is a perception in the field that Bayesian inference is a way to "find the parameter errors" more robustly than MLE methods, and careful use of language can help to overcome that.. but this is a style preference only :)
I noticed that one of the models was an interpolated light curve. Is this actually used in fitting? If it is used as a template where the shape does not change except for the scaling, this should be fine. But HMC typically goes bonkers with interpolated models that change their shape. This is more of curiosity of mine... but I also may have not understood what that model is used for in the library.
As a non-expert, I see this as a framework that includes some models and can be built upon. Is this a correct interpretation? Have you experimented with different parameterizations of the models and/or have suggestions for users. HMC is great, but it can still be hampered by poor parameterization. I do note that scale parameters are "logged" as they should be (in general).
@grburgess: Thanks for this review and the opened issues! Can I ask you to expand a bit on what you mean by:
But HMC typically goes bonkers with interpolated models that change their shape.
And especially by "change their shape". I haven't found major issues with using interpolated models and I don't see any fundamental issue with using interpolation, so I'd love to hear more about your experiences!
@dfm In the past in Stan, when trying to use high-D interpolation, I've found that HMC gets really confused around the nodes. But, 1D interpolation, where the coefficients are precomputed and used like a template works fine. Example, you interpolate a light curve outside of the program from some complex model, and then are simply fitting for an amplitude that scales the interpolation. But I've had issues in the past were you also try to scale the time axis (stretching). I'm sure it has something to do with HMC diverging... but I never really investigated further.
I was not sure what the intended use was here. If it is working that is great.
@arfon I am still unable to interact with the checkboxes.
@whedon re-invite @grburgess as reviewer
OK, the reviewer has been re-invited.
@grburgess please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations
@grburgess – could you try clicking this link and accepting the invite? https://github.com/openjournals/joss-reviews/invitations
@arfon that worked. thanks!
:wave: @benjaminpope, please update us on how your review is going (this is an automated reminder).
Re Whedon post - just waiting for confirmation on all the pull requests that changes have been made. Everything looks good so far - the comment about paper content & adding a figure (which is not compulsory at all) hasn't been decided either way, so as far as I can tell I'm just waiting on that.
Yes - Definitely in my court still!
Finishing all the changes will probably take me a while still, but I'll have updates on individual issues as we go. Thanks again!
@benjaminpope, @grburgess: Thanks again so much for your thorough reviews! I've gone through and addressed all of your comments, I think. Here's the project showing all the related issues and PRs. The big ones were:
Overall the major changes were to the documentation, so:
I completely overhauled the documentation page and case studies. The big conceptual difference is that I moved some of the more specific tutorials to the case studies page and then added some more high level tutorials about autodiff, the range of models supported by the code, and some (limited) notes about reparameterization.
I updated the infrastructure to use the sphinx-book-theme
for both the docs and case studies which includes buttons for launching the ipynbs using binder or downloading them directly. This should make it easier for users to dive right in without having to copy and paste the code and worry about missing imports.
I dealt with (and tried to solve) various compatibility (ugh macos multiprocessing!) and runtime issues, added some more inline comments, and fixed various small bugs or issues that were identified.
Let me know if you see any remaining issues!!
@whedon generate pdf
:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:
@dfm - thanks for the updates. I have looked through the paper - happy with the new density of citations and the new figures, it's great. I have also looked at the new case studies and autodiff page - these are some of the most professional tutorial pages I have ever seen. I will be linking people to the autodiff one especially.
With that, all requested changes are complete and to a very high standard, and I am happy to recommend the paper for acceptance by JOSS.
@dfm I second acceptance. The docs look fantastic and very useful. Thanks for taking the time to make this project pedagogical as well as a beneficial to the community.
@dfm – At this point could you make a new release of this software that includes the changes that have resulted from this review. Then, please make an archive of the software in Zenodo/figshare/other service and update this thread with the DOI of the archive? For the Zenodo/figshare archive, please make sure that:
I can then move forward with accepting the submission.
@arfon: Thanks! I'm waiting for a final round of comments from co-authors, but I'll tag a release next week.
@arfon: I've bumped the version number to v0.5.1
and the archive is on Zenodo with the correct metadata at 10.5281/zenodo.5006965
. Thanks!
@whedon set 10.5281/zenodo.5006965 as archive
OK. 10.5281/zenodo.5006965 is the archive.
@whedon set v0.5.1 as version
OK. v0.5.1 is the version.
@whedon recommend-accept
Attempting dry run of processing paper acceptance...
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):
OK DOIs
- 10.5281/zenodo.1998447 is OK
- 10.5281/zenodo.4695331 is OK
- 10.7717/peerj-cs.55 is OK
- 10.21105/joss.01143 is OK
- 10.1093/mnras/stt1435 is OK
- 10.1093/mnrasl/slt075 is OK
- 10.3847/1538-3881/aaf22f is OK
- 10.1051/0004-6361/201322068 is OK
- 10.3847/1538-3881/aabc4f is OK
- 10.3847/1538-3881/aae8e5 is OK
- 10.3847/1538-3881/ab4fee is OK
- 10.3847/1538-3881/aa9332 is OK
- 10.3847/2515-5172/aaaf6c is OK
- 10.1051/0004-6361/201118085 is OK
- 10.1093/mnras/stz2870 is OK
- 10.1086/669497 is OK
- 10.1093/mnras/stz2688 is OK
- 10.3847/1538-4365/abe70e is OK
- 10.1086/683602 is OK
- 10.1088/1538-3873/aaaaa8 is OK
- 10.3847/1538-3881/ab6663 is OK
- 10.1051/0004-6361/201628579 is OK
- 10.1093/mnras/sty2472 is OK
- 10.1093/mnras/stv894 is OK
- 10.1093/mnras/stz3251 is OK
- 10.3847/1538-4357/abc686 is OK
- 10.3847/1538-3881/abbc16 is OK
- 10.18637/jss.v076.i01 is OK
- 10.1038/s41586-020-2649-2 is OK
- 10.3847/1538-4357/abebe3 is OK
- 10.3847/1538-3881/aba4b2 is OK
- 10.1038/s41586-020-2400-z is OK
- 10.1093/mnras/stx138 is OK
- 10.1093/mnras/stv1857 is OK
- 10.1088/0004-637X/742/2/123 is OK
- 10.1111/j.1365-2966.2004.07871.x is OK
- 10.3847/1538-4365/abb4e2 is OK
MISSING DOIs
- None
INVALID DOIs
- None
:wave: @openjournals/joss-eics, this paper is ready to be accepted and published.
Check final proof :point_right: https://github.com/openjournals/joss-papers/pull/2400
If the paper PDF and Crossref deposit XML look good in https://github.com/openjournals/joss-papers/pull/2400, then you can now move forward with accepting the submission by compiling again with the flag deposit=true
e.g.
@whedon accept deposit=true
@whedon accept deposit=true
Doing it live! Attempting automated processing of paper acceptance...
🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦
🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨
Here's what you must now do:
Party like you just published a paper! 🎉🌈🦄💃👻🤘
Any issues? Notify your editorial technical team...
@grburgess, @benjaminpope – many thanks for your reviews here! JOSS relies upon volunteer efforts from people like you and we wouldn't be able to do this without you! ✨
@dfm – your paper is now accepted and published in JOSS :zap::rocket::boom:
:tada::tada::tada: Congratulations on your paper acceptance! :tada::tada::tada:
If you would like to include a link to your paper from your README use the following code snippets:
Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.03285/status.svg)](https://doi.org/10.21105/joss.03285)
HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.03285">
<img src="https://joss.theoj.org/papers/10.21105/joss.03285/status.svg" alt="DOI badge" >
</a>
reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.03285/status.svg
:target: https://doi.org/10.21105/joss.03285
This is how it will look in your documentation:
We need your help!
Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:
Submitting author: @dfm (Daniel Foreman-Mackey) Repository: https://github.com/exoplanet-dev/exoplanet Version: v0.5.1 Editor: @arfon Reviewer: @grburgess, @benjaminpope Archive: 10.5281/zenodo.5006965
:warning: JOSS reduced service mode :warning:
Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.
Status
Status badge code:
Reviewers and authors:
Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)
Reviewer instructions & questions
@grburgess & @benjaminpope, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:
The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @arfon know.
✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨
Review checklist for @grburgess
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper
Review checklist for @benjaminpope
Conflict of interest
Code of Conduct
General checks
Functionality
Documentation
Software paper