search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
703 stars 36 forks source link

[REVIEW]: CalibrateEmulateSample.jl: Accelerated Parametric Uncertainty Quantification #6372

Closed editorialbot closed 4 months ago

editorialbot commented 6 months ago

Submitting author: !--author-handle-->@odunbar<!--end-author-handle-- (Oliver Dunbar) Repository: https://github.com/CliMA/CalibrateEmulateSample.jl Branch with paper.md (empty if default branch): Version: v0.5.1 Editor: !--editor-->@sappelhoff<!--end-editor-- Reviewers: @matt-graham, @Vaibhavdixit02, @nluetts Archive: 10.5281/zenodo.10946875

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/c761e46a8a82ce342fad84a2f3e1ae69"><img src="https://joss.theoj.org/papers/c761e46a8a82ce342fad84a2f3e1ae69/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/c761e46a8a82ce342fad84a2f3e1ae69/status.svg)](https://joss.theoj.org/papers/c761e46a8a82ce342fad84a2f3e1ae69)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@matt-graham & @Vaibhavdixit02 & @nluetts, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @sappelhoff know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @matt-graham

📝 Checklist for @nluetts

📝 Checklist for @Vaibhavdixit02

editorialbot commented 6 months ago

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf
editorialbot commented 6 months ago 
Software report:

github.com/AlDanial/cloc v 1.88  T=0.12 s (701.2 files/s, 115307.0 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
Julia                            36           1453           1228           6686
Markdown                         24            632              0           2558
YAML                              8             50              7            319
TeX                               1             26              0            248
TOML                             11             10              0            160
SVG                               1              0              0             71
Bourne Again Shell                1             10             15             11
--------------------------------------------------------------------------------
SUM:                             82           2181           1250          10053
--------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository
editorialbot commented 6 months ago

Wordcount for paper.md is 1514

editorialbot commented 6 months ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1137/141000671 is OK
- 10.21105/joss.04869 is OK
- 10.1029/2022MS002994 is OK
- 10.1109/TPAMI.2021.3097011 is OK
- 10.1214/13-STS421 is OK
- 10.21105/joss.04561 is OK
- 10.1137/19M1304891 is OK
- 10.1093/bioinformatics/btaa078 is OK
- 10.1088/1361-6420/ac99fa is OK
- 10.1029/2022MS003245 is OK
- 10.22541/essoar.170365299.96491153/v1 is OK

MISSING DOIs

- 10.1088/0266-5611/29/4/045001 may be a valid DOI for title: Ensemble Kalman methods for inverse problems
- 10.1109/allerton.2008.4797607 may be a valid DOI for title: Uniform approximation of functions with random bases
- 10.1214/10-sts327 may be a valid DOI for title: The Random Walk Metropolis: Linking Theory and Practice Through a Case Study

INVALID DOIs

- https://doi.org/10.1016/j.jcp.2020.109716 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1029/2020MS002454 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1029/2021MS002735 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1029/2022MS002997 is INVALID because of 'https://doi.org/' prefix
editorialbot commented 6 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

sappelhoff commented 6 months ago

Hello again! 👋


@matt-graham, @Vaibhavdixit02, @nluetts

FYI @odunbar

This is the review thread for the paper. All of our higher-level communications will happen here from now on, review comments and discussion can happen in the repository of the project (details below).

📓 Please read the "Reviewer instructions & questions" in the comment from our editorialbot (above).

✅ All reviewers get their own checklist with the JOSS requirements - you generate them as per the details in the editorialbot comment. As you go over the submission, please check any items that you feel have been satisfied.

💻 The JOSS review is different from most other journals: The reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention the link to https://github.com/openjournals/joss-reviews/issues/6372 so that a link is created to this thread. That will also help me to keep track!

❓ Please also feel free to comment and ask questions on this thread if you are unsure about something!

🎯 We aim for the review process to be completed within about 4-6 weeks but please make a start well ahead of this as JOSS reviews are by their nature iterative and any early feedback you may be able to provide to the author will be very helpful in meeting this schedule. NOTE: I am aware that some of you asked for some additional time due to having other obligations for now. That's no problem, please just communicate with me if your schedule for this review changes.

nluetts commented 6 months ago

Review checklist for @nluetts

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

matt-graham commented 6 months ago

Review checklist for @matt-graham

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Vaibhavdixit02 commented 6 months ago

Review checklist for @Vaibhavdixit02

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

sappelhoff commented 6 months ago

@Vaibhavdixit02 if you are done with your review (seems like it from your checkmarks), I'd appreciate if you could write a short summary whether you recommend this paper for acceptance or/and if some points still need to be addressed.

Same for @matt-graham once you are finished, please 🙏

@nluetts I know that you will only be able to really get started on this in early March, please keep me updated should anything change with that.

Vaibhavdixit02 commented 6 months ago

@sappelhoff the package is quite well-developed and has comprehensive documentation and usage tutorials. It is a novel implementation as well so from my end it is a clear accept. The points raised by Matt are reasonable and I am following those issues barring which I don't have any other comments here.

nluetts commented 6 months ago

@nluetts I know that you will only be able to really get started on this in early March, please keep me updated should anything change with that.

All fine, @sappelhoff , I just started with the review.

sappelhoff commented 6 months ago

@matt-graham may I ask you for an update on your ongoing review, please? What are the outstanding points, and the approximate timeline for them?

matt-graham commented 6 months ago

@sappelhoff - I've currently got no spare capacity due to other commitments so will not be able to look at this again till next week. I saw that the authors responded to the issues I previously raised but haven't had a chance to look through their responses to two of the issues yet. I also have some minor feedback on the paper which I haven't yet had a chance to transcribe in to an issue.

sappelhoff commented 5 months ago

@matt-graham @nluetts just to check in with you: Are there any blockers that are holding you back from completing your reviews? Are you able to share an approximate date with us of when you might finish the last steps? Thanks!

nluetts commented 5 months ago

@sappelhoff The cold season has taken its toll on me a little, but I resumed the review now and have some time during this week, so I try to provide all my feedback by Friday. Apropos feedback, when I have questions and feedback to the JOSS paper draft itself, am I supposed to open an issue over at https://github.com/CliMA/CalibrateEmulateSample.jl or comment in this issue?

sappelhoff commented 5 months ago

Sorry to hear that, get well soon @nluetts! Thanks for your updated schedule as well. Please open issues at https://github.com/CliMA/CalibrateEmulateSample.jl also when it's about the JOSS paper draft 🙏

matt-graham commented 5 months ago

Apologies for the delay in finishing off my review. I have opened an issue with some minor suggestions for changes to text to paper (and have therefore left a couple of items on reviewer checklist unticked but just to be clear I don't have any concerns about overall quality of writing). @odunbar has addressed everything in the other issues I raised so from my perspective it is only the minor changes to the paper that are outstanding.

odunbar commented 5 months ago

Hi everyone! Thanks for all the comments on the documentation, examples and manuscript. I believe as of now we have addressed all points raised, with PRs documenting all changes that link to the issues above. Let me know if any item was missed.

I'd like to make one additional change related to adding one further author to the paper. @nefrathenrici . Though they have not made visible direct contributions to the exact CES repository so far, Nat has been involved in a lot of recent decisionmaking, and added other auxilliary code such as pipelines to enable automated workflows for CalibrateEmulateSample needs for HPC applications, with various PRs to our sister repositories. I think as Nat is also primed to play more of a role moving forwards with CalibrateEmulateSample.jl they will prove a responsible and reliable steward for this codebase in future.

I will link an issue/PR adding Nat to the code here once this is done.

I think our docs and examples have definitely benefited from your reviews, and should hopefully help future users!

Cheers, Ollie

nluetts commented 5 months ago

@sappelhoff : @odunbar addressed all points I raised with regard to the paper and the documentation of the package, so I ticked off the remaining items on my checklist. I agree with @Vaibhavdixit02 conclusions and recommend this paper for acceptance.

sappelhoff commented 5 months ago

Thanks a lot @nluetts for your final verdict.

And thanks @odunbar for the update.

Once @matt-graham has finished the last steps, we can move on to a final evaluation by me and the next steps.

matt-graham commented 5 months ago

Once @matt-graham has finished the last steps, we can move on to a final evaluation by me and the next steps.

@sappelhoff - I've now completed my reviewer checklist and the authors have addressed all the points I raised, so I think everything now ready for your final evaluation.

sappelhoff commented 5 months ago

Thanks a lot @matt-graham, and also @nluetts and @Vaibhavdixit02 again for completing your reviews! 👏

sappelhoff commented 5 months ago

@nluetts I could not find you in our reviewer database. If you enjoyed the process of reviewing for JOSS, we'd be very grateful if you considered signing up for potential future services as a reviewer🙂 ... You can do that here: https://reviewers.joss.theoj.org/join

sappelhoff commented 5 months ago

Post-Review Checklist for Editor and Authors

Additional Author Tasks After Review is Complete

Editor Tasks Prior to Acceptance

sappelhoff commented 5 months ago

@odunbar please let me know once you finished your items from the list above: https://github.com/openjournals/joss-reviews/issues/6372#issuecomment-2043015005

I will then work on my final items, and then it's (hopefully) finally time to ship it!

odunbar commented 5 months ago

Hi @sappelhoff

Let me know if there are any issues

odunbar commented 5 months ago

@sappelhoff (just in case this didn't ping you)

sappelhoff commented 5 months ago

Hi @odunbar thanks for the ping -- I have it on my list and will get to it in time -- latest next Monday ✅ but perhaps earlier if I can squeeze it in.

sappelhoff commented 4 months ago

@editorialbot generate pdf

editorialbot commented 4 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

sappelhoff commented 4 months ago

@editorialbot check references

editorialbot commented 4 months ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1137/141000671 is OK
- 10.21105/joss.04869 is OK
- 10.1029/2022MS002994 is OK
- 10.1109/TPAMI.2021.3097011 is OK
- 10.1214/13-STS421 is OK
- 10.21105/joss.04561 is OK
- 10.1137/19M1304891 is OK
- 10.1093/bioinformatics/btaa078 is OK
- 10.1088/1361-6420/ac99fa is OK
- 10.1029/2022MS003245 is OK
- 10.22541/essoar.170365299.96491153/v1 is OK
- 10.21105/joss.05428 is OK

MISSING DOIs

- No DOI given, and none found for title: Handbook of Approximate Bayesian Computation
- 10.1201/9781315117195-8 may be a valid DOI for title: High-Dimensional ABC
- No DOI given, and none found for title: Supervised Calibration and Uncertainty Quantificat...
- No DOI given, and none found for title: Gaussian processes for machine learning
- 10.1088/0266-5611/29/4/045001 may be a valid DOI for title: Ensemble Kalman methods for inverse problems
- No DOI given, and none found for title: Random Features for Large-Scale Kernel Machines.
- 10.1109/allerton.2008.4797607 may be a valid DOI for title: Uniform approximation of functions with random bas...
- 10.1214/10-sts327 may be a valid DOI for title: The Random Walk Metropolis: Linking Theory and Pra...
- No DOI given, and none found for title: Scikit-learn: Machine Learning in Python
- No DOI given, and none found for title: GaussianProcesses. jl: A Nonparametric Bayes Packa...
- No DOI given, and none found for title: Equation of state calculations by fast computing m...
- 10.1088/1475-7516/2023/10/021 may be a valid DOI for title: Fast and robust Bayesian Inference using Gaussian ...
- 10.1111/rssb.12482 may be a valid DOI for title: The Barker proposal: combining robustness and effi...
- No DOI given, and none found for title: The No-U-Turn sampler: adaptively setting path len...

INVALID DOIs

- https://doi.org/10.1016/j.jcp.2020.109716 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1029/2020MS002454 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1029/2021MS002735 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1029/2022MS002997 is INVALID because of 'https://doi.org/' prefix
sappelhoff commented 4 months ago

@odunbar sorry for the delay. Here is a list of comments from my side. I would appreciate a response to all of them (preferably with links to the commits in which you addressed them), and if you don't want to follow my suggestions on a particular point, I'd appreciate a reason for why not.

Thanks!

sappelhoff commented 4 months ago

@editorialbot set 10.5281/zenodo.10946875 as archive

editorialbot commented 4 months ago

Done! archive is now 10.5281/zenodo.10946875

sappelhoff commented 4 months ago

@editorialbot set v0.5.1 as version

editorialbot commented 4 months ago

Done! version is now v0.5.1

odunbar commented 4 months ago

Thank you for the comments, I'll get to it.

Regarding the author name, I asked them to provide a name that they wished to be stated on the manuscript and this is what they provided, which I respected. I'm happy to ask them again (but I am only really comfortable changing it if they are however).

odunbar commented 4 months ago

@sappelhoff I've addressed your comments.

Please see https://github.com/CliMA/CalibrateEmulateSample.jl/pull/305 for the details of line-by-line changes etc.

The points I was unable to (fully) address

PS Do I need to release a new version and get a new Zenodo link now or is this OK?

sappelhoff commented 4 months ago

Thanks a lot @odunbar, I'll have a look later today.

PS Do I need to release a new version and get a new Zenodo link now or is this OK?

nope, this is fine as is, because the changes were all more in terms of the paper.

sappelhoff commented 4 months ago

@editorialbot generate pdf

editorialbot commented 4 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

sappelhoff commented 4 months ago

@editorialbot check references

editorialbot commented 4 months ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1137/141000671 is OK
- 10.1201/9781315117195 is OK
- 10.1201/9781315117195-8 is OK
- 10.1016/j.jcp.2020.109716 is OK
- 10.21105/joss.04869 is OK
- 10.1029/2022MS002994 is OK
- 1721.1/145140 is OK
- 10.1142/S0129065704001899 is OK
- 10.1088/0266-5611/29/4/045001 is OK
- 10.1109/allerton.2008.4797607 is OK
- 10.1109/TPAMI.2021.3097011 is OK
- 10.1214/13-STS421 is OK
- 10.1214/10-sts327 is OK
- 10.1029/2020MS002454 is OK
- 10.1029/2021MS002735 is OK
- 10.1029/2022MS002997 is OK
- 10.18637/jss.v102.i01 is OK
- 10.21105/joss.04561 is OK
- 10.1137/19M1304891 is OK
- 10.1093/bioinformatics/btaa078 is OK
- 10.1088/1361-6420/ac99fa is OK
- 10.1029/2022MS003245 is OK
- 10.22541/essoar.170365299.96491153/v1 is OK
- 10.1063/1.1699114 is OK
- 10.21105/joss.05428 is OK
- 10.1088/1475-7516/2023/10/021 is OK
- 10.1111/rssb.12482 is OK

MISSING DOIs

- No DOI given, and none found for title: Random Features for Large-Scale Kernel Machines.
- No DOI given, and none found for title: Scikit-learn: Machine Learning in Python
- No DOI given, and none found for title: The No-U-Turn sampler: adaptively setting path len...

INVALID DOIs

- None
sappelhoff commented 4 months ago

@editorialbot recommend-accept

editorialbot commented 4 months ago 
Attempting dry run of processing paper acceptance...
editorialbot commented 4 months ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1137/141000671 is OK
- 10.1201/9781315117195 is OK
- 10.1201/9781315117195-8 is OK
- 10.1016/j.jcp.2020.109716 is OK
- 10.21105/joss.04869 is OK
- 10.1029/2022MS002994 is OK
- 1721.1/145140 is OK
- 10.1142/S0129065704001899 is OK
- 10.1088/0266-5611/29/4/045001 is OK
- 10.1109/allerton.2008.4797607 is OK
- 10.1109/TPAMI.2021.3097011 is OK
- 10.1214/13-STS421 is OK
- 10.1214/10-sts327 is OK
- 10.1029/2020MS002454 is OK
- 10.1029/2021MS002735 is OK
- 10.1029/2022MS002997 is OK
- 10.18637/jss.v102.i01 is OK
- 10.21105/joss.04561 is OK
- 10.1137/19M1304891 is OK
- 10.1093/bioinformatics/btaa078 is OK
- 10.1088/1361-6420/ac99fa is OK
- 10.1029/2022MS003245 is OK
- 10.22541/essoar.170365299.96491153/v1 is OK
- 10.1063/1.1699114 is OK
- 10.21105/joss.05428 is OK
- 10.1088/1475-7516/2023/10/021 is OK
- 10.1111/rssb.12482 is OK

MISSING DOIs

- No DOI given, and none found for title: Random Features for Large-Scale Kernel Machines.
- No DOI given, and none found for title: Scikit-learn: Machine Learning in Python
- No DOI given, and none found for title: The No-U-Turn sampler: adaptively setting path len...

INVALID DOIs

- None
editorialbot commented 4 months ago

:warning: Error preparing paper acceptance. The generated XML metadata file is invalid. 

Element doi: [facet 'pattern'] The value '1721.1/145140' is not accepted by the pattern '10\.[0-9]{4,9}/.{1,200}'.
sappelhoff commented 4 months ago

@openjournals/dsais-eics I wanted to recommend this paper for acceptance, but I get the above error for a DOI: 1721.1/145140 ... However, this DOI does seem to resolve correctly: https://doi.org/1721.1/145140

How shall we proceed?