search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
696 stars 36 forks source link

[REVIEW]: UQTestFuns: A Python3 library of uncertainty quantification (UQ) test functions #5671

Closed editorialbot closed 8 months ago

editorialbot commented 12 months ago

Submitting author: !--author-handle-->@damar-wicaksono<!--end-author-handle-- (Damar Wicaksono) Repository: https://github.com/damar-wicaksono/uqtestfuns Branch with paper.md (empty if default branch): dev-joss Version: v0.4.1 Editor: !--editor-->@Nikoleta-v3<!--end-editor-- Reviewers: @Himscipy, @AnjaliSandip Archive: 10.5281/zenodo.10047512

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/cc63dbaea06ee71e1effea1dcf204679"><img src="https://joss.theoj.org/papers/cc63dbaea06ee71e1effea1dcf204679/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/cc63dbaea06ee71e1effea1dcf204679/status.svg)](https://joss.theoj.org/papers/cc63dbaea06ee71e1effea1dcf204679)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@Himscipy & @AnjaliSandip, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @Nikoleta-v3 know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @AnjaliSandip

📝 Checklist for @Himscipy

editorialbot commented 12 months ago

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf
editorialbot commented 12 months ago 
Software report:

github.com/AlDanial/cloc v 1.88  T=0.15 s (1176.8 files/s, 183541.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Markdown                        80           2937              0          11558
Python                          85           2490           4103           5543
TeX                              2             95              0            849
YAML                             6             16             16            374
reStructuredText                 6             32             31             22
TOML                             1              1              0              6
-------------------------------------------------------------------------------
SUM:                           180           5571           4150          18352
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository
editorialbot commented 12 months ago

Wordcount for paper.md is 2347

editorialbot commented 12 months ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.7712/120221.8034.19093 is OK
- 10.1007/978-3-319-12385-1_64 is OK
- 10.23919/eucap.2017.7928679 is OK
- 10.3850/978-981-07-2219-7_p321 is OK
- 10.1007/978-1-4899-7547-8_5 is OK
- 10.1007/978-1-4471-6269-8_8 is OK
- 10.1016/j.strusafe.2018.02.005 is OK
- 10.1002/9780470725184 is OK
- 10.1016/j.advengsoft.2019.03.005 is OK
- 10.1016/0951-8320(95)00099-2 is OK
- 10.21105/joss.00097 is OK
- 10.18174/sesmo.18155 is OK
- 10.1002/cnm.2755 is OK
- 10.13182/nse16-37 is OK
- 10.1137/16m1061928 is OK
- 10.1016/j.ress.2023.109211 is OK
- 10.1038/s41592-019-0686-2 is OK
- 10.1038/s41586-020-2649-2 is OK

MISSING DOIs

- None

INVALID DOIs

- None
editorialbot commented 12 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

Nikoleta-v3 commented 12 months ago

👋🏼 @damar-wicaksono, @Himscipy, @AnjaliSandip this is the review thread for the paper. All of our communications will happen here from now on.

As a reviewer, the first step is to create a checklist for your review by entering

@editorialbot generate my checklist

as the top of a new comment in this thread.

These checklists contain the JOSS requirements ✅ As you go over the submission, please check any items that you feel have been satisfied. The first comment in this thread also contains links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention https://github.com/openjournals/joss-reviews/issues/5671 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 2-4 weeks. Please let me know if any of you require some more time. We can also use EditorialBot (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@Nikoleta-v3) if you have any questions/concerns. 😄 🙋🏻

damar-wicaksono commented 12 months ago

👋 Thank you very much @Nikoleta-v3 for handling the submission and @Himscipy and @AnjaliSandip for agreeing to review the submission. I'm looking forward to your review and feedback.

I'll be off for 1.5 week but suppose to have an internet access. During the period, I'll try to reply and address any issues you raised but perhaps with a bit of delay.

Thank you very much again!

AnjaliSandip commented 11 months ago

Review checklist for @AnjaliSandip

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Nikoleta-v3 commented 11 months ago

Enjoy your break! 👋🏻

👋 Thank you very much @Nikoleta-v3 for handling the submission and @Himscipy and @AnjaliSandip for agreeing to review the submission. I'm looking forward to your review and feedback.

I'll be off for 1.5 week but suppose to have an internet access. During the period, I'll try to reply and address any issues you raised but perhaps with a bit of delay.

Thank you very much again!

AnjaliSandip commented 11 months ago

This article presents an open-source Python3 library of test functions commonly used within the applied uncertainty quantification (UQ) community. The authors do a good job of explaining the installation and usage of the open-source software. I suggest adding a few more examples to the repository. The software paper is well written. The second author’s contribution to the software and the paper should be made clear. One minor issue I had with the documentation was it does not explicitly address what problems the software is designed to solve. Once the issues stated above have been addressed, I support publication.

AnjaliSandip commented 11 months ago

@Nikoleta-v3 I have completed the review.

damar-wicaksono commented 10 months ago

@AnjaliSandip, thank you very much for your thorough review and helpful suggestions to improve the package!

I opened two issues in the repo related to your comments above and will deal with them as soon as possible:

And regarding the second author's contribution to the paper and package:

I conceived and coded UQTestFuns with the support of a research project of M. Hecht. Furthermore, M. Hecht's contributions have been indispensable via conceptualization of the package (especially at the early stage), reviewing the high-level interface at the later stage, general supervision of the project (including direction and timeline). Finally, M. Hecht has contributed to the final version of the manuscript. I believe his contributions are sufficient to be a co-author of this submission and hope that this answers your question regarding the contributions of the authors.

damar-wicaksono commented 10 months ago

@AnjaliSandip, I tried to address your comments above and updated the documentation of the package. In particular:

For comparison the previous docs are still online.

Please let me know of any further comments and feedbacks! Thanks a lot!

AnjaliSandip commented 10 months ago

@damar-wicaksono Thank you for addressing the comments. I will suggest adding a section describing briefly the authors' contributions to the manuscript and the software developed.

I support the manuscript for publication.

damar-wicaksono commented 10 months ago

@AnjaliSandip: Thank you very much for the prompt response!

@Nikoleta-v3: Is it okay to add another section in the draft manuscript giving a statement of the authors' contributions?

Nikoleta-v3 commented 10 months ago

Hey @damar-wicaksono 😄👋🏻 Yup, please go ahead and add the section. Several other publications in JOSS include a contribution section as well.

Himscipy commented 10 months ago

Review checklist for @Himscipy

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Nikoleta-v3 commented 10 months ago

Hey @Himscipy 👋🏻 Do you have any updates on your review? 😄

damar-wicaksono commented 9 months ago

@editorialbot check references

editorialbot commented 9 months ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.7712/120221.8034.19093 is OK
- 10.1007/978-3-319-12385-1_64 is OK
- 10.23919/eucap.2017.7928679 is OK
- 10.3850/978-981-07-2219-7_p321 is OK
- 10.1007/978-1-4899-7547-8_5 is OK
- 10.1007/978-1-4471-6269-8_8 is OK
- 10.1016/j.strusafe.2018.02.005 is OK
- 10.1002/9780470725184 is OK
- 10.1016/j.advengsoft.2019.03.005 is OK
- 10.1016/0951-8320(95)00099-2 is OK
- 10.21105/joss.00097 is OK
- 10.18174/sesmo.18155 is OK
- 10.1002/cnm.2755 is OK
- 10.13182/nse16-37 is OK
- 10.1137/16m1061928 is OK
- 10.1016/j.ress.2023.109211 is OK
- 10.1038/s41592-019-0686-2 is OK
- 10.1038/s41586-020-2649-2 is OK

MISSING DOIs

- None

INVALID DOIs

- None
damar-wicaksono commented 9 months ago

@editorialbot generate pdf

editorialbot commented 9 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

damar-wicaksono commented 9 months ago

Hi @AnjaliSandip, @Nikoleta-v3: as suggested I've updated the acknowledgments section of the draft manuscript with authors' contribution statement (according to CRediT). A new draft manuscript has been generated (see above). Thank you!

AnjaliSandip commented 9 months ago

@damar-wicaksono Looks good. I support the manuscript for publication.

Nikoleta-v3 commented 9 months ago

@Himscipy 👋🏻 😄 📳

Nikoleta-v3 commented 9 months ago

@damar-wicaksono, just to keep you up to date, I have reached out to the second reviewer by email, and I am waiting for their response😄

damar-wicaksono commented 9 months ago

Hi @Nikoleta-v3, thank you very much for the update! Indeed, I was going to ask you what would be the next step as your last inquiry via GitHub (here) was about two weeks ago.

Nikoleta-v3 commented 9 months ago

Let's give the reviewer a bit more time 😄 and in the meantime I will do my review of the package! Thank you for your patience 🙏🏻

Nikoleta-v3 commented 9 months ago

Hey @damar-wicaksono,

I've now reviewed the paper. I can confirm that the package installed smoothly, and I had no issues running the tests. I really appreciate the contributions file. It's very detailed and beginner-friendly.

Although this isn't my field of expertise, from a development standpoint, the package appears to be exceptionally well-developed.

I don't think I've seen another package use Jupyter Books for documentation. I think it works very well! The documentation includes a lot of information, which could be overwhelming, but it's structured very well now. Anja's comment and suggestion regarding this was spot-on.

I'm curious, have you ever used the documentation/package for teaching at your institute or collaborating universities?

damar-wicaksono commented 9 months ago

Hi @Nikoleta-v3,

Thank you very much for taking the time to review the paper and the package!

I don't think I've seen another package use Jupyter Books for documentation. I think it works very well!

Thank you! Actually, this is also my first experience of using Jupyter Book to create a documentation for a package (having previously used Sphinx, the community standard). For the current package, I think Jupyter Book provides all what we need out-of-the-box without having to install bunch of supporting plugins.

The documentation includes a lot of information, which could be overwhelming, but it's structured very well now. Anja's comment and suggestion regarding this was spot-on.

Indeed, we really appreciate @AnjaliSandip's feedbacks on that; it can be difficult for the developer to see which parts are missing in order to make a connection to different parts of the docs, especially if one is new to the package. I'm glad that it now works better, at least, for a knowledgeable new user.

I'm curious, have you ever used the documentation/package for teaching at your institute or collaborating universities?

For teaching per se, not yet. But one of the reasons we are developing the package is also because when we introduce a method for, say, sensitivity analysis in a classroom setting we can conveniently use one of the known test functions from the literature implemented in the package as an example or illustration. We are planning to have such a session in our institute in the future.

That being said, there's a student project in our group (an investigation of a metamodeling/surrogate modeling method) that employed the test functions of the package to benchmark the proposed method. Convenient access to test functions for testing is one of the main goals of the package.

Nikoleta-v3 commented 9 months ago

Hello @damar-wicaksono. Thank you very much for your reply!

For teaching per se, not yet. But one of the reasons we are developing the package is also because when we introduce a method for, say, sensitivity analysis in a classroom setting, we can conveniently use one of the known test functions from the literature implemented in the package as an example or illustration. We are planning to have such a session in our institute in the future.

That's amazing. I think it would be great to eventually have a dedicated section in the documentation about the various ways you can use the package in teaching, based on your experiences.

That being said, there's a student project in our group (an investigation of a metamodeling/surrogate modeling method) that employed the test functions of the package to benchmark the proposed method. Convenient access to test functions for testing is one of the main goals of the package.

And, as you mentioned here, eventually see the package being cited in such projects!

I am very happy to say that I have no further comments at this point. I have already spoken to your second reviewer, and they should be reviewing the paper soon. Thank you again for your patience! 🙏🏻

Himscipy commented 8 months ago

Hello @damar-wicaksono, I found UQTestFuns an interesting software tool for training students on doing UQ for scientific applications. Based on the current implementation it will require additional capabilities to be generalized for doing UQ for black-box functions or complex scientific application.

The package is well documented and checks all the review criteria according to my review. I agree with @Nikoleta-v3 that the documentation is well structured to guide the new user about the package. In addition, the revisions made based on @AnjaliSandip comments have improved the paper and the repository.

I have completed my review and accept the repo for publication. Thank you again for your patience !

Nikoleta-v3 commented 8 months ago

Thank you all for your time and help with this submission 🙏🏻

Nikoleta-v3 commented 8 months ago

@damar-wicaksono could you please:

I can then move forward with accepting the submission.

damar-wicaksono commented 8 months ago

@AnjaliSandip, @Himscipy, and @Nikoleta-v3 thank you very much for your time reviewing the package and the paper, and providing me with valuable feedback!

@Nikoleta-v3: I will make a new tagged (minor) version of the software to incorporate the feedback in the new release. I will update you here when I finish. Thanks again!

damar-wicaksono commented 8 months ago

@Nikoleta-v3: by the way, should I modify the metadata of the paper to include you as the editor as well as the reviewers? It seems, the editorialbot is doing that to compile the draft version posted here...

Nikoleta-v3 commented 8 months ago

@Nikoleta-v3: by the way, should I modify the metadata of the paper to include you as the editor as well as the reviewers? It seems, the editorialbot is doing that to compile the draft version posted here...

No you don't have to change anything 😃

damar-wicaksono commented 8 months ago

@editorialbot check references

editorialbot commented 8 months ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.7712/120221.8034.19093 is OK
- 10.1007/978-3-319-12385-1_64 is OK
- 10.23919/eucap.2017.7928679 is OK
- 10.3850/978-981-07-2219-7_p321 is OK
- 10.1007/978-1-4899-7547-8_5 is OK
- 10.1007/978-1-4471-6269-8_8 is OK
- 10.1016/j.strusafe.2018.02.005 is OK
- 10.1002/9780470725184 is OK
- 10.1016/j.advengsoft.2019.03.005 is OK
- 10.1016/0951-8320(95)00099-2 is OK
- 10.21105/joss.00097 is OK
- 10.18174/sesmo.18155 is OK
- 10.1002/cnm.2755 is OK
- 10.13182/nse16-37 is OK
- 10.1137/16m1061928 is OK
- 10.1016/j.ress.2023.109211 is OK
- 10.1038/s41592-019-0686-2 is OK
- 10.1038/s41586-020-2649-2 is OK

MISSING DOIs

- None

INVALID DOIs

- None
damar-wicaksono commented 8 months ago

@editorialbot generate pdf

editorialbot commented 8 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

damar-wicaksono commented 8 months ago

@Nikoleta-v3:

Please let me know if there's anything else I should do. Thanks a lot!

Nikoleta-v3 commented 8 months ago

@editorialbot set 10.5281/zenodo.10047512 as archive

editorialbot commented 8 months ago

Done! archive is now 10.5281/zenodo.10047512

Nikoleta-v3 commented 8 months ago

@editorialbot set v0.4.1 as version

editorialbot commented 8 months ago

Done! version is now v0.4.1

Nikoleta-v3 commented 8 months ago

@editorialbot recommend-accept

editorialbot commented 8 months ago 
Attempting dry run of processing paper acceptance...
editorialbot commented 8 months ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.7712/120221.8034.19093 is OK
- 10.1007/978-3-319-12385-1_64 is OK
- 10.23919/eucap.2017.7928679 is OK
- 10.3850/978-981-07-2219-7_p321 is OK
- 10.1007/978-1-4899-7547-8_5 is OK
- 10.1007/978-1-4471-6269-8_8 is OK
- 10.1016/j.strusafe.2018.02.005 is OK
- 10.1002/9780470725184 is OK
- 10.1016/j.advengsoft.2019.03.005 is OK
- 10.1016/0951-8320(95)00099-2 is OK
- 10.21105/joss.00097 is OK
- 10.18174/sesmo.18155 is OK
- 10.1002/cnm.2755 is OK
- 10.13182/nse16-37 is OK
- 10.1137/16m1061928 is OK
- 10.1016/j.ress.2023.109211 is OK
- 10.1038/s41592-019-0686-2 is OK
- 10.1038/s41586-020-2649-2 is OK

MISSING DOIs

- None

INVALID DOIs

- None
editorialbot commented 8 months ago

The paper's PDF and metadata files generation produced some warnings that could prevent the final paper from being published. Please fix them before the end of the review process.

\mathsemicolon
              ^
unexpected control sequence \mathsemicolon
expecting "%", "\\label", "\\tag", "\\nonumber" or whitespace