editorialbot commented 10 months ago

Submitting author: !--author-handle-->@spinjet@kyleniemeyer<!--end-editor-- Reviewers: @e-dub, @jbussemaker Archive: 10.5281/zenodo.10732017

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/3fffd89737e3202ae555d7b4a47a4965"><img src="https://joss.theoj.org/papers/3fffd89737e3202ae555d7b4a47a4965/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/3fffd89737e3202ae555d7b4a47a4965/status.svg)](https://joss.theoj.org/papers/3fffd89737e3202ae555d7b4a47a4965)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@e-dub & @jbussemaker, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @kyleniemeyer know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @jbussemaker

📝 Checklist for @e-dub

editorialbot commented 10 months ago

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

editorialbot commented 10 months ago

Software report:

github.com/AlDanial/cloc v 1.88  T=0.05 s (433.6 files/s, 121303.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          13           1375           1205           2790
Markdown                         3             34              0             85
TeX                              1              8              0             73
YAML                             1              1              4             18
JSON                             2              0              0              2
-------------------------------------------------------------------------------
SUM:                            20           1418           1209           2968
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository

editorialbot commented 10 months ago

Wordcount for paper.md is 664

editorialbot commented 10 months ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1088/1757-899X/1226/1/012064 is OK
- 10.1088/1757-899X/1226/1/012064 is OK
- 10.3390/aerospace9030147 is OK
- 10.1088/1742-6596/2526/1/012021 is OK
- 10.2514/6.2023-0837 is OK

MISSING DOIs

- None

INVALID DOIs

- None

editorialbot commented 10 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

kyleniemeyer commented 10 months ago

👋🏼 @spinjet @e-dub, @jbussemaker this is the review thread for the paper. All of our communications will happen here from now on.

As a reviewer, the first step is to create a checklist for your review by entering

@editorialbot generate my checklist

as the top of a new comment in this thread.

These checklists contain the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. The first comment in this thread also contains links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#REVIEW_NUMBER so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 2-4 weeks. Please let me know if any of you require some more time. We can also use EditorialBot (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@kyleniemeyer) if you have any questions/concerns.

jbussemaker commented 10 months ago

Review checklist for @jbussemaker

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the https://github.com/spinjet/pdopt-code?
[x] License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@spinjet) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
[x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
[x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
[x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

kyleniemeyer commented 9 months ago

Hi @spinjet, @jbussemaker has made some comments as issues over in the software repo. Can you respond/address them when you get a chance?

spinjet commented 9 months ago

Dear @kyleniemeyer,

I apologise for the lack of response. We are currently near the university closure period and weren't able to address the suggestions for improvement brought forward by @jbussemaker due to other pressing issues.

We appreciate most of the suggestions and I'll be addressing them shortly, making improvements both on the paper and the code as soon as possible. Unfortunetly, I'll be on leave from tomorrow until the beginning of January, so I'll be slow on the response.

However, I can mention that the second author, Dr Kipouros, was my supervisor in my PhD, hence he's co-authoring despite no direct contribution in the software. I can directly mention him for confirmation @timosk .

Also, I believe the FutureWarning is caused by a pickled object that was created with a previous version of Pandas. This is however in the test case and should not affect the functionality of the software.

Thank you very much for your input and I'll be adding the improvements to the software and paper as soon as possible.

Kind regards, Andrea

kyleniemeyer commented 9 months ago

@spinjet thanks for the update, sounds good

kyleniemeyer commented 8 months ago

Hello @spinjet, I wanted to check in, as it's been about a month.

kyleniemeyer commented 8 months ago

For awareness, some discussion around updates to the paper is happening at https://github.com/spinjet/pdopt-code/issues/1

spinjet commented 8 months ago

Apologies @kyleniemeyer I got lost on where to provide the progress! I was writing mainly on my repository.

Automatic tests with CI has been completed now, so the list of important software engineering bullet points is now complete.

spinjet commented 8 months ago

Update: most of the code improvement suggestions have been addressed. https://github.com/spinjet/pdopt-code/issues/2 These should be completed by the end of tomorrow.

e-dub commented 8 months ago

Review checklist for @e-dub

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the https://github.com/spinjet/pdopt-code?
[x] License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@spinjet) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
[x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
[x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
[x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

editorialbot commented 8 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

spinjet commented 8 months ago

Dear @kyleniemeyer

I've begun addressing the points @jbussemaker and @e-dub have made on the paper. Currently I've added a State of the field section to provide some context on similar approaches for design space exploration.

Paper corrections is still WIP, and unfortunately next week I wont' be able to make any progress until Friday due to a workshop that will take most of the week.

Please let me know if any suggestions come through.

spinjet commented 8 months ago

Dear @kyleniemeyer I've completed the paper improvements, as required by the reviewers. https://github.com/spinjet/pdopt-code/issues/1 and https://github.com/spinjet/pdopt-code/issues/4

I think all the important steps for publication are completed. I'm going to have the code published on pyPI and have an online documentation for the code to polish the submission. Let me know if anything else is required.

kyleniemeyer commented 8 months ago

@editorialbot generate pdf

editorialbot commented 8 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

kyleniemeyer commented 8 months ago

@e-dub do the changes made address your comments? Also, wanted to see if you are able to complete your review soon.

jbussemaker commented 8 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

@spinjet this paper version doesn't have a state of the field section, and still only a few self-references. Were your improvements committed and pushed?

spinjet commented 8 months ago

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

@spinjet this paper version doesn't have a state of the field section, and still only a few self-references. Were your improvements committed and pushed?

I've committed and pushed the review improvements in this branch: https://github.com/spinjet/pdopt-code/tree/dev-joss-review I just realized the editorial bot is reading from the main, I'll merge them together so it should be visible from your end.

jbussemaker commented 8 months ago

Yeah when creating the submission originally you can also enter the paper branch ;)

Maybe it's possible to still change it?

kyleniemeyer commented 8 months ago

@editorialbot set dev-joss-review as branch

We can fix this! Totally fine to leave the paper in the branch as the review is ongoing (or permanently)

editorialbot commented 8 months ago

Done! branch is now dev-joss-review

kyleniemeyer commented 8 months ago

@editorialbot generate pdf

editorialbot commented 8 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

jbussemaker commented 8 months ago

@spinjet thank you for the updates to the paper, I think it has really improved the quality and made it more relevant for readers.

The only thing is that I wouldn't call the Surrogate Modeling Toolbox (SMT) a development of UoM: it's quite a large consortium, and the latest developments have been more lead by ONERA. Also, its purpose is not only DOE's, but also surrogate modeling (focusing on Gaussian Processes, which are relevant to PDOPT) of course ;)

After this, I'll be happy to recommend publication.

spinjet commented 8 months ago

Thank you for the feedback @jbussemaker.

I actually noticed there were more institutions tied to it when browsing the main page... I'll make the changes to the paper.

spinjet commented 7 months ago

I've pushed the changes to the paper @jbussemaker. I'm also working on the API documentation, albeit it's a bit tricky to have it compiled on readthedocs. Somehow in my local machine it works (I'm using mkdocs) but not on the hosted version.

I'll see tomorrow if I can get it working.

spinjet commented 7 months ago

Dear @jbussemaker @kyleniemeyer I have completed the online documentation, can be found here: https://pdopt-code.readthedocs.io/en/latest/ Currently it sits on the joss review branch.

There's only versioning and creating a PyPI submission left in the reviewer's checklist. I reckon once these are completed the review process is complete and ready for publication?

jbussemaker commented 7 months ago

@spinjet yes indeed! Don't forget to update the installation instructions too once you have a PyPI release ;)

kyleniemeyer commented 7 months ago

@spinjet that will satisfy the review checklist, but we'll have a few final steps before actually publishing in JOSS

e-dub commented 7 months ago

@e-dub do the changes made address your comments? Also, wanted to see if you are able to complete your review soon.

Nice work was done to address points. There are still some open points left and would ask @spinjet to reopen my issue and address these. Thanks :thumbsup:

spinjet commented 7 months ago

@e-dub do the changes made address your comments? Also, wanted to see if you are able to complete your review soon.

Nice work was done to address points. There are still some open points left and would ask @spinjet to reopen my issue and address these. Thanks 👍

I've re-opened it, it got closed accidentally when I merged the dev branch. Apologies for this happening!

spinjet commented 7 months ago

https://github.com/spinjet/pdopt-code/issues/4#issuecomment-1942157939

Paper updated with the comments of the second reviewer.

jbussemaker commented 7 months ago

@kyleniemeyer all items in my checklist and issues have been addressed, I support publication of the JOSS paper :+1:

kyleniemeyer commented 7 months ago

Thanks @jbussemaker!

@e-dub, do the changes made in https://github.com/spinjet/pdopt-code/issues/4 address your comments?

kyleniemeyer commented 7 months ago

@editorialbot generate pdf

editorialbot commented 7 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

e-dub commented 7 months ago

@kyleniemeyer my review checklist is complete and issues in my issue have been addressed. I support publication of the JOSS article.

kyleniemeyer commented 7 months ago

Post-Review Checklist for Editor and Authors

Additional Author Tasks After Review is Complete

Double check authors and affiliations (including ORCIDs)
Make a release of the software with the latest changes from the review and post the version number here. This is the version that will be used in the JOSS paper.
Archive the release on Zenodo/figshare/etc and post the DOI here.
Make sure that the title and author list (including ORCIDs) in the archive match those in the JOSS paper.
Make sure that the license listed for the archive is the same as the software license.

Editor Tasks Prior to Acceptance

[x] Read the text of the paper and offer comments/corrections (as either a list or a pull request)
[x] Check that the archive title, author list, version tag, and the license are correct
[x] Set archive DOI with @editorialbot set <DOI here> as archive
[x] Set version with @editorialbot set <version here> as version
[x] Double check rendering of paper with @editorialbot generate pdf
[x] Specifically check the references with @editorialbot check references and ask author(s) to update as needed
[ ] Recommend acceptance with @editorialbot recommend-accept

kyleniemeyer commented 7 months ago

Hi @spinjet, both reviewers have recommended acceptance, so now we have the above steps to complete. Please make a checklist for your tasks, and I'll start working through mine.

kyleniemeyer commented 7 months ago

@spinjet I'll add that at this point, the software will not change, so you can merge the reviewed branch (if that applies). The paper may change, but doesn't need to be in the final archive for the software—that can remain in a separate paper-only branch if desired.

spinjet commented 7 months ago

Post-Review Checklist for Editor and Authors

Additional Author Tasks After Review is Complete

[x] Double check authors and affiliations (including ORCIDs)
[x] Make a release of the software with the latest changes from the review and post the version number here. This is the version that will be used in the JOSS paper.
[x] Archive the release on Zenodo/figshare/etc and post the DOI here.
[x] Make sure that the title and author list (including ORCIDs) in the archive match those in the JOSS paper.
[x] Make sure that the license listed for the archive is the same as the software license.
[x] Merge the JOSS review branch into main, create a paper-only branch for tidyness.

Hi @kyleniemeyer, I am glad the submission has been accepted, I will go through my checklist as soon as possible.

spinjet commented 7 months ago

@editorialbot set joss-paper as branch

editorialbot commented 7 months ago

Done! branch is now joss-paper

spinjet commented 7 months ago

Dear @kyleniemeyer,

I've completed the checklist.

The released version for the JOSS publication is the latest one (v0.5.1) https://github.com/spinjet/pdopt-code/releases/tag/0.5.1 This has been archived on Zenodo: https://zenodo.org/doi/10.5281/zenodo.10732017

I hope this satisfies every requirement for publication.

kyleniemeyer commented 7 months ago

@editorialbot set 0.5.1 as version

openjournals / joss-reviews

[REVIEW]: PDOPT: A Python library for Probabilistic Design space exploration and OPTimisation. #6110

Status

Reviewer instructions & questions

Checklists

Review checklist for @jbussemaker

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @e-dub

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Post-Review Checklist for Editor and Authors

Additional Author Tasks After Review is Complete

Editor Tasks Prior to Acceptance

Post-Review Checklist for Editor and Authors

Additional Author Tasks After Review is Complete