[REVIEW]: ronswanson: Building Table Models for 3ML

[editorialbot]

Submitting author: !--author-handle-->@grburgess<!--end-author-handle-- (J. Michael Burgess) Repository: https://github.com/grburgess/ronswanson Branch with paper.md (empty if default branch): Version: v0.2.10 Editor: !--editor-->@dfm<!--end-editor-- Reviewers: @cosimoNigro, @volodymyrss Archive: 10.5281/zenodo.7778865

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/b9ab6f687a9f3018b6b08c6575cfc6ee"><img src="https://joss.theoj.org/papers/b9ab6f687a9f3018b6b08c6575cfc6ee/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/b9ab6f687a9f3018b6b08c6575cfc6ee/status.svg)](https://joss.theoj.org/papers/b9ab6f687a9f3018b6b08c6575cfc6ee)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@cosimoNigro & @volodymyrss, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @dfm know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @volodymyrss

📝 Checklist for @cosimoNigro

[editorialbot]

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.88  T=0.10 s (485.2 files/s, 70510.1 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          30           1646           1301           3214
Markdown                         4            161              0            405
YAML                             9             82             19            282
TeX                              1              6              0             44
JSON                             1              8              0             33
INI                              1              4              0             20
reStructuredText                 3             10              9             11
TOML                             1              1              0             10
-------------------------------------------------------------------------------
SUM:                            50           1918           1329           4019
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository

[editorialbot]

Wordcount for paper.md is 833

[dfm]

@cosimoNigro, @volodymyrss — This is the review thread for the paper. All of our communications will happen here from now on. Thanks again for agreeing to participate!

👉 Please read the "Reviewer instructions & questions" in the first comment above, and generate your checklists by commenting @editorialbot generate my checklist on this issue ASAP. As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#4969 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for the review process to be completed within about 4-6 weeks but please try to make a start ahead of this as JOSS reviews are by their nature iterative and any early feedback you may be able to provide to the author will be very helpful in meeting this schedule.

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.5281/zenodo.5646925 is OK

MISSING DOIs

- 10.1007/10968987_3 may be a valid DOI for title: SLURM: Simple Linux Utility for Resource Management
- 10.22323/1.312.0130 may be a valid DOI for title: The Multi-Mission Maximum Likelihood framework (3ML)

INVALID DOIs

- None

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[volodymyrss]

Review checklist for @volodymyrss

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the https://github.com/grburgess/ronswanson?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@grburgess) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• [x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• [x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• [x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[volodymyrss]

@editorialbot commands

[editorialbot]

Hello @volodymyrss, here are the things you can ask me to do:

# List all available commands
@editorialbot commands

# Get a list of all editors's GitHub handles
@editorialbot list editors

# Check the references of the paper for missing DOIs
@editorialbot check references

# Perform checks on the repository
@editorialbot check repository

# Adds a checklist for the reviewer using this command
@editorialbot generate my checklist

# Set a value for branch
@editorialbot set joss-paper as branch

# Generates the pdf paper
@editorialbot generate pdf

# Generates a LaTeX preprint file
@editorialbot generate preprint

# Get a link to the complete list of reviewers
@editorialbot list reviewers

[cosimoNigro]

Review checklist for @cosimoNigro

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the https://github.com/grburgess/ronswanson?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@grburgess) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• [x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• [x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• [x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[dfm]

@cosimoNigro, @volodymyrss, @grburgess — Happy new year! I'm writing to check in on the progress of this review, and to keep it on your radars. Please let me know if there are any major stoppers or if there's anything I can do to help move things along. Thanks!

[volodymyrss]

@cosimoNigro, @volodymyrss, @grburgess — Happy new year! I'm writing to check in on the progress of this review, and to keep it on your radars. Please let me know if there are any major stoppers or if there's anything I can do to help move things along. Thanks!

Sorry, when I was going through check boxes I realized I want to request a change but it was not totally clear how. Just writing it here?..

[dfm]

@volodymyrss — Great question! As I mentioned briefly above, the usual workflow is to open issues on the project repo (it is useful to include a link back to this thread) with any suggested changes. You're also welcome to list smaller things directly here in this thread, but that can get things a little cluttered. Let me know if you have further questions!

[volodymyrss]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[grburgess]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[volodymyrss]

@editorialbot recommend-accept

for the version at this time

[editorialbot]

I'm sorry @volodymyrss, I'm afraid I can't do that. That's something only editors are allowed to do.

[volodymyrss]

I'm sorry @volodymyrss, I'm afraid I can't do that. That's something only editors are allowed to do.

hm, I just extrapolated ordinary review and did not read the guide again. Anyway, I checked all.

[dfm]

🎉 Thank you @volodymyrss! We'll wait on @cosimoNigro's review then I'll start with the next steps.

[cosimoNigro]

Hello @grburgess, @dfm, @volodymyrss,

Apologies for the delay in the review, I am really sorry for it.

Find here a resume of the issues I opened in the software repository and the corresponding items in the Review checklist left unchecked.

General checks

Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

https://github.com/grburgess/ronswanson/issues/6

Documentation

A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?

https://github.com/grburgess/ronswanson/issues/7

Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).

https://github.com/grburgess/ronswanson/issues/8

Software paper

I checked all the items of this part, here I write some comments on the text of the draft:

• Line 5: "ronswanson provides a simple to use framework" ... -> ronswanson is a python package providing a simple-to-use framework...;

• Line 7: "This allows for the construction of fast evaluating interpolation table of expensive computer simulations of physical models which can then be fitted to data in a reasonable amount of time." -> this part is confused, would be made more clear by moving here the sentence that goes from line 16 to line 20. That part of the text clearly states the problem at hand;

• Line 12: "Users can easily prototype their pipeline on multi-core workstations and then switch to a multi-node HPC system" -> so ronswanson can also generate scripts to parallelise the statistical analysis, beside the model evaluation? This is not clear;

• Line 16: "Spatio-spectral fitting" -> Spectro-morphological?

• Line 28: "While astromodels provides user friendly factories" -> user-friendly factories;

• Line 29: "the workflow for using these factories on desktop workstations or HPC systems can be complex." -> yet not clear if ronswanson can provide also script to parallelise the statistical analysis, beside the table model creation;

  • Line 35, 36: "it's output", "it's virtual run" -> its;

• Line 36-44: it would be nice if readers could visualise the process of model creation from the parameters. Perhaps adding a simple plot / scheme - the grid with the parameters values displayed in the documentation might be a good start;

• Line 45: "Additionally, the energy grid corresponding to the evaluation of each of the simulation outputs must be specified in this file." -> why is the energy specified, can ronswanson only create tables for spectral models?

• Line 52: "This automatically generates all the required python and SLURM scripts required for the construction of the table model" -> do you mean that they actually allow to evaluate the model for the different parameters and collect the different outputs? When does the interpolation that is supposed to speed up the statistical analysis happen? At this moment? After the table creation?

• Line 60: "an table model" -> a table model;

• Line 63: "that certain fits" -> that certain statistical analyses?

• Line 67: full stop missing.

P.S. In all citations the parentheses immediately follow the text without a space, is this a problem with \citep?

[grburgess]

@cosimoNigro no worries. Thank you for the edits. I will apply them shortly.

[dfm]

@grburgess — I wanted to check in on the status of your edits here. It looks like we're getting pretty close here! Let me know if there are any issues or anything I can do to help!

[grburgess]

@dfm I am just waiting for the replies to @cosimoNigro questions.

[dfm]

@grburgess — ah I see! Sorry, I didn't check over on those threads.

@cosimoNigro — it looks like @grburgess is waiting on some responses from you over on the ronswanson repo. Can you take a look at that when you get a chance? Thanks!

[cosimoNigro]

I just replied, I will try to be more reactive in the future.

[dfm]

@grburgess — Another ping here because I think this is back in your court now. Please let us know where things stand!

[grburgess]

@dfm @cosimoNigro sorry, this fell to the bottom of my list. I have now implemented the changes.

[cosimoNigro]

No problem, will try to take a look between today and tomorrow.

[cosimoNigro]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[cosimoNigro]

Thanks @grburgess!

For what concerns the software, all of my issues were addressed - I closed them.

For what concerns the manuscript, I would kindly ask you to consider the following improvements / corrections.

• Line 1: "simple to use framework" -> I think "simple to use" should be hyphenated (simple-to-use) as it is an adjective of framework;
• Line 2: "the modeling package for multi-messenger astrophysical data analysis framework, 3ML" -> the modeling package for the multi-messenger astrophysical data-analysis framework 3ML;
• Line 7: "This allows for the construction of fast evaluating interpolation table of expensive computer simulations of physical models which can then be fitted to data in a reasonable amount of time" -> Not clear, how about: "With astromodels and 3ML one can build the interpolation table of a physical model result of an expensive computer simulation. This then allow to fastly re-evaluate the model several times, for example while fitting it to a dataset.;
• Line 11: HPC acronym not introduced;
• Line 13: "ronswanson auto generates all python and SLURM scripts required for the construction of the table model." -> ronswanson automatically generates all python and SLURM scripts to scale the execution of 3ML with the astromodel's table models on a HPC system;
• Line 14: "Spatio-spectral fitting of astrophysical data many times requires using complex physical models whose output is that of an computationally expensive simulation" -> Not clear, how about: "Spatio-spectral fitting of astrophysical data might require the iterative evaluation of a complex physical model obtained from a computationally expensive simulation.";
• Line 35: "and collect it's output" -> its output;
• Line 36: "and defining it's virtual run member function" -> its virtual run member function;
• Line 67: a reference to Gamampy is missing here you have them.

@dfm I recommend acceptance (checked all the boxes in my checklist).

[dfm]

@cosimoNigro — Many thanks for your review and for the update!!

@grburgess — These recommendations from @cosimoNigro for the paper all sound like good additions. Can you ping me once you've incorporated them, and I'll proceed with the final processing. Thanks!

[grburgess]

@cosimoNigro @dfm I think it is done now.

[grburgess]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[dfm]

@cosimoNigro, @volodymyrss — Thanks for your thorough and constructive reviews!!

@grburgess — I've opened a small PR with some minor edits to the manuscript, please take a look and merge or let me know what you think.

Once you've done that:

1. Take one last read through the manuscript to make sure that you're happy with it (it's harder to make changes later!), especially the author names and affiliations. I've taken a pass and it looks good to me!
2. Increment the version number of the software and report that version number back here.
3. Create an archived release of that version of the software (using Zenodo or something similar). Please make sure that the metadata (title and author list) exactly match the paper. Then report the DOI of the release back to this thread.

[grburgess]

@dfm version: v0.2.10 doi: 10.5281/zenodo.7778865

thanks!

[dfm]

@editorialbot generate pdf

[dfm]

@editorialbot check references

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.5281/zenodo.5646925 is OK
- 10.5281/zenodo.7734804 is OK

MISSING DOIs

- 10.1007/10968987_3 may be a valid DOI for title: SLURM: Simple Linux Utility for Resource Management
- 10.22323/1.312.0130 may be a valid DOI for title: The Multi-Mission Maximum Likelihood framework (3ML)
- 10.22323/1.301.0766 may be a valid DOI for title: Gammapy - A prototype for the CTA science tools

INVALID DOIs

- None

[dfm]

@editorialbot set 10.5281/zenodo.7778865 as archive

[editorialbot]

Done! Archive is now 10.5281/zenodo.7778865

[dfm]

@editorialbot set v0.2.10 as version

[editorialbot]

Done! version is now v0.2.10

[dfm]

@grburgess — Thanks! Can you check the DOIs listed above and add them to the bibliography if they are correct?

[grburgess]

@dfm done!

[dfm]

@editorialbot check references