[REVIEW]: Zoobot: Adaptable Deep Learning Models for Galaxy Morphology

editorialbot commented 1 year ago

Submitting author: !--author-handle-->@mwalmsley@plaplant<!--end-editor-- Reviewers: @crhea93, @devanshkv Archive: 10.5281/zenodo.7896025

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/447561ee2de4709eddb704e18bee846f"><img src="https://joss.theoj.org/papers/447561ee2de4709eddb704e18bee846f/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/447561ee2de4709eddb704e18bee846f/status.svg)](https://joss.theoj.org/papers/447561ee2de4709eddb704e18bee846f)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@crhea93 & @devanshkv, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @plaplant know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @devanshkv

📝 Checklist for @crhea93

editorialbot commented 1 year ago

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

editorialbot commented 1 year ago

Software report:

github.com/AlDanial/cloc v 1.88  T=0.19 s (673.5 files/s, 76798.1 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          66           2064           3281           4749
Markdown                         7            130              0            419
reStructuredText                29            345            345            325
TeX                              1             23              0            209
Bourne Shell                    10             84            146            208
Jupyter Notebook                 2              0           1638            158
YAML                             6             15             15            107
DOS Batch                        1              8              1             26
Dockerfile                       1              7              7             18
make                             1              4              7              9
HCL                              1              4              4              6
TOML                             1              0              0              6
-------------------------------------------------------------------------------
SUM:                           126           2684           5444           6240
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository

editorialbot commented 1 year ago

Wordcount for paper.md is 1255

editorialbot commented 1 year ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.48550/arxiv.2206.11927 is OK
- 10.1093/mnras/stab2093 is OK
- 10.48550/arXiv.1110.3193 is OK
- 10.1093/mnras/stz2816 is OK
- 10.48550/ARXIV.1603.04467 is OK
- 10.1017/S1743921319008615 is OK
- 10.1017/pasa.2022.55 is OK
- 10.1038/nature14539 is OK
- 10.48550/ARXIV.2302.05442 is OK
- 10.48550/ARXIV.2104.10972 is OK
- 10.5281/zenodo.4414861 is OK
- 10.5281/zenodo.3828935 is OK
- 10.48550/ARXIV.1711.10604 is OK
- 10.48550/ARXIV.2303.00366 is OK

MISSING DOIs

- None

INVALID DOIs

- None

editorialbot commented 1 year ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

devanshkv commented 1 year ago

Review checklist for @devanshkv

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the https://github.com/mwalmsley/zoobot?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@mwalmsley) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
[x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
[x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
[x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

plaplant commented 1 year ago

@crhea93 thanks again for agreeing to review this submission! When you get a chance, please generate your reviewer checklist by replying to this thread with the comment

@editorialbot generate my checklist

Then, please read through the submission and update your checklist as you feel is appropriate. Let me know if you have any questions!

crhea93 commented 1 year ago

Review checklist for @crhea93

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the https://github.com/mwalmsley/zoobot?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@mwalmsley) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
[x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
[x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
[x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

crhea93 commented 1 year ago

The authors is missing references in the software paper.

crhea93 commented 1 year ago

The requirements file is incorrectly formatted. There is also no reference to the requirements.txt file in the installation.

plaplant commented 1 year ago

@crhea93 thanks for reviewing the software! For issues that should be addressed as part of the review, please open issues on the upstream repository. Thanks!

crhea93 commented 1 year ago

@editorialbot commands

editorialbot commented 1 year ago

Hello @crhea93, here are the things you can ask me to do:


# List all available commands
@editorialbot commands

# Get a list of all editors's GitHub handles
@editorialbot list editors

# Check the references of the paper for missing DOIs
@editorialbot check references

# Perform checks on the repository
@editorialbot check repository

# Adds a checklist for the reviewer using this command
@editorialbot generate my checklist

# Set a value for branch
@editorialbot set joss-paper as branch

# Generates the pdf paper
@editorialbot generate pdf

# Generates a LaTeX preprint file
@editorialbot generate preprint

# Get a link to the complete list of reviewers
@editorialbot list reviewers

crhea93 commented 1 year ago

@plaplant Should the issues have a specific template? I can't find one in the JOSS documentation.

mwalmsley commented 1 year ago

Hi @crhea93. Thanks for taking the time to review our JOSS submission!

The requirements file is incorrectly formatted. There is also no reference to the requirements.txt file in the installation.

The requirements.txt file is deprecated (the first line is """Deprecated - see setup.py""" and the entire file is commented out). I will remove it now for clarity. Installation dependencies are instead included via setup.py, which allows for more configuration (e.g. zoobot[pytorch], zoobot[tensorflow]).

The authors is missing references in the software paper.

Which references are missing? Do you mean missing as in 'should be cited' or 'not linked correctly to bibliography'?

I'm happy to chat directly on here about issues not related to the software itself, or you're welcome to open issues (I don't think a specific structure is required, it's only me reading them ;) )

plaplant commented 1 year ago

@crhea93 we do not have an official template, but thanks for checking! Sometimes reviewers make individual issues for each point they'd like addressed, and others make a single large "laundry list" issue. Either is fine, but my take is that having a separate issue for each point to be addressed can help keep the discussion more focused within the thread (and can potentially be addressed/fixed with multiple PRs).

In either case, you can link to the upstream issue(s) in this thread so everyone here can see the outstanding hurdles to acceptance. Thanks again for taking time to review!

mwalmsley commented 1 year ago

Hi reviewers @devanshkv @crhea93. Thank you for your feedback thus far.

I just wanted to highlight a few pieces of code that may be relevant to your remaining checklist items:

There is a Colab notebook for finetuning Zoobot. This is aimed at new users but can be used to verify the software works.
The models can be trained from scratch with using the scripts in /benchmarks to train the models from scratch. This is a manual functional test that new versions of Zoobot still perform well (and hopefully better). Results are uploaded to WandB and I'm happy to provide access to verify, if you like.
You can use the scripts to train most of the models yourself (though you will need some significant resources and potentially some adaption to your own cluster). All the data is provided in a separate repo, mwalmsley/galaxy-datasets, designed for easy use with ML. This includes GZ2, GZ CANDELS, GZ Hubble, and GZ DECALS. The only training data not yet available via that repo is GZ DESI, which is not original data in this paper - it is being published separately and is currently under review at MNRAS.
There are a few automated tests for some specific fiddly bits (see /tests) but mostly I find unit tests not that helpful for ML models. Having one-press slurm scripts to train the models has been much more useful to me.

crhea93 commented 1 year ago

@plaplant I've completed my review. I am very impressed with this work and @mwalmsley has cleared up any issues that I had. I recommend this code for publication.

plaplant commented 1 year ago

@crhea93 thanks very much for your review!

@devanshkv are there any outstanding issues that you feel need to be addressed prior to publication? If so, please make an issue on the upstream repository and reply to this thread so we're aware. Otherwise, if you feel this is ready for publication, please update your reviewer checklist accordingly. Thanks again for reviewing!

devanshkv commented 1 year ago

@crhea93, all good from my side.

Great work @mwalmsley! Looking forward to seeing how the community utilizes this. My only suggestion would be to have a more comprehensive test suite. Nevertheless, I recommend this code for publication.

plaplant commented 1 year ago

@devanshkv I see that there is still an unchecked item on your checklist about verifying the functional claims of the software. Please update your checklist accordingly if you have done this, or first verify the functionality before checking this box. (Per the JOSS reviewer docs, a review isn't complete until all items of the checklist have been ticked off.) Thanks!

devanshkv commented 1 year ago

@plaplant apologies, that was left unticked by mistake. All good from my side now.

plaplant commented 1 year ago

@devanshkv thanks for the quick response!

@mwalmsley both reviewers have recommended acceptance of the submission, so we can move forward with the final acceptance checks. If the software has changed in the course of the review, please make a new tagged release of the repository. After that, please archive it (on Zenodo, figshare, or some other long-term hosting platform). Once those are done, please post the version number and DOI of the archive in this thread. Let me know if you have any questions!

plaplant commented 1 year ago

@editorialbot generate pdf

editorialbot commented 1 year ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

plaplant commented 1 year ago

@editorialbot check references

editorialbot commented 1 year ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.48550/arxiv.2206.11927 is OK
- 10.1093/mnras/stab2093 is OK
- 10.48550/arXiv.1110.3193 is OK
- 10.1093/mnras/stz2816 is OK
- 10.48550/ARXIV.1603.04467 is OK
- 10.1017/S1743921319008615 is OK
- 10.1017/pasa.2022.55 is OK
- 10.1038/nature14539 is OK
- 10.48550/ARXIV.2302.05442 is OK
- 10.48550/ARXIV.2104.10972 is OK
- 10.5281/zenodo.4414861 is OK
- 10.5281/zenodo.3828935 is OK
- 10.48550/ARXIV.1711.10604 is OK
- 10.48550/ARXIV.2303.00366 is OK

MISSING DOIs

- None

INVALID DOIs

- None

plaplant commented 1 year ago

@mwalmsley looking at the references in the paper, it seems like the reference for Phan et al. does not have a DOI associated with it. For this paper, you can find it here, so please update the reference in paper.bib accordingly.

Also, it looks like the recommended reference for Pyro is this:

@article{bingham2019pyro,
  author    = {Eli Bingham and
               Jonathan P. Chen and
               Martin Jankowiak and
               Fritz Obermeyer and
               Neeraj Pradhan and
               Theofanis Karaletsos and
               Rohit Singh and
               Paul A. Szerlip and
               Paul Horsfall and
               Noah D. Goodman},
  title     = {Pyro: Deep Universal Probabilistic Programming},
  journal   = {J. Mach. Learn. Res.},
  volume    = {20},
  pages     = {28:1--28:6},
  year      = {2019},
  url       = {http://jmlr.org/papers/v20/18-403.html}
}

It does not seem like the journal version has a DOI associated with it, but I think this is still the better version to cite (as opposed to the arXiv version). Thanks!

mwalmsley commented 1 year ago

@editorialbot generate pdf

editorialbot commented 1 year ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

mwalmsley commented 1 year ago

Thanks for the references note @plaplant. I've added the DOI to Phan et al and updated the Pyro reference to the suggested reference.

I've created a new Zoobot tagged release, 1.0.3, which is versioned on Zenodo here. The all-versions DOI is 10.5281/zenodo.6483175

plaplant commented 1 year ago

@mwalmsley thanks for the info!

One last question: I notice that not all of the authors have an ORCID associated with them. Does everyone in the author list who has an ORCID have one listed in the paper metadata? (Note that not all authors need an ORCID, just wanted to make sure they were linked if they had one.) Also, please add author ORCIDs to the Zenodo metadata as well. Thanks very much!

plaplant commented 1 year ago

@editorialbot check references

editorialbot commented 1 year ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.48550/arxiv.2206.11927 is OK
- 10.1093/mnras/stab2093 is OK
- 10.48550/arXiv.1110.3193 is OK
- 10.1093/mnras/stz2816 is OK
- 10.48550/ARXIV.1603.04467 is OK
- 10.1017/S1743921319008615 is OK
- 10.1017/pasa.2022.55 is OK
- 10.1038/nature14539 is OK
- 10.48550/ARXIV.2302.05442 is OK
- 10.48550/ARXIV.2104.10972 is OK
- 10.5281/zenodo.4414861 is OK
- 10.48550/arXiv.1912.11554 is OK
- 10.5281/zenodo.3828935 is OK
- 10.48550/ARXIV.1711.10604 is OK
- 10.48550/ARXIV.2303.00366 is OK

MISSING DOIs

- None

INVALID DOIs

- None

plaplant commented 1 year ago

@mwalmsley sorry, one last change to the paper references: please include the DOI link for Bommasani et al. (2021). Note that a new version + Zenodo release is not necessary for this, as these archives are for the code rather than the paper. Thanks!

plaplant commented 1 year ago

@editorialbot set 10.5281/zenodo.7896025 as archive

editorialbot commented 1 year ago

Done! archive is now 10.5281/zenodo.7896025

plaplant commented 1 year ago

@editorialbot set v1.0.3 as version

editorialbot commented 1 year ago

Done! version is now v1.0.3

mwalmsley commented 1 year ago

Added DOI for Bommasani, thanks for noting.

On Thu, May 4, 2023 at 8:00 PM The Open Journals editorial robot < @.***> wrote:

Done! version is now v1.0.3

— Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/5312#issuecomment-1535258805, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB3BY3X37XEUBS2J2C4F5MDXEP4DJANCNFSM6AAAAAAWMUBWTM . You are receiving this because you were mentioned.Message ID: @.***>

plaplant commented 1 year ago

Thanks! I think there are just a few outstanding issues:

Please confirm that all of the co-authors with ORCIDs are the only ones who have one.
Please add the co-authors' ORCIDs to the Zenodo metadata.
Please adjust the title of the Zenodo archive to match the name of the paper ("Zoobot: Adaptable Deep Learning Models for Galaxy Morphology").

At that point I think the paper is ready for acceptance. Let me know if you have any questions!

mwalmsley commented 1 year ago

Brilliant, thank you!

I've added three additional ORCIDs following a manual search. The remaining co-authors without ORCIDs did not tell me their ORCID (only affiliations) and I can't identify any ORCID from the database. 2/3 I've updated the Zenodo details to match the JOSS paper details (it was first created last year as a simple code archive)

mwalmsley commented 1 year ago

@editorialbot generate pdf

editorialbot commented 1 year ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

plaplant commented 1 year ago

@mwalmsley fantastic! Thanks so much. I think this is ready to go.

plaplant commented 1 year ago

@editorialbot recommend-accept

editorialbot commented 1 year ago

Attempting dry run of processing paper acceptance...

editorialbot commented 1 year ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.48550/arxiv.2206.11927 is OK
- 10.1093/mnras/stab2093 is OK
- 10.48550/arXiv.1110.3193 is OK
- 10.1093/mnras/stz2816 is OK
- 10.48550/ARXIV.1603.04467 is OK
- 10.1017/S1743921319008615 is OK
- 10.1017/pasa.2022.55 is OK
- 10.1038/nature14539 is OK
- 10.48550/ARXIV.2302.05442 is OK
- 10.48550/arXiv.2108.07258 is OK
- 10.48550/ARXIV.2104.10972 is OK
- 10.5281/zenodo.4414861 is OK
- 10.48550/arXiv.1912.11554 is OK
- 10.5281/zenodo.3828935 is OK
- 10.48550/ARXIV.1711.10604 is OK
- 10.48550/ARXIV.2303.00366 is OK

MISSING DOIs

- None

INVALID DOIs

- None

editorialbot commented 1 year ago

The paper's PDF and metadata files generation produced some warnings that could prevent the final paper from being published. Please fix them before the end of the review process.

citation bingham2018pyro not found

editorialbot commented 1 year ago

:warning: Error preparing paper acceptance. The generated XML metadata file is invalid.

IDREFS attribute rid references an unknown ID "ref-bingham2018pyro"

openjournals / joss-reviews