openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
712 stars 38 forks source link

[REVIEW]: lenstronomy II: A gravitational lensing software ecosystem #3283

Closed whedon closed 3 years ago

whedon commented 3 years ago

Submitting author: @sibirrer (Simon Birrer) Repository: https://github.com/sibirrer/lenstronomy Version: 1.8.2_JOSS Editor: @danielskatz Reviewer: @smsharma, @coljac Archive: 10.5281/zenodo.4913700

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/6a562375312c9a9e4466912a16f27589"><img src="https://joss.theoj.org/papers/6a562375312c9a9e4466912a16f27589/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/6a562375312c9a9e4466912a16f27589/status.svg)](https://joss.theoj.org/papers/6a562375312c9a9e4466912a16f27589)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@smsharma & @coljac, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @danielskatz know.

Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest

Review checklist for @smsharma

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @coljac

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 3 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @smsharma, @coljac it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 3 years ago

PDF failed to compile for issue #3283 with the following error:

 Can't find any papers to compile :-(
danielskatz commented 3 years ago

@smsharma and @coljac - Thanks for agreeing to review this submission. This is the review thread for the paper. All of our communications will happen here from now on.

Both reviewers have checklists at the top of this thread with the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

Please read the first couple of comments in this issue carefully, so that you can accept the invitation from JOSS and be able to check items, and so that you don't get overwhelmed with notifications from other activities in JOSS.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#3283 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 2-4 weeks. Please let me know if either of you require some more time. We can also use Whedon (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@danielskatz) if you have any questions/concerns.

whedon commented 3 years ago
Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=3.02 s (235.4 files/s, 57340.1 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
HTML                           231           8777              0          82405
Python                         415          10161          16077          38117
JavaScript                      10           2229           2248           8871
reStructuredText                42            761            952           1078
CSS                              4            199             42            766
make                             2             39              6            189
Markdown                         1             39              0             89
XML                              1             10              0             53
INI                              1              4              0             36
YAML                             3              8             22             35
Ruby                             1              0              0              2
-------------------------------------------------------------------------------
SUM:                           711          22227          19347         131641
-------------------------------------------------------------------------------

Statistical information for the repository 'f8e639d75559d63621df7284' was
gathered on 2021/05/13.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Anowar Jaman Shajib            124          5701           3132            3.21
Aymeric Galan                  220         11415           6456            6.50
AymericG                         1            57              1            0.02
Daniel Gilman                  183         10082           7874            6.53
Ewoud                            2            13             18            0.01
Ewoud Wempe                     20           481            133            0.22
Giulia Pagano                   11          1016             83            0.40
Jelle Aalbers                   17           927             69            0.36
Luca Teodori                    26          1778           1140            1.06
Lyne VdV                         4           172              6            0.06
LyneVdV                          7           455             69            0.19
Madi Ueland                     15           738             91            0.30
Nicolas Tessore                  2           357              3            0.13
Robert Morgan                    4           173              5            0.06
Sebastian Wagner-Car             3            32             30            0.02
Simon Birrer                   557         81553          40247           44.32
ajshajib                         1            41             12            0.02
dangilman                       24          3242           1297            1.65
jiwoncpark                      11           109            101            0.08
lilan                           11           256             45            0.11
martin-millon                   16            50            571            0.23
sibirrer                       860         58299          36468           34.49

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Anowar Jaman Shajib        2766           48.5         21.0                8.89
Aymeric Galan              3299           28.9         16.3                8.88
Daniel Gilman              3792           37.6         14.0                6.94
Ewoud                         9           69.2          7.3               11.11
Ewoud Wempe                 354           73.6          2.5                5.93
Giulia Pagano               927           91.2         10.5               19.42
Jelle Aalbers               858           92.6          6.8                4.78
Luca Teodori                643           36.2          4.3               10.73
Lyne VdV                    144           83.7          8.7                6.94
LyneVdV                     369           81.1          4.7               10.03
Madi Ueland                 648           87.8          9.2                9.72
Nicolas Tessore             289           81.0         11.9               16.26
Robert Morgan               168           97.1          8.1                9.52
Sebastian Wagner-Car         29           90.6          2.4                3.45
Simon Birrer              19918           24.4         37.7               12.04
jiwoncpark                   78           71.6         14.8                6.41
lilan                       203           79.3         22.9                7.88
martin-millon                 9           18.0         16.1                0.00
sibirrer                  43210           74.1         19.2               10.05
danielskatz commented 3 years ago

@whedon generate pdf from branch JOSS_paper

whedon commented 3 years ago
Attempting PDF compilation from custom branch JOSS_paper. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

danielskatz commented 3 years ago

@whedon check references from branch JOSS_paper

whedon commented 3 years ago
Attempting to check references... from custom branch JOSS_paper
whedon commented 3 years ago
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.dark.2018.11.002 is OK
- 10.1088/0004-637X/813/2/102 is OK
- 10.21105/joss.02825 is OK
- 10.1093/mnras/sty1264 is OK
- 10.3847/0004-637X/826/2/112 is OK
- 10.3847/1538-4357/abdf59 is OK
- 10.1093/mnras/stab984 is OK
- 10.1051/0004-6361/202038730 is OK
- 10.1051/0004-6361/202039363 is OK
- 10.3847/1538-4357/abdfc4 is OK
- 10.1051/0004-6361/201731042 is OK
- 10.1051/0004-6361/201526773 is OK
- 10.1093/mnras/stz1593 is OK
- 10.1093/mnras/stab484 is OK
- 10.1103/PhysRevD.102.063502 is OK
- 10.21105/joss.02854 is OK
- 10.1093/mnras/stz2254 is OK
- 10.1051/0004-6361/201937351 is OK
- 10.1051/0004-6361/202038942 is OK
- 10.1111/j.1365-2966.2009.14654.x is OK
- 10.1088/1475-7516/2021/04/010 is OK
- 10.1093/mnras/sty3397 is OK
- 10.1088/1475-7516/2017/04/049 is OK
- 10.1093/mnras/stab532 is OK
- 10.1093/mnras/staa1649 is OK
- 10.1093/mnras/staa3713 is OK
- 10.1093/mnras/stz1796 is OK
- 10.1088/1475-7516/2016/08/020 is OK
- 10.1093/mnras/stz200 is OK
- 10.3847/1538-4357/ab5b90 is OK
- 10.1093/mnras/stab536 is OK
- 10.1088/0004-637X/800/2/94 is OK
- 10.1093/mnras/staa1600 is OK
- 10.1093/mnras/stz3480 is OK
- 10.1093/mnras/stz3177 is OK
- 10.1038/nature10669 is OK
- 10.3847/0004-637X/823/1/37 is OK
- 10.1088/1475-7516/2017/05/037 is OK
- 10.1111/j.1365-2966.2010.16639.x is OK
- 10.1088/0004-637X/811/1/20 is OK
- 10.1051/0004-6361/202038861 is OK
- 10.1093/mnras/staa828 is OK
- 10.1093/mnras/stz3094 is OK
- 10.3847/1538-4357/ab042c is OK
- 10.1038/s41550-017-0295-x is OK
- 10.1038/nature11446 is OK
- 10.1109/JPROC.2009.2021005 is OK
- 10.3847/1538-4365/abd62c is OK
- 10.1111/j.1365-2966.2006.10040.x is OK
- 10.1051/0004-6361/201322068 is OK
- 10.3847/1538-3881/aabc4f is OK

MISSING DOIs

- 10.1093/mnras/stab1240 may be a valid DOI for title: Improved time-delay lens modelling and H_0 inference with transient sources

INVALID DOIs

- None
danielskatz commented 3 years ago

👋 @sibirrer - I forgot to ask before: Is this submission associated with a paper submitted to a AAS journal?

sibirrer commented 3 years ago

👋 @sibirrer - I forgot to ask before: Is this submission associated with a paper submitted to a AAS journal?

@danielskatz No, this is a stand-alone submission. There was an original paper in 2018 (Physics of the Dark Universe) (without code review, highlighting the design feature).

danielskatz commented 3 years ago

Thanks - it doesn't really matter in terms of the review process, but it would lead to a minor difference in the publication at the end - see https://blog.joss.theoj.org/2018/12/a-new-collaboration-with-aas-publishing for details

sibirrer commented 3 years ago

I see, thanks for the pointer! Was not fully aware of this, but also think that a full-fledged paper is too much work in writing. I hear your argument of financial support through AAS submissions. Perhaps future papers that are responsible for implementing specific features in lenstronomy might be worth going through a JOSS review (it would only be specific modules of lenstronomy). This would both ensure a steady high quality of added features and a rewarding structure for both JOSS and the authors that implemented specific features (it's not just me at this point). It's just a thought but it would also also put more work on JOSS and more frequent reviews.

danielskatz commented 3 years ago

Note that one of our (JOSS) sensitivities is in being sure that submissions are substantial units of work, in some equivalent to a science paper, so this might be an issue. In any case, the question has been answered for this submission :)

coljac commented 3 years ago

@danielskatz I'm just turning to this review. I thought I accepted the invite but it appears I did not and it has expired. Can this be reissued?

danielskatz commented 3 years ago

@whedon re-invite @coljac as reviewer

whedon commented 3 years ago

OK, the reviewer has been re-invited.

@coljac please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

whedon commented 3 years ago

:wave: @coljac, please update us on how your review is going (this is an automated reminder).

whedon commented 3 years ago

:wave: @smsharma, please update us on how your review is going (this is an automated reminder).

smsharma commented 3 years ago

@danielskatz Likewise would it be possible to re-issue the invitation? Thank you!

danielskatz commented 3 years ago

@Whedon re-invite @smsharma as reviewer

whedon commented 3 years ago

OK, the reviewer has been re-invited.

@smsharma please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

coljac commented 3 years ago

OK, sorry it took so long to get this done.

This is in easy review in that Lenstronomy is a mature, feature-rich and widely used package. The scientific significance is not in doubt, and a lot of time and effort has been invested into the code, docs and examples. It shows. Congrats all round to the authors!

I have no issues with the paper.

@sbirrer, could you clarify these final issues for me:

Barring the above, which are very minor, I'm happy to sign off.

coljac commented 3 years ago

Thanks @sbirrer. OK - I ticked the final box, we're good barring any issues @smsharma might identify.

sibirrer commented 3 years ago

Thanks @sbirrer. OK - I ticked the final box, we're good barring any issues @smsharma might identify.

Thank you @coljac! I will update the notebooks as well in the next couple of days too.

smsharma commented 3 years ago

Apologies likewise for the delay on this!

lenstronomy is a mature package widely used in strong lensing analyses for modeling a variety of relevant physical systems (e.g., time-delay lenses, galaxy-galaxy lenses) and enabling downstream tasks such as Hubble constant measurement and dark matter substructure inference. The modular API additionally makes it easy to adapt for a given purpose as well as to contribute to the codebase. This has also enabled a small ecosystem of more specialized packages around lenstronomy. The package clearly demonstrates substantial scholarly effort.

I had no issues with installation as recommended in the documentation, and was able to go through the Jupyter notebook examples in the lenstronomy_extensions repository. The package ticks all the boxes at this stage, and I'm happy to sign off. Congrats to @sibirrer and all involved!

sibirrer commented 3 years ago

Thank you @smsharma and @coljac for the very encouraging and positive feedback! @coljac I also updated the notebooks slightly, in particular the astrofunc dependency got removed (was a leftover as the functionalities are itself available in lenstronomy).

@danielskatz what are the next steps for me/us?

danielskatz commented 3 years ago

@sibirrer - sorry I missed this comment.

The next steps are for you to:

I can then move forward with accepting the submission, which will involve me proofreading the final paper.

sibirrer commented 3 years ago
  • [ ] Make a tagged release of your software, and list the version tag of the archived version here. tag: 1.8.2_JOSS
danielskatz commented 3 years ago

@whedon set 1.8.2_JOSS as version

whedon commented 3 years ago

OK. 1.8.2_JOSS is the version.

sibirrer commented 3 years ago

And here is the DOI: 10.5281/zenodo.4913700

@danielskatz : Please let me know what you need from my side in addition. Thank you very much!

danielskatz commented 3 years ago

@whedon set 10.5281/zenodo.4913700 as archive

whedon commented 3 years ago

OK. 10.5281/zenodo.4913700 is the archive.

danielskatz commented 3 years ago

I note the zenodo DOI isn't yet working, but I see it listed in https://zenodo.org/record/4913700#.YL-vNi1h131 so will proceed for now

danielskatz commented 3 years ago

@whedon recommend-accept

whedon commented 3 years ago
Attempting dry run of processing paper acceptance...
whedon commented 3 years ago
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.dark.2018.11.002 is OK
- 10.1088/0004-637X/813/2/102 is OK
- 10.21105/joss.02825 is OK
- 10.1093/mnras/sty1264 is OK
- 10.3847/0004-637X/826/2/112 is OK
- 10.3847/1538-4357/abdf59 is OK
- 10.1093/mnras/stab984 is OK
- 10.1051/0004-6361/202038730 is OK
- 10.1051/0004-6361/202039363 is OK
- 10.3847/1538-4357/abdfc4 is OK
- 10.1051/0004-6361/201731042 is OK
- 10.1051/0004-6361/201526773 is OK
- 10.1093/mnras/stz1593 is OK
- 10.1093/mnras/stab484 is OK
- 10.1103/PhysRevD.102.063502 is OK
- 10.21105/joss.02854 is OK
- 10.1093/mnras/stz2254 is OK
- 10.1051/0004-6361/201937351 is OK
- 10.1051/0004-6361/202038942 is OK
- 10.1111/j.1365-2966.2009.14654.x is OK
- 10.1088/1475-7516/2021/04/010 is OK
- 10.1093/mnras/sty3397 is OK
- 10.1088/1475-7516/2017/04/049 is OK
- 10.1093/mnras/stab532 is OK
- 10.1093/mnras/staa1649 is OK
- 10.1093/mnras/staa3713 is OK
- 10.1093/mnras/stz1796 is OK
- 10.1088/1475-7516/2016/08/020 is OK
- 10.1093/mnras/stz200 is OK
- 10.3847/1538-4357/ab5b90 is OK
- 10.1093/mnras/stab536 is OK
- 10.1088/0004-637X/800/2/94 is OK
- 10.1093/mnras/staa1600 is OK
- 10.1093/mnras/stz3480 is OK
- 10.1093/mnras/stz3177 is OK
- 10.1038/nature10669 is OK
- 10.3847/0004-637X/823/1/37 is OK
- 10.1088/1475-7516/2017/05/037 is OK
- 10.1111/j.1365-2966.2010.16639.x is OK
- 10.1088/0004-637X/811/1/20 is OK
- 10.1051/0004-6361/202038861 is OK
- 10.1093/mnras/staa828 is OK
- 10.1093/mnras/stz3094 is OK
- 10.3847/1538-4357/ab042c is OK
- 10.1038/s41550-017-0295-x is OK
- 10.1038/nature11446 is OK
- 10.1109/JPROC.2009.2021005 is OK
- 10.3847/1538-4365/abd62c is OK
- 10.1111/j.1365-2966.2006.10040.x is OK
- 10.1051/0004-6361/201322068 is OK
- 10.3847/1538-3881/aabc4f is OK

MISSING DOIs

- 10.1093/mnras/stab1240 may be a valid DOI for title: Improved time-delay lens modelling and H_0 inference with transient sources

INVALID DOIs

- None
whedon commented 3 years ago

:wave: @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof :point_right: https://github.com/openjournals/joss-papers/pull/2372

If the paper PDF and Crossref deposit XML look good in https://github.com/openjournals/joss-papers/pull/2372, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true
danielskatz commented 3 years ago

@sibirrer - it looks like the @whedon DOI suggestion is correct - can you add that DOI to the appropriate bib entry?

sibirrer commented 3 years ago

@danielskatz Yes, can do that. Do I need to regenerate a Zenodo DOI and GitHub release or can I just add it to the branch on GitHub?

danielskatz commented 3 years ago

You don't need to update the code repository if changes are just in the paper. But tell me which branch (if not main/master) I need to build the paper from

danielskatz commented 3 years ago

I also have suggested some small edits in the paper in https://github.com/sibirrer/lenstronomy/pull/261 - please merge this, or let me know what you disagree with.

sibirrer commented 3 years ago

Thank you @danielskatz ! I added the DOI and merged the JOSS_paper branch to main. So you can trigger either from the JOSS_paper branch or main branch.

danielskatz commented 3 years ago

@whedon recommend-accept

whedon commented 3 years ago
Attempting dry run of processing paper acceptance...
whedon commented 3 years ago
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.dark.2018.11.002 is OK
- 10.1088/0004-637X/813/2/102 is OK
- 10.21105/joss.02825 is OK
- 10.1093/mnras/sty1264 is OK
- 10.3847/0004-637X/826/2/112 is OK
- 10.3847/1538-4357/abdf59 is OK
- 10.1093/mnras/stab984 is OK
- 10.1093/mnras/stab1240 is OK
- 10.1051/0004-6361/202038730 is OK
- 10.1051/0004-6361/202039363 is OK
- 10.3847/1538-4357/abdfc4 is OK
- 10.1051/0004-6361/201731042 is OK
- 10.1051/0004-6361/201526773 is OK
- 10.1093/mnras/stz1593 is OK
- 10.1093/mnras/stab484 is OK
- 10.1103/PhysRevD.102.063502 is OK
- 10.21105/joss.02854 is OK
- 10.1093/mnras/stz2254 is OK
- 10.1051/0004-6361/201937351 is OK
- 10.1051/0004-6361/202038942 is OK
- 10.1111/j.1365-2966.2009.14654.x is OK
- 10.1088/1475-7516/2021/04/010 is OK
- 10.1093/mnras/sty3397 is OK
- 10.1088/1475-7516/2017/04/049 is OK
- 10.1093/mnras/stab532 is OK
- 10.1093/mnras/staa1649 is OK
- 10.1093/mnras/staa3713 is OK
- 10.1093/mnras/stz1796 is OK
- 10.1088/1475-7516/2016/08/020 is OK
- 10.1093/mnras/stz200 is OK
- 10.3847/1538-4357/ab5b90 is OK
- 10.1093/mnras/stab536 is OK
- 10.1088/0004-637X/800/2/94 is OK
- 10.1093/mnras/staa1600 is OK
- 10.1093/mnras/stz3480 is OK
- 10.1093/mnras/stz3177 is OK
- 10.1038/nature10669 is OK
- 10.3847/0004-637X/823/1/37 is OK
- 10.1088/1475-7516/2017/05/037 is OK
- 10.1111/j.1365-2966.2010.16639.x is OK
- 10.1088/0004-637X/811/1/20 is OK
- 10.1051/0004-6361/202038861 is OK
- 10.1093/mnras/staa828 is OK
- 10.1093/mnras/stz3094 is OK
- 10.3847/1538-4357/ab042c is OK
- 10.1038/s41550-017-0295-x is OK
- 10.1038/nature11446 is OK
- 10.1109/JPROC.2009.2021005 is OK
- 10.3847/1538-4365/abd62c is OK
- 10.1111/j.1365-2966.2006.10040.x is OK
- 10.1051/0004-6361/201322068 is OK
- 10.3847/1538-3881/aabc4f is OK

MISSING DOIs

- None

INVALID DOIs

- None
whedon commented 3 years ago

:wave: @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof :point_right: https://github.com/openjournals/joss-papers/pull/2373

If the paper PDF and Crossref deposit XML look good in https://github.com/openjournals/joss-papers/pull/2373, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true
danielskatz commented 3 years ago

@whedon accept deposit=true