search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
703 stars 36 forks source link

[REVIEW]: pyCSEP: A Python Package For Earthquake Forecast Developers #3658

Closed whedon closed 2 years ago

whedon commented 3 years ago

Submitting author: @wsavran (William Savran) Repository: https://github.com/SCECcode/pycsep Version: v0.5.2 Editor: @kbarnhart Reviewers: @nvanderelst, @mbarall Archive: 10.5281/zenodo.5904124

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/43f25170c9329d9bb9abc27e3dfabace"><img src="https://joss.theoj.org/papers/43f25170c9329d9bb9abc27e3dfabace/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/43f25170c9329d9bb9abc27e3dfabace/status.svg)](https://joss.theoj.org/papers/43f25170c9329d9bb9abc27e3dfabace)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@nvanderelst and @mbarall, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @kbarnhart know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @nvanderelst

✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @mbarall

✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 3 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @nvanderelst it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 3 years ago

PDF failed to compile for issue #3658 with the following error: 

 Can't find any papers to compile :-(
whedon commented 3 years ago 
Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=3.01 s (32.6 files/s, 246647.9 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
XML                               5              0              0         717119
JSON                              9              0              0           7184
Python                           50           1999           3492           6769
YAML                             10            137             15           3482
reStructuredText                 14            476            395            851
Markdown                          3             69              0            183
Perl                              1             10             12             41
HTML                              2              4             11             22
Bourne Again Shell                1             16             28             19
make                              1              5              6             12
Jupyter Notebook                  1              0            180             10
Bourne Shell                      1              0              0              2
--------------------------------------------------------------------------------
SUM:                             98           2716           4139         735694
--------------------------------------------------------------------------------

Statistical information for the repository '0d8ef45d20f9efdd17e304bc' was
gathered on 2021/08/25.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
MH                               2             5              4            0.02
Pablo Iturrieta                  7           637            155            1.69
Philip Maechling                 1           386              0            0.82
Thomas Beutin ("tb")             4           103             18            0.26
William                         12            54             48            0.22
William Savran                 297         25334          16064           88.17
bayonato89                       2           104             27            0.28
khawajasim                      14          1161            131            2.75
pciturri                        20           810            333            2.43
tb                               6           124              6            0.28
wsavran                         32          1038            411            3.09

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
MH                            4           80.0          2.4               50.00
Pablo Iturrieta             592           92.9          5.7                9.46
Philip Maechling            155           40.2          0.0               54.84
Thomas Beutin ("tb")         29           28.2         16.5                0.00
William Savran            10100           39.9         18.9               14.90
bayonato89                  102           98.1          1.9                9.80
khawajasim                   35            3.0         18.2               11.43
pciturri                    440           54.3          9.2               19.09
wsavran                     803           77.4          4.6                9.59
kbarnhart commented 3 years ago

@nvanderelst - thanks for agreeing to review this submission to JOSS. In the prior comments there is a checklist that you can use to guide you through your review. There are also instructions to manage your github notifications so you don't get all notifications for all JOSS reviews.

We have an automatic reminder set up in two weeks to ask you how the review is going. At present we request that reviewers complete their reviews within 6 weeks. I know you indicated you would be able to get to this sometime in late September (totally fine). JOSS is trying to be mindful of changes people have experienced due to COVID-19.

As you work through your review, if there are any issues that come up, please make an issue in the PyCEP repository, and link to this issue (paste openjournals/joss-reviews/issues/3658 into the issue you create). That way most of the discussion can occur on in-repo issues.

If you have any questions, please let me know (tag me here or email krbarnhart@usgs.gov).

I plan to find a second reviewer for this submission and will manually edit this page to provide them a checklist once they have been identified.

kbarnhart commented 3 years ago

@whedon add @mbarall as reviewer

whedon commented 3 years ago

OK, @mbarall is now a reviewer

kbarnhart commented 3 years ago

@mbarall - thanks for agreeing to review this submission to JOSS. In the prior comments there is a checklist that you can use to guide you through your review. There are also instructions to manage your github notifications so you don't get all notifications for all JOSS reviews.

We have an automatic reminder set up in two weeks to ask you how the review is going. At present we request that reviewers complete their reviews within 6 weeks. You indicated to me that you expect your review to take a few weeks longer, which is totally fine. JOSS is trying to be mindful of changes people have experienced due to COVID-19. Please update me if anything else comes up.

As you work through your review, if there are any issues that come up, please make an issue in the PyCEP repository, and link to this issue (paste openjournals/joss-reviews/issues/3658 into the issue you create). That way most of the discussion can occur on in-repo issues.

Thanks again for contributing a review to JOSS.

kbarnhart commented 3 years ago

@whedon generate pdf from branch pycsep_joss

whedon commented 3 years ago 
Attempting PDF compilation from custom branch pycsep_joss. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 3 years ago

:wave: @nvanderelst, please update us on how your review is going (this is an automated reminder).

kbarnhart commented 3 years ago

@nvanderelst - no need to update now since you let me know ahead of time that you'd have availability to do this review closer to the end of September.

kbarnhart commented 2 years ago

@wsavran @nvanderelst @mbarall I wanted to let you know I'll be in the field with very limited internet access from Sept 24 through Sept 30. If any questions come up during that time, I'll respond when I return.

@nvanderelst, @mbarall I know that we are getting close (or are in) the time you expected to be able to do this review. Please let me know if you have any questions as you undertake your reviews. Thanks again for being willing to serve as a reviewer for JOSS.

kbarnhart commented 2 years ago

/ooo September 24 until September 30

kbarnhart commented 2 years ago

@nvanderelst, @mbarall - I wanted to check in and see how your reviews were going. Please let me know if you have any questions.

kbarnhart commented 2 years ago

@whedon re-invite @nvanderelst as reviewer

whedon commented 2 years ago

OK, the reviewer has been re-invited.

@nvanderelst please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

mbarall commented 2 years ago

@kbarnhart I have completed my review.

kbarnhart commented 2 years ago

@mbarall many thanks for your review!

@wsavran - I would recommend you comment on SCECcode/pycsep/issues/163 once you've addressed the issues outlined there. At that point, @mbarall and I can assess whether you have addressed those concerns.

It is up to you whether you want to address these concerns now, or wait until @nvanderelst has completed his review.

Let me know with any questions, comments, or concerns.

nvanderelst commented 2 years ago

Hi Katy, I hoped to have the review completed today, but I have stalled out because I cannot get some of the aspects of the code to work. I'm no python whiz, so I'm probably doing something stupid -- I think Michael got things to work just fine. I will need to work with Bill to figure things out, but this means the review will probably not be finished until after the thanksgiving holiday, as I'm heading into the desert next week.

My apologies! -Nicholas

From: Katy Barnhart @.> Sent: Friday, November 19, 2021 10:46 AM To: openjournals/joss-reviews @.> Cc: Van Der Elst, Nicholas J @.>; Mention @.> Subject: [EXTERNAL] Re: [openjournals/joss-reviews] [REVIEW]: pyCSEP: A Python Package For Earthquake Forecast Developers (#3658)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

@mbarallhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmbarall&data=04%7C01%7Cnvanderelst%40usgs.gov%7Cb5d0ec91d5974c2c4c0108d9ab8cfb04%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637729444278047587%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2Fg8Dlmu5aC9fqsoMT%2F6k2ceFLAwoF3entGg%2F0M0ZUog%3D&reserved=0 many thanks for your review!

@wsavranhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fwsavran&data=04%7C01%7Cnvanderelst%40usgs.gov%7Cb5d0ec91d5974c2c4c0108d9ab8cfb04%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637729444278057537%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=9zybSDAzi8h98k6%2F5a%2BTuPZshR4VwkY34QAg4%2FULgYs%3D&reserved=0 - I would recommend you comment on SCECcode/pycsep/issues/163https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSCECcode%2Fpycsep%2Fissues%2F163&data=04%7C01%7Cnvanderelst%40usgs.gov%7Cb5d0ec91d5974c2c4c0108d9ab8cfb04%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637729444278057537%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qB4xwApOPYB5OKZ91FNfU%2BenPlOMNj746pDf2ShGofc%3D&reserved=0 once you've addressed the issues outlined there. At that point, @mbarallhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmbarall&data=04%7C01%7Cnvanderelst%40usgs.gov%7Cb5d0ec91d5974c2c4c0108d9ab8cfb04%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637729444278067496%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=oHg4GmJJPLRqQdUITN%2BAFtB27doznofnQ4vOI0JlWPw%3D&reserved=0 and I can assess whether you have addressed those concerns.

It is up to you whether you want to address these concerns now, or wait until @nvanderelsthttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnvanderelst&data=04%7C01%7Cnvanderelst%40usgs.gov%7Cb5d0ec91d5974c2c4c0108d9ab8cfb04%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637729444278067496%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=uR%2BParqTxqp%2BR%2Fq1OKTwrFzZsN4uy48FLoy%2BsZprQp4%3D&reserved=0 has completed his review.

Let me know with any questions, comments, or concerns.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenjournals%2Fjoss-reviews%2Fissues%2F3658%23issuecomment-974319574&data=04%7C01%7Cnvanderelst%40usgs.gov%7Cb5d0ec91d5974c2c4c0108d9ab8cfb04%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637729444278067496%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=CrmGvlQduL5DHCKr5%2BsUfE%2FUl7xlPxP2y8qfchwFEwM%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAC5MGOS3O5VE2IQK4GQY64LUM2LRLANCNFSM5CZKWTMA&data=04%7C01%7Cnvanderelst%40usgs.gov%7Cb5d0ec91d5974c2c4c0108d9ab8cfb04%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637729444278077454%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=rFPt76KtzJSiDc7%2BksZhSsg0ftmc8se%2FUmLXJq1aS%2F4%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Cnvanderelst%40usgs.gov%7Cb5d0ec91d5974c2c4c0108d9ab8cfb04%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637729444278077454%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PhoNqkBoC%2BJmC5QIJPmdzAE8O%2BxhIOr3ArpZ14jRjbM%3D&reserved=0 or Androidhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Cnvanderelst%40usgs.gov%7Cb5d0ec91d5974c2c4c0108d9ab8cfb04%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637729444278087409%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=YmwEO%2FRNfqc9YZjr3GhsNMKoI5YXHLM2DePv7Yz2Ba4%3D&reserved=0.

kbarnhart commented 2 years ago

@nvanderelst Thanks for the update, I'll check back early in the week of 11/29. Enjoy the desert 🌵 !

wsavran commented 2 years ago

@nvanderelst nope, you didn't do anything stupid at all. this one is one me. i merged a new pull request that added new features (ROC plots and Kagan I1 score) including updated documentation. i didn't publish a new release that included those changes, which caused the errors you were getting. i'm fixing that stuff now, and also addressing the trouble with getting the plots to show in the terminal. have fun next week!

kbarnhart commented 2 years ago

@nvanderelst - it looks like you have filled out most of the checklist (thanks!), have you gotten to a point where all of the remaining check boxes need @wsavran to address the outstanding issues described here:

If so, please indicate so here so @wsavran knows the scope of remaining work. If not, please make one or more additional issue indicating what else needs addressing.

Let me know with any questions/comments/concerns. Thanks!

kbarnhart commented 2 years ago

@nvanderelst a quick ping to answer questions in the above comment. Thanks!

kbarnhart commented 2 years ago

@wsavran based on looking at the checklist, I suspect that at this point, both reviewers have raised all the issues that remain.

Once you have completed addressing these issues, let me know. At that point we should be able to move into the final stages of the review process. Thanks for your patience.

(FYI, I'll be out of the office Dec 24-Jan 2).

kbarnhart commented 2 years ago

@wsavran I wanted to check in and see if you needed anything from me and/or the reviewers at this point. If not, just let me know when you've completed any revisions based on reviewer comments. Thanks!

wsavran commented 2 years ago

@kbarnhart I just got back from holiday break. I hope you had a refreshing holiday as well. I'm working on their comments, so I'll let you know when I'm finished with everything. Thanks!

kbarnhart commented 2 years ago

Sounds good @wsavran. Thanks for the update.

wsavran commented 2 years ago

@kbarnhart I think I have addressed all of the comments from the reviewers. I'll summarize them here:

  1. added a page on the documentation explaining the mathematical theory behind the evaluations
  2. updated the tutorials so they display plots properly when run via the terminal
  3. added a new branching model to ensure that documentation changes are not pushed into production that could break the tutorials

Thanks for managing this process and thank you to the reviewers for working through the software package.

kbarnhart commented 2 years ago

Thanks @wsavran. I'll be able to look carefully at this first thing Tuesday and will move forward then.

kbarnhart commented 2 years ago

@wsavran I've looked over your recent changes and it looks to me that you've addressed issues identified by the reviewers. I made one minor comment about documentation in that issue thread.

At this point in the JOSS review process what happens is that @mbarall and @nvanderelst revisit their reviews in light of your changes. If they find that you have addressed their concerns, they close any issues they opened and complete their checklists above. If they find anything that they still think should be addressed, they can point it out. I'll write a quick email to each of them now indicating they should do this.

Once they have revisited and completed their reviews, we enter the final stage of the JOSS review process which involves a light copyedit from me and updating of some metadata. I then pass off handling to one of our managing editors in chief.

mbarall commented 2 years ago

@kbarnhart I am satisfied with the changes. I recommend that you accept pyCSEP for publication.

kbarnhart commented 2 years ago

@mbarall thanks for contributing your review.

nvanderelst commented 2 years ago

@kbarnhart I was able to get everything to work and I've completed my review.

kbarnhart commented 2 years ago

@nvanderelst thanks for contributing your review.

@wsavran I will be able to do my final steps for this submission on Monday and will move forward then.

mjw98 commented 2 years ago

@kbarnhart I am satisfied with the changes. I recommend that you accept pyCSEP for publication.

Thank you, @mbarall , for the thorough review.

mjw98 commented 2 years ago

@kbarnhart I was able to get everything to work and I've completed my review.

Thanks @nvanderelst for the useful review!

kbarnhart commented 2 years ago

@whedon check references

kbarnhart commented 2 years ago

@whedon generate pdf

whedon commented 2 years ago

PDF failed to compile for issue #3658 with the following error: 

 Can't find any papers to compile :-(
kbarnhart commented 2 years ago

@whedon generate pdf from branch pycsep_joss

whedon commented 2 years ago 
Attempting PDF compilation from custom branch pycsep_joss. Reticulating splines etc...
whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

kbarnhart commented 2 years ago

@whedon check references from branch pycsep_joss

whedon commented 2 years ago 
Attempting to check references... from custom branch pycsep_joss
whedon commented 2 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1785/gssrl.78.1.30 is OK
- 10.1002/cpe.1519 is OK
- 10.4401/ag-5350 is OK
- 10.2478/s11600-011-0013-5 is OK
- 10.1785/0120090340 is OK
- 10.1785/0120120186 is OK
- 10.1785/0220180033 is OK
- 10.1785/0220180051 is OK
- 10.1785/0220180031 is OK
- 10.1785/0220180053 is OK
- 10.1093/gji/ggaa554 is OK
- 10.1785/0220170045 is OK
- 10.1785/0220180161 is OK
- 10.1785/0120200026 is OK
- 10.1038/s41586-020-2649-2 is OK
- 10.1038/s41592-019-0686-2 is OK
-  10.25080/Majora-92bf1922-00a  is OK

MISSING DOIs

- 10.1785/gssrl.78.1.17 may be a valid DOI for title: Earthquake likelihood model testing
- 10.1109/mcse.2007.55 may be a valid DOI for title: Matplotlib: A 2D graphics environment

INVALID DOIs

- https://doi.org/10.1785/gssrl.78.1.7 is INVALID because of 'https://doi.org/' prefix
kbarnhart commented 2 years ago

@wsavran @mjw98 and co-authors:

Please address the following:

Sorry to miss some of those reference issues on my first run through with the paper.

Once you've addressed those minor issues, please re-run

and verify that and verify that no missing or invalid DOIs are returned and that the PDF compiles without issue.

At that point, please do the following:

Once all these steps are completed, I will update the paper metadata and recommend that the submission be accepted. At that point whedon will do a dry run of paper publishing and the JOSS managing editor in chief will handle the final steps.

If at any point you have questions for me, please reach out here or at krbarnhart@usgs.gov

wsavran commented 2 years ago

@whedon check references from branch pycsep_joss

whedon commented 2 years ago 
Attempting to check references... from custom branch pycsep_joss
wsavran commented 2 years ago

@whedon generate pdf from branch pycsep_joss