search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
717 stars 38 forks source link

[REVIEW]: Council Data Project: Software for Municipal Data Collection, Analysis, and Publication #3904

Closed whedon closed 2 years ago

whedon commented 2 years ago

Submitting author: !--author-handle-->@evamaxfield<!--end-author-handle-- (Eva Maxfield Brown) Repository: https://github.com/CouncilDataProject/cookiecutter-cdp-deployment Branch with paper.md (empty if default branch): Version: v3.0.0 Editor: !--editor-->@chartgerink<!--end-editor-- Reviewers: @hknd23, @chainsawriot Archive: 10.5281/zenodo.5748663

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/a3ed732d18a07a70ac256501911a8122"><img src="https://joss.theoj.org/papers/a3ed732d18a07a70ac256501911a8122/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/a3ed732d18a07a70ac256501911a8122/status.svg)](https://joss.theoj.org/papers/a3ed732d18a07a70ac256501911a8122)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@hknd23 & @chainsawriot, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @chartgerink know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @hknd23

✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @chainsawriot

✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 2 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @hknd23, @chainsawriot it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 2 years ago

Wordcount for paper.md is 1234

whedon commented 2 years ago 
Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.06 s (899.6 files/s, 59882.8 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
YAML                            17            172             77            917
Markdown                        12            336              0            712
Python                          11            138             92            603
JSON                             6              0              0            103
TeX                              1              7              0             75
HTML                             1              2              3             67
make                             2             12              0             52
JSX                              1              3              0             24
-------------------------------------------------------------------------------
SUM:                            51            670            172           2553
-------------------------------------------------------------------------------

Statistical information for the repository '03dd601aed249b85514be476' was
gathered on 2021/11/11.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Hawk Ticehurst                   2            23            170           13.08
Isaac Na                         3             3              3            0.41
Jackson Maxfield Bro            13           932            147           73.15
JacksonMaxfield                  1           195              0           13.22
Toni Wells                       1             1              1            0.14

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Hawk Ticehurst                5           21.7          5.2                0.00
Isaac Na                      1           33.3          1.1                0.00
JacksonMaxfield             826          423.6          2.5                9.93
Toni Wells                    1          100.0          7.5                0.00
whedon commented 2 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1017/S1537592709990892 is OK
- 10.21105/joss.00411 is OK

MISSING DOIs

- 10.5860/choice.49-1821 may be a valid DOI for title: CourtListener

INVALID DOIs

- None
whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

evamaxfield commented 2 years ago

Excited to see this move forward! Thanks all for agreeing to review.

As a reminder this is three tools together:

chainsawriot commented 2 years ago

@JacksonMaxfield @chartgerink

I read with interest the paper by @JacksonMaxfield et al. I think the entire software suite is useful for political scientists, among others, who are interested in US local politics. I am very impressed with the live demo.

The software uses speech-to-text technology from Google to make speeches machine-readable and searchable in a Firebase database. The software is well-documented. I've attempted to create a template. Due to the cost, I've not tried to deploy the project.

I only have two minor concerns related to the project and the paper. Other than the two, I don't have further concerns.

Project

Paper

hknd23 commented 2 years ago

Hi @JacksonMaxfield,

The CDP project has many potential applications in the social sciences such as Political Science, Public Policy, and Sociology, as well as Legal Studies. The website and interface are well developed, and the ability to collect rich data is noteworthy. Please note that due to the cost to host my own instance, I also have not deployed the project myself. However, I have followed the steps up to the creation of the repository to host the instance with few suggestions. There are a few remaining points that can help the project to be more user-friendly:

  1. Installation instructions: a. As the self-deployment of the instances requires the use of gcloud, gsutil, and Pulumim, in addition to the things to know section, it may be helpful to first-time users to mention installing these tools early on in the installation instructions. b. The step-by-step Youtube video is extremely helpful as a tutorial detailing the steps needed to deploy an instance. In addition to this, you should also have a tutorial page as a walkthrough guide and example deployed repository. These can serve as quick reference tools and are easier to update/maintain as you add more features in the future.

  2. Community guidelines: There are CONTRIBUTING files for the frontend and backend repositories. The main repositories should similarly have contributing guidelines and link to the guidelines in the other two as well.

  3. The text in the figure is hard to read. I agree with @chainsawriot that the icons are not informative as their labels. Breaking the figure down to smaller steps or a list describing the process may help.

Overall, this is a very interesting project that can facilitate research progress in local governments. Please let me know if you have questions about my comments.

chartgerink commented 2 years ago

Thank you @chainsawriot and @hknd23 for the constructive reviews! 🥳

@JacksonMaxfield the ball is back in your court regarding the reviewer's comments. I agree that the main points to respond to are:

Please feel free to ping me here once you feel you've worked on these points to resolve the reviewer's questions/comments to your satisfaction.

evamaxfield commented 2 years ago

Thank you all!

I will work on the community guidelines + contributing for the cookiecutter repository and I will work on the figure 1 readability.

A follow up question for @hknd23 regarding installation steps:

As the self-deployment of the instances requires the use of gcloud, gsutil, and Pulumim, in addition to the things to know section, it may be helpful to first-time users to mention installing these tools early on in the installation instructions.

Agree. Will add a "prerequisites" section.

In addition to this, you should also have a tutorial page as a walkthrough guide and example deployed repository. These can serve as quick reference tools and are easier to update/maintain as you add more features in the future.

These SETUP steps are provided with the generated repository. Can you clarify what should be added or should those directions simply be copied to the README or something similar? An entirely new documentation page?

hknd23 commented 2 years ago

Hi @JacksonMaxfield,

The instructions from SETUP look great. I was able to follow the steps without much issue when I first set up my example, and my impression is that these steps are the same with every generated repository. My suggestion is to have a section for or link to these steps in the main installation instructions so users can see them before having to set up the generated repository. I hope this answers your question. Please let me know if you have other questions.

evamaxfield commented 2 years ago

Hello all! (cc @chartgerink)

I just pushed some changes that I believe resolve your comments: https://github.com/CouncilDataProject/cookiecutter-cdp-deployment/commit/a3c202858a1cfa9d6c4a2619dffb0188410fbeb0

Thanks again for your comments :slightly_smiling_face:

evamaxfield commented 2 years ago

@whedon generate pdf

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

chainsawriot commented 2 years ago

@JacksonMaxfield Thanks for updating the paper. I have checked the community guidelines box above.

I don't want to be a "Reviewer 2". The figure actually looks okay on its own. However, it got resized in the paper because it needs to be inside that small column. Sorry for my old eyes, but I need to magnify 300% to read the text. For the graph to be readable, I believe the text should have the same size as the caption in the paper.

I am trying to redo the figure for you. I will submit a PR later.

evamaxfield commented 2 years ago

@JacksonMaxfield Thanks for updating the paper. I have checked the community guidelines box above.

I don't want to be a "Reviewer 2". The figure actually looks okay on its own. However, it got resized in the paper because it needs to be inside that small column. Sorry for my old eyes, but I need to magnify 300% to read the text. For the graph to be readable, I believe the text should have the same size as the caption in the paper.

I am trying to redo the figure for you. I will submit a PR later.

Ahhh sorry @chainsawriot. That's not reviewer #2 that's actually incredibly kind of you to offer but don't worry we will handle it!

chainsawriot commented 2 years ago

@JacksonMaxfield I am halfway creating it. I have tried to test it by embedding it in your paper. It works great with this size. If you will handle it then I stop it from there. But you can take it as a reference.

https://docs.google.com/drawings/d/1IPebeFrgqOLVRpuANxOXkxZ7I54DYC3EOKWFYbcdZck/edit?usp=sharing

The before and after

Screenshot from 2021-11-24 20-44-25

chartgerink commented 2 years ago

Thank you all for such an engaged review process! Fantastic to see 😊

@hknd23 there are two open checks on your checklist - are these resolved to your satisfaction based on the author's response? If so, could you mark those as done as well?

@chainsawriot thanks for being so kind to help out with the figure! @JacksonMaxfield feel free to regenerate the PDF when that's ready (you seem to know your way around Whedon 😉 )

evamaxfield commented 2 years ago

@whedon generate pdf

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

evamaxfield commented 2 years ago

@chartgerink @chainsawriot

I pushed another update to the paper. I tried to clean up all of the parts of figure one where possible: removing / minimizing white space, bolding the text, etc. It is still a bit hard to read without zooming in and I understand that so I additionally rewrote the figure one description. Figure one description now has, in my opinion, the entire figure just restated as text.

I don't know if we will ever get it perfect and I am personally okay with having to zoom in on figures, we do it all the time for other paper's so I am going to leave it as is for now.

I think just waiting on @hknd23 to confirm the changes are okay?

hknd23 commented 2 years ago

Hi @JacksonMaxfield and @chartgerink,

I went through the repo one more time and it is looking great! All the necessary items are there, and the instructions are much easier to follow. I have checked off the final 2 items of the list.

evamaxfield commented 2 years ago

Thanks @hknd23! @chartgerink I think this is now back in your court. Please just let me know what I should do next.

chainsawriot commented 2 years ago

@JacksonMaxfield Thanks for updating the figure. It looks okay now.

chartgerink commented 2 years ago

Okay - wow I am loving this specific process 😊 Thank you all for the speed, kindness, and considerateness with which you've gone through this review process 💜

Running final checks now...

chartgerink commented 2 years ago

@whedon check references

whedon commented 2 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1017/S1537592709990892 is OK
- 10.21105/joss.00411 is OK

MISSING DOIs

- 10.5860/choice.49-1821 may be a valid DOI for title: CourtListener

INVALID DOIs

- None
chartgerink commented 2 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1017/S1537592709990892 is OK
- 10.21105/joss.00411 is OK

MISSING DOIs

- 10.5860/choice.49-1821 may be a valid DOI for title: CourtListener

INVALID DOIs

- None

This is not a missing DOI as @JacksonMaxfield indicated previously in the pre-review thread (https://github.com/openjournals/joss-reviews/issues/3891#issuecomment-963817116)

chartgerink commented 2 years ago

@whedon generate pdf

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

chartgerink commented 2 years ago

@JacksonMaxfield I am happy to recommend this paper for publication, based on this review process. Thank you to @chainsawriot and @hknd23 🤗

@JacksonMaxfield could you archive the code on Zenodo/Figshare and provide me with the DOI? I will add it here and then accept it and pass it on to the editor in chief 🥳

evamaxfield commented 2 years ago

@chartgerink Zenodo now has the CDP cookiecutter archived:

Link: https://doi.org/10.5281/zenodo.5744988 DOI: 10.5281/zenodo.5744988

chartgerink commented 2 years ago

@whedon set 10.5281/zenodo.5744988 as archive

whedon commented 2 years ago

OK. 10.5281/zenodo.5744988 is the archive.

chartgerink commented 2 years ago

@whedon recommend-accept

whedon commented 2 years ago 
Attempting dry run of processing paper acceptance...
whedon commented 2 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1017/S1537592709990892 is OK
- 10.21105/joss.00411 is OK

MISSING DOIs

- 10.5860/choice.49-1821 may be a valid DOI for title: CourtListener

INVALID DOIs

- None
whedon commented 2 years ago

:wave: @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof :point_right: https://github.com/openjournals/joss-papers/pull/2778

If the paper PDF and Crossref deposit XML look good in https://github.com/openjournals/joss-papers/pull/2778, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true
kthyng commented 2 years ago

Hi @JacksonMaxfield! I'll be helping to wrap up this submission.

I see we have version 3.0 listed at the top of this review issue for your software, but your most recent tag is 0.0.4. What version should be associated with your JOSS paper?

Also, can you manually update the Zenodo archive metadata so that the title and author list exactly match your JOSS paper?

kthyng commented 2 years ago

@JacksonMaxfield Your paper looks good, but you'll need to preserve capitalization so your reference list is good. You can do this by adding {} around characters in your .bib file in which you want to preserve capitalization. One listing has R and Australia uncapitalized, but please check all carefully.

evamaxfield commented 2 years ago

Hi @JacksonMaxfield! I'll be helping to wrap up this submission.

Hey @kthyng! Thanks!

I see we have version 3.0 listed at the top of this review issue for your software, but your most recent tag is 0.0.4. What version should be associated with your JOSS paper?

The paper has the correct version, I will push a v3.0.0 tag to the repo later today.

Also, can you manually update the Zenodo archive metadata so that the title and author list exactly match your JOSS paper?

Ahhhh sorry. I used the default GitHub + Zenodo integration. From a brief look, it seems I need to add a .zenodo.json file and that should solve it. Will add that tonight as well.

@JacksonMaxfield Your paper looks good, but you'll need to preserve capitalization so your reference list is good. You can do this by adding {} around characters in your .bib file in which you want to preserve capitalization. One listing has R and Australia uncapitalized, but please check all carefully.

Similarly, will do!

Thanks for the comments.

kthyng commented 2 years ago

Ok let me know when you're finished!

evamaxfield commented 2 years ago

@kthyng All have now been resolved!

Want to point out that the DOI I provided previously points to Zenodo's "all versions" DOI. If that isn't what is desired, the v3.0.0 DOI is: 10.5281/zenodo.5748663

kthyng commented 2 years ago

@JacksonMaxfield Ah, thank you for pointing that out with the doi. Actually I would like to use the specific DOI for the version associated with the JOSS paper so users know that for future reference as the software changes over time. With this in mind, can you update the title of that DOI to match your title? Looks like the author list maybe already matches?

kthyng commented 2 years ago

@whedon set 10.5281/zenodo.5748663 as archive

whedon commented 2 years ago

OK. 10.5281/zenodo.5748663 is the archive.

kthyng commented 2 years ago

@whedon generate pdf

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

kthyng commented 2 years ago

everything else looks ready to go.

evamaxfield commented 2 years ago

With this in mind, can you update the title of that DOI to match your title? Looks like the author list maybe already matches?

Done!