search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
701 stars 36 forks source link

[REVIEW]: covid19.analytics: An R Package to Obtain, Analyze and Visualize Data from the 2019 Corona Virus Disease Pandemic #2995

Closed whedon closed 3 years ago

whedon commented 3 years ago

Submitting author: @mponce0 (Marcelo Ponce) Repository: https://github.com/mponce0/covid19.analytics/ Version: 2.1.0 Editor: @fboehm Reviewer: @LogIN-, @JDRomano2 Archive: 10.5281/zenodo.4640307

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/43bab096ad574f4510a7258c20a1153d"><img src="https://joss.theoj.org/papers/43bab096ad574f4510a7258c20a1153d/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/43bab096ad574f4510a7258c20a1153d/status.svg)](https://joss.theoj.org/papers/43bab096ad574f4510a7258c20a1153d)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@LogIN- & @JDRomano2, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @fboehm know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @LogIN-

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @JDRomano2

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 3 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @LogIN-, @JDRomano2 it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 3 years ago

PDF failed to compile for issue #2995 with the following error:

Can't find any papers to compile :-(

mponce0 commented 3 years ago

@whedon generate pdf from branch literature

whedon commented 3 years ago 
Attempting PDF compilation from custom branch literature. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

fboehm commented 3 years ago

@JDRomano2 and @LogIN- - I just wanted to check on the status of your reviews. Please let me know if I can assist with anything. Thanks again!

JDRomano2 commented 3 years ago

Thanks very much @fboehm for the invitation to review this submission! Please see below for individual comments on each of the checklist items that I feel need attention:

General checks

Contribution and authorship: Has the submitting author (@mponce0) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

The contributions made by the second author are not clear, as they are not a contributor to the GitHub source repository, and their contributions are not described elsewhere in the paper. I would strongly recommend including information about author contributions in an "About" heading within the GitHub README, as well as current contact information for at least the primary author.

Documentation

Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?

Documentation is provided primarily by means of examples. No API documentation is given.

If a reference manual with API documentation is available on CRAN, there should be a link to it from the GitHub README and/or the project website.

Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

I cannot find any community guidelines in the paper, the code repository, or the project's website.

The process for reporting bugs and issues is especially important to include, and should be clearly defined in the README. Ideally this would make use of GitHub's issue tracking functionality.

Software paper

Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?

The summary is somewhat vague and nonspecific.

"With the emergence of a new pandemic worldwide, a novel strategy to approach it has also emerged."

What does 'approach it' mean? The paper should clearly describe the types of data-related problems that the software is meant to solve.

A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?

The authors do describe the motivation behind the project in the context of other open-source COVID-19 data resources. However, the "Statement of Need" section needs to be cleaned up. Much of the text in here should go either into the Summary (keeping in mind the recommended edits mentioned above) or into a separate section following the Statement of Need describing the software's functionality.

The GitHub README file contains a fairly good description of the needs this software is meant to address, and some of that text can probably be reused in the paper.

Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?

The paper is in need of stylistic and typographical revisions. A few examples (not comprehensive) include:

These are just a few examples that stand out.

References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Some of the citations are incomplete. For example, the citation for "The Comprehensive R Archive Network" should include the journal name, author, publication date, volume, and issue number. These are easily found by clicking on the DOI link provided in the reference. Similar omissions are found in several of the other references.

I'm not sure that enough of the citations point to appropriate publications or data sources. For example, you list the Visual Capitalist infographic as one of the data sources used by your software, but the linked article doesn't seem to provide any way to access the data.

mponce0 commented 3 years ago

Many thanks @JDRomano2 for the thorough review and useful comments! The editions and updates addressing this reviewer's comments have been added to the manuscript on the "literature" branch of the https://github.com/mponce0/covid19.analytics/ repository.

General checks

Contribution and authorship: Has the submitting author (@mponce0) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

The contributions made by the second author are not clear, as they are not a contributor to the GitHub source repository, and their contributions are not described elsewhere in the paper. I would strongly recommend including information about author contributions in an "About" heading within the GitHub README, as well as current contact information for at least the primary author.

Authors contributions were (and are) originally described in the "Description" file, which is the standard place for doing it in R packages.

Following the reviewer's recommendation we have also added an "About" heading to the README document stating more clearly the contributions of the authors, as well as the way for the community to contribute to the package.

Documentation

Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?

Documentation is provided primarily by means of examples. No API documentation is given.

If a reference manual with API documentation is available on CRAN, there should be a link to it from the GitHub README and/or the project website.

A link has been added to the README for API documentation, in this case, functions descriptions from CRAN immediately after the Overview of the Main Functions from the "covid19.analytics" Package section.

Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

I cannot find any community guidelines in the paper, the code repository, or the project's website.

The process for reporting bugs and issues is especially important to include, and should be clearly defined in the README. Ideally this would make use of GitHub's issue tracking functionality.

This was mentioned both in the paper as well as in the README file, in the first bullet point of "Statement of need":
Users can also submit tickets for bugs, suggestions or comments using the "issues" tab" --referring to the GitHub repository--. We have also added a paragraph to the paper about how to seek for support using the "ISSUES tracker" and contribute via "pull request". As a side note please notice that both features have been in place and used, as you can verify in the repo (https://github.com/mponce0/covid19.analytics/issues) since quite some time ago.

Software paper

Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?

The summary is somewhat vague and nonspecific.

"With the emergence of a new pandemic worldwide, a novel strategy to approach it has also emerged."

What does 'approach it' mean? The paper should clearly describe the types of data-related problems that the software is meant to solve.

We have clarified the summary and added a few sentences to describe what the package can allow users to achieve in terms of the data availability and its challenges.

A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?

The authors do describe the motivation behind the project in the context of other open-source COVID-19 data resources. However, the "Statement of Need" section needs to be cleaned up. Much of the text in here should go either into the Summary (keeping in mind the recommended edits mentioned above) or into a separate section following the Statement of Need describing the software's functionality.

The GitHub README file contains a fairly good description of the needs this software is meant to address, and some of that text can probably be reused in the paper.

We have restructured and edited the manuscript to incorporate the reviewer's comments.

Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?

The paper is in need of stylistic and typographical revisions. A few examples (not comprehensive) include:

  • "Coronavirus"; not "Corona virus"
  • "COVID-19"; not "CoViD19"
  • "GitHub"; not "GitHUB"
  • Capitalized letters at the beginning of each item in the bulleted lists
  • "the R language"; not "the R Language and Environment for Statistical Computing" (which is just the title of the publication, not the actual language's title).
  • Why is the word "backup" in bold on the first page?

These are just a few examples that stand out.

We thank the reviewer for noticing this, and we have revisit and edited the manuscript to fix all of these.

References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Some of the citations are incomplete. For example, the citation for "The Comprehensive R Archive Network" should include the journal name, author, publication date, volume, and issue number. These are easily found by clicking on the DOI link provided in the reference. Similar omissions are found in several of the other references.

We have reviewed the citations and completed the information for the CRAN publication. We couldn't identify any other reference missing any of the elements described above.

I'm not sure that enough of the citations point to appropriate publications or data sources. For example, you list the Visual Capitalist infographic as one of the data sources used by your software, but the linked article doesn't seem to provide any way to access the data.

This is because the data from the infographics was gather manually, ie. no actual data repository is offered. WRT the other data sources, we will be OK to follow any other recommendation in how to better reference/cite these data sources.

mponce0 commented 3 years ago

@whedon generate pdf from branch literature

whedon commented 3 years ago 
Attempting PDF compilation from custom branch literature. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

mponce0 commented 3 years ago

@whedon generate pdf from branch literature

whedon commented 3 years ago 
Attempting PDF compilation from custom branch literature. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 3 years ago

:wave: @JDRomano2, please update us on how your review is going.

whedon commented 3 years ago

:wave: @LogIN-, please update us on how your review is going.

LogIN- commented 3 years ago

Hi @fboehm thank you for invitation for the review.

Authors did a good job in developing a package, it can be successfully installed and used.

Here is my opinion on what needs to be changed:

When this self-citation is removed authors should describe in more detail functionality of this R package, similar to their preprint publication, otherwise from current paper many aspects of the package will stay unclear.

Maybe the best solution is to have peer review of their preprint paper and having that published in JOSS.

fboehm commented 3 years ago

@LogIN- thanks for your message. Can you clarify - is the preprint article the same (same title, same text) that is being considered here? Or does it merely have the same title? Thanks again!

mponce0 commented 3 years ago

@LogIN- thanks for your message. Can you clarify - is the preprint article the same (same title, same text) that is being considered here? Or does it merely have the same title? Thanks again!

The title is the same but the pre-print article (~47 pages) is quite lengthy. When considering submitting the manuscript to JOSS we thought about it but when checking the structure and recommended format we felt that it wouldn't fit with this format. Here is a link to the pre-print https://arxiv.org/abs/2009.01091 We will happy to submit the arXiv version instead if that's OK?

mponce0 commented 3 years ago

Hi @fboehm thank you for invitation for the review.

Authors did a good job in developing a package, it can be successfully installed and used.

Here is my opinion on what needs to be changed:

  • Authors should remove self-citation to the same article available to preprint server (Ponce, 2020)

When this self-citation is removed authors should describe in more detail functionality of this R package, similar to their preprint publication, otherwise from current paper many aspects of the package will stay unclear.

Maybe the best solution is to have peer review of their preprint paper and having that published in JOSS.

Hi @LogIN- , many thanks for your review and encouraging words. We just wanted to clarify and expand a little bit on our previous comment to @fboehm , we did not submit the original manuscript from arXiv to JOSS as we thought the format and in particular the length of it wouldn't fit the style of JOSS. In addition to that, we thought of using the JOSS publication as the main point of references to the "code/package" itself while working in additional publications more aimed to applications and introductory tutorials in how to use the package itself. In particular, considering that we will continue working and adding new features to the package we thought of having a very high level description and reference to the code in this JOSS publication. As mentioned above, we are OK with submitting the longer manuscript as show in the arXiv version, but even considering previous JOSS publications (eg. https://doi.org/10.21105/joss.02376) we decided to submit this current shorter version of the paper.

fboehm commented 3 years ago

Yes, that's right, @mponce0 . The purpose of JOSS is to publish the software and, typically, we prefer short articles that discuss the software.

@LogIN- I think it would be ok, and, possibly helpful, to maintain the citation to the preprint, especially considering that it's not the same as the manuscript that accompanies the software submission to JOSS. I agree, @LogIN-, with you encouraging the authors to detail functionality of the R package, while aiming to keep the JOSS manuscript length at ~ 3 pages or less.

Does that seem reasonable?

mponce0 commented 3 years ago

Yes, that's right, @mponce0 . The purpose of JOSS is to publish the software and, typically, we prefer short articles that discuss the software.

@LogIN- I think it would be ok, and, possibly helpful, to maintain the citation to the preprint, especially considering that it's not the same as the manuscript that accompanies the software submission to JOSS. I agree, @LogIN-, with you encouraging the authors to detail functionality of the R package, while aiming to keep the JOSS manuscript length at ~ 3 pages or less.

Does that seem reasonable?

I'm certainly more inclined to keep the JOSS paper short, we are already at 5 pages (a bit above 3 without including references). We will change the title of the pre-print to reflect that arXiv article is the "User's Manual and Examples of Applications" and the JOSS paper as the main reference of the package.

@fboehm There are two main issues with adding the description for the functions: 1) it will increase the length of the manuscript (we added some parts due to some at the request from other reviewer), 2) my main concern is that we will keep adding new functions and capabilities in the future as the situation unfolds and as requested by users (we have already done this a couple of times) and we develop further projects related to the package. Then the description of the functions presented in this paper would turn to be incomplete.

Personally, I think the best would be to use the JOSS paper as the main reference to the package without going into the details of the implementation of the API and allowing users to get the explicit details from the package website and documentation. As mentioned before, I know other paper at JOSS follow this very same approach.

mponce0 commented 3 years ago

@fboehm WRT this last point about the functionality of the package, we went back to the manuscript and try to add a few more sentences detailing some of the functionalities at a very "high" level (end of the "Functionalities and Main Features" section).

Please let us know what do you think.

mponce0 commented 3 years ago

@whedon generate pdf from branch literature

whedon commented 3 years ago 
Attempting PDF compilation from custom branch literature. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

mponce0 commented 3 years ago

@whedon generate pdf from branch literature

whedon commented 3 years ago 
Attempting PDF compilation from custom branch literature. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

mponce0 commented 3 years ago

@whedon generate pdf from branch literature

whedon commented 3 years ago 
Attempting PDF compilation from custom branch literature. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

LogIN- commented 3 years ago

Hi @fboehm & @mponce0, sounds reasonable. I would keep exactly the same title for both and as suggested we can use preprint version as more detail reference.. I checked the latest manuscript build and it's clear that preprint contains more information about the package. If you follow this route and this JOSS publication is the main point for the package I guess this will also become "published version" of that preprint article. In this case I am completely fine in recommending this manuscript for publication.

mponce0 commented 3 years ago

Hi @fboehm & @mponce0, sounds reasonable. I would keep exactly the same title for both and as suggested we can use preprint version as more detail reference.. I checked the latest manuscript build and it's clear that preprint contains more information about the package. If you follow this route and this JOSS publication is the main point for the package I guess this will also become "published version" of that preprint article. In this case I am completely fine in recommending this manuscript for publication.

Hi @LogIN- & @fboehm, Sure we can do that, we can keep the same title and even more from the arXiv page refer to the JOSS version of the article using the DOI and a "comment" line mentioning that it was published in the JOSS journal.

mponce0 commented 3 years ago

@fboehm just wondering whether there is anything else that would be needed from our side to move forward with the review?

fboehm commented 3 years ago

@JDRomano2 - How is your review going? Do you have outstanding issues for the authors to address? Please feel free to check off items in the checklist once you're satisfied.

Thanks again!

fboehm commented 3 years ago

@JDRomano2 - How is the review going? Please let me know if you encounter any issues.

Thanks again!

fboehm commented 3 years ago

@JDRomano2 how is your review going? Do you have additional comments for the authors? Please feel free to check the boxes if the current state of the repository meets the requirements. Thanks again!

fboehm commented 3 years ago

@JDRomano2 - I hope that the review is going well. Please let me know if I can help with anything. Thanks again!

mponce0 commented 3 years ago

@fboehm just wondering whether there is a procedure/protocol to deal with situations in which a reviewer is unreachable (unresponsive) for quite some time? Could this be escalated somehow or perhaps invite a new reviewer?

JDRomano2 commented 3 years ago

@mponce0, please accept my sincerest apologies for the delayed response. An unforeseen personal emergency made this take far longer than it should have.

You have thoroughly replied to and addressed all of my concerns with the original paper. Please note that all check boxes are checked in the review form above.

@fboehm if there is anything else I can do please let me know. All future responses will be prompt.

mponce0 commented 3 years ago

@JDRomano2 Many thanks for your response and so sorry to hear about your personal emergency, I hope everything is OK and thank you again for your careful and thorough review.

fboehm commented 3 years ago

@whedon generate pdf from branch literature

whedon commented 3 years ago 
Attempting PDF compilation from custom branch literature. Reticulating splines etc...
fboehm commented 3 years ago

Thank you, @JDRomano2 and @LogIN- !! Your thorough reviews are really appreciated. I have no remaining tasks or questions for you.

I will double-check the pdf of the paper before proceeding towards publication.

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

fboehm commented 3 years ago

@mponce0 - The references need a few small fixes.

  1. Ponce & Sandhel reference needs "R" capitalized. One way to do this is to verify that the bib file has capital R and put curly braces around the letter R. Upon compiling, this should yield a capitalized R in the references section.
  2. The url for Smith, et al. reference needs to be fixed.
  3. Hornik's article, in the references section, needs a capital R.
  4. Health Canada reference - I think that it needs a capital C for Covid19. Please correct me if I'm mistaken on this.

Please make these small fixes. Once they're done, regenerate the pdf and double-check that the fixes worked. Thanks again!

fboehm commented 3 years ago

I manually checked the DOIs on the downloaded pdf. They all worked and took me to the expected articles.

mponce0 commented 3 years ago

@whedon generate pdf from branch literature

whedon commented 3 years ago 
Attempting PDF compilation from custom branch literature. Reticulating splines etc...