search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
725 stars 38 forks source link

[REVIEW]: Tidytacos: An R package for analyses on taxonomic composition of microbial communities #6313

Open editorialbot opened 10 months ago

editorialbot commented 10 months ago

Submitting author: !--author-handle-->@wsmets<!--end-author-handle-- (Wenke Smets) Repository: https://github.com/LebeerLab/tidytacos Branch with paper.md (empty if default branch): master Version: v0.3.0 Editor: !--editor-->@diazrenata<!--end-editor-- Reviewers: @kelly-sovacool, @david-barnett Archive: Pending

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/0dbd375ebae0c76924830fd01dac7ba6"><img src="https://joss.theoj.org/papers/0dbd375ebae0c76924830fd01dac7ba6/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/0dbd375ebae0c76924830fd01dac7ba6/status.svg)](https://joss.theoj.org/papers/0dbd375ebae0c76924830fd01dac7ba6)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@kelly-sovacool & @david-barnett, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @diazrenata know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @kelly-sovacool

📝 Checklist for @david-barnett

editorialbot commented 10 months ago

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf
editorialbot commented 10 months ago 
Software report:

github.com/AlDanial/cloc v 1.88  T=0.11 s (391.0 files/s, 198012.3 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
SVG                              2              0              0          16171
R                               28            694           1476           2470
Markdown                         7             62              0            446
TeX                              1             26              0            233
YAML                             5             29             15            154
-------------------------------------------------------------------------------
SUM:                            43            811           1491          19474
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository
editorialbot commented 10 months ago

Wordcount for paper.md is 910

editorialbot commented 10 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

editorialbot commented 10 months ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- None

MISSING DOIs

- 10.7287/peerj.preprints.1318 may be a valid DOI for title: A method for simultaneous measurement of soil bacterial abundances and community composition via 16S rRNA gene sequencing
- 10.3390/microorganisms9091797 may be a valid DOI for title: Current applications of absolute bacterial quantification in microbiome studies and decision-making regarding different biological questions
- 10.1101/2022.08.16.504148 may be a valid DOI for title: Leaf side determines the relative importance of dispersal versus host filtering in the phyllosphere microbiome
- 10.1128/spectrum.02420-21 may be a valid DOI for title: Bacterial succession and community dynamics of the emerging leaf phyllosphere in spring
- 10.1128/spectrum.01755-22 may be a valid DOI for title: The Greenhouse Phyllosphere Microbiome and Associations with Introduced Bumblebees and Predatory Mites
- 10.3389/fmicb.2017.02372 may be a valid DOI for title: Comparing the healthy nose and nasopharynx microbiota reveals continuity as well as niche-specificity
- 10.1038/s41598-020-64705-x may be a valid DOI for title: Impact of a lactobacilli-containing gel on vulvovaginal candidosis and the vaginal microbiome
- 10.1128/msphere.00239-22 may be a valid DOI for title: Impacts of menstruation, community type, and an oral yeast probiotic on the vaginal microbiome
- 10.1128/msystems.00056-21 may be a valid DOI for title: Case-control microbiome study of chronic otitis media with effusion in children points at streptococcus salivarius as a pathobiont-inhibiting species
- 10.1016/j.xcrm.2022.100521 may be a valid DOI for title: Selective targeting of skin pathobionts and inflammation with topically applied lactobacilli
- 10.1186/s42523-019-0010-6 may be a valid DOI for title: The microbiome of the invertebrate model host Galleria mellonella is dominated by Enterococcus
- 10.2139/ssrn.3422481 may be a valid DOI for title: Lactobacilli have a niche in the human nose
- 10.3389/fnut.2022.916607 may be a valid DOI for title: Spontaneous riboflavin-overproducing Limosilactobacillus reuteri for biofortification of fermented foods
- 10.1016/j.isci.2021.103306 may be a valid DOI for title: Microbial enrichment and storage for metagenomics of vaginal, skin, and saliva samples
- 10.1016/j.isci.2021.102978 may be a valid DOI for title: The nasal mutualist Dolosigranulum pigrum AMBR11 supports homeostasis via multiple mechanisms
- 10.1016/j.biortech.2022.128285 may be a valid DOI for title: Boosting aerobic microbial protein productivity and quality on brewery wastewater: Impact of anaerobic acidification, high-rate process and biomass age
- 10.1021/acs.est.9b06404.s001 may be a valid DOI for title: Return-sludge treatment with endogenous free nitrous acid limits nitrate production and N2O emission for mainstream partial nitritation/Anammox
- 10.1039/d1ew00525a may be a valid DOI for title: Piloting carbon-lean nitrogen removal for energy-autonomous sewage treatment
- 10.1016/j.biortech.2021.125996 may be a valid DOI for title: Oxygen control and stressor treatments for complete and long-term suppression of nitrite-oxidizing bacteria in biofilm-based partial nitritation/anammox
- 10.1038/s41564-023-01500-0 may be a valid DOI for title: A citizen-science-enabled catalogue of the vaginal microbiome and associated factors

INVALID DOIs

- None
diazrenata commented 10 months ago

@kelly-sovacool, @david-barnett - Thank you for agreeing to review this submission! We'll continue the conversation on the review here.

In case it is helpful, here is the JOSS reviewers' guide: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html

Thank you both!

kelly-sovacool commented 9 months ago

Review checklist for @kelly-sovacool

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

david-barnett commented 9 months ago

Review checklist for @david-barnett

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

david-barnett commented 9 months ago

Hello all, 👋

I started to work through the reviewer checklist today. I have not completed my review of tidytacos but I have some questions and comments already.

Points 1 and 2 seem to me like practical issues worth addressing before proceeding with further review. Points 3 and 4 refer to issues with the manuscript and package documentation, which if addressed would make it easier to continue with a comprehensive and fair review of the package.

1. Checking contribution of submitting author

From the JOSS reviewer guidelines: https://joss.readthedocs.io/en/latest/review_criteria.html#authorship

As part of the review process, you are asked to check whether the submitting author has made a ‘substantial contribution’ to the submitted software (as determined by the commit history)

Wenke Smets is the submitting author, but I did not find evidence of code contributions from this author.

From the commit history on github, I initially found only two commits from @wsmets, adding the manuscript for JOSS submission. https://github.com/LebeerLab/tidytacos/commits/dev?author=wsmets

I later found a couple more commits authored by "wsmets" but not linked to a github account (and so not easily searchable). What I found seemed to be small updates only to the documentation of several functions. https://github.com/LebeerLab/tidytacos/pull/17/commits/8ad017a838e93576823113e30b9a8fabf51e6a0a

@wsmets , is this a fair assessment of your contributions to tidytacos? If not, could you clarify what I missed?

2. Version for review? dev branch is 61 commits ahead of master

It looks like the dev branch is quite active, and seems to include several features and documentation updates? It would be good to get clarity on the short term plans for this development. e.g. should this all be merged to master and tagged as a new version before further review?

3. Reference to similar work

From JOSS reviewer checklist :

State of the field: Do the authors describe how this software compares to other commonly-used packages?

From JOSS review criteria page:

Submissions that implement solutions already solved in other software packages are accepted into JOSS provided that they meet the criteria listed above and cite prior similar work. Reviewers should point out relevant published work which is not yet cited.

Currently the manuscript does not cite or mention similar work. tidytacos provides a generalist framework for handling data about the taxonomic composition of microbiomes, and facilitating the application of various relevant analyses and visualisations. Here are some R packages that address similar aims:

It would be great to see a few words added to the manuscript acknowledging related work. I think there is no need or space for an extensive comparison, but given that there are now so many related packages with overlapping functionality, it would be good to see some justification for the usefulness of tidytacos included in the paper. The philosophy section of the wiki claims that the simplicity of tidytacos' design provides some advantages over packages based around a more complex object structure e.g. phyloseq. I think this is a reasonable line of argument to expand upon in the paper.

From JOSS submission requirements:

Your software should be a significant contribution to the available open source software that either enables some new research challenges to be addressed or makes addressing research challenges significantly better (e.g., faster, easier, simpler).

4. Package documentation and other user guidance

I installed tidytacos from the master branch and ran all the wiki tutorial material locally.

https://github.com/LebeerLab/tidytacos/wiki/Tutorials

This tutorial started by demonstrating: making stacked bar charts; plotting observed ASV richness; making a 3D PCoA using Bray-Curtis dissimilarities computed on genus-level aggregated features. That worked fine, and it was mostly clear enough how I could make some simple changes to modify these procedures.

However, I have some suggestions for the documentation overall:

  1. There appears to be a substantial amount of tidytacos functionality not covered by tutorial material. For example, there are several functions related to network analyses in the network.R file. These have minimal docs, without examples of use. Network analysis functionality is not mentioned in the manuscript either. I think there should be a more comprehensive structured overview available of the package's functionality available, to help guide new users. As well as adding tutorial material or other examples of use for more of the functionality, the authors could also consider structuring the reference page by theme, instead of listing all \~80 functions alphabetically.

  2. In the Differential Abundance Analysis section of the tutorial wiki - the add_codifab and tacoplot_codifab functions are introduced. I do not recall seeing this approach for statistical analysis and visualisation before. It would be great to get a reference if this is an established method? Regardless, the method could be better described and justified in the documentation, as the details are currently unclear without looking at the underlying code. None of the codifab.R functionality is mentioned in the manuscript either.

diazrenata commented 9 months ago

@david-barnett, thank you for digging into the package and raising these points! @wsmets, could you respond to @david-barnett's questions (especially regarding package contributions, development plans, and documentation) before we proceed?

diazrenata commented 9 months ago

Hi @wsmets, just a quick re-up - have you had the chance to respond to @david-barnett's questions?

wsmets commented 8 months ago

@david-barnett, @kelly-sovacool, and @diazrenata, thank you so much for taking the time to look into our tool and considering its publication! As a response to the first points raised by @david-barnett:

  1. As for the limited contribution of the submitting author, I am actually new to GitHub and never tracked my changes to the package before. Besides the improvements to function descriptions, my greatest code contribution consists of four functions regarding the transformation of data to absolute abundances in case a spike was added. This is, to the best of our knowledge, an entirely new feature to the tools of amplicon sequencing analyses, but these four functions compared to the whole package remain a limited one. My other contributions consist more of giving user-based input, joining in meetings of package conceptualization, and supporting the organization and the writing of this publication, hence my role as submitting author. However, if these contributions are considered insufficient, we don’t mind changing the submitting author. @diazrenata, in that case, please let us know how we should proceed.

  2. We are actively maintaining and expanding the package as user-based feedback comes in. We think the package can be reviewed as such, without considering the dev branch.

  3. We think it is a valuable addition to mention existing work as you propose, and we are currently updating the manuscript to include the related packages.

    • More tutorials are indeed a valid point of improvement. We are making additional tutorials and will update you here once they are available.
    • Following your other suggestion, a list of functions sorted thematically was added to the github page here: https://lebeerlab.github.io/tidytacos/reference/index.html#file-handling
    • We are currently also working on a clarification of the principles of codifab in the manuscript.

We want to thank you for your patience while we are implementing your feedback in the package and the manuscript.

diazrenata commented 8 months ago

Hi @wsmets, thanks for the update! It sounds like you've made contributions to the code and manuscript that aren't captured by GitHub's tracking, which I completely understand! Please let us know when you're done making modifications and we can proceed with review.

diazrenata commented 7 months ago

Hi @wsmets, just checking in on progress on the revisions to tidytacos?

wsmets commented 7 months ago

Hi @diazrenata, thanks for checking in. We are working on it, but alternately went on holiday, had lab retreat and conferences. We hope to get back to you in the next 2 weeks!

diazrenata commented 7 months ago

Great! Thanks for the update!

On Thu, Apr 25, 2024 at 1:59 AM Wenke @.***> wrote:

External Email

Hi @diazrenata https://github.com/diazrenata, thanks for checking in. We are working on it, but alternately went on holiday, had lab retreat and conferences. We hope to get back to you in the next 2 weeks!

— Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/6313#issuecomment-2076702923, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEH6DN6YWODPTP7JL3RDN5LY7DAWRAVCNFSM6AAAAABCVVFLMSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZWG4YDEOJSGM . You are receiving this because you were mentioned.Message ID: @.***>

wsmets commented 6 months ago

Dear @david-barnett, @kelly-sovacool, and @diazrenata,

Sorry for the delay, but we have further addressed the points raised by @david-barnett:

wsmets commented 5 months ago

Dear @david-barnett, @kelly-sovacool, and @diazrenata, did you have an opportunity to look at our revised submission already?

diazrenata commented 5 months ago

@david-barnett 👋 When you have a moment, could you look over the revisions to "tidytacos"?

diazrenata commented 5 months ago

@kelly-sovacool 👋 Just checking in on the status of your review here?

david-barnett commented 5 months ago

@david-barnett 👋 When you have a moment, could you look over the revisions to "tidytacos"?

Hi, I'm currently quite swamped, but I'll probably be able to have a look in the week beginning 15th July

kelly-sovacool commented 5 months ago

@kelly-sovacool 👋 Just checking in on the status of your review here?

I plan on getting to this in the next couple weeks!

diazrenata commented 4 months ago

@kelly-sovacool and @david-barnett - just checking in to see how things are going and when you may have the chance to revisit tidytacos?

david-barnett commented 4 months ago

I will try to look today

david-barnett commented 4 months ago

@editorialbot generate pdf

editorialbot commented 4 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

diazrenata commented 4 months ago

@david-barnett Great! Thanks!

david-barnett commented 4 months ago

Summary

I've completed a second round of review, which is the first time I assessed the software itself in depth. I'm pleased with the additional tutorial material available on the pkgdown website, and I still find the base design of tidytacos quite beginner-friendly. I worked through the quick start, stacked barplots, and data processing tutorials, which gave me a good walkthrough of the core functionality of the package. I encountered a few bugs when experimenting with the tools introduced in the tutorials and have filed GitHub issues for these.

My main criticism at this point concerns function-level documentation. Many functions are minimally described on their help pages with no examples of use. Additionally, many functions are not covered in the tutorials, making them difficult to use without reading the underlying code.

I would also like to see a more detailed description/justification of the required structure of the taxonomy table, such as which ranks must be included and the special treatment given to particular variables (e.g., species, taxon_id, sequence). This info would help prevent confusion when users try to import their own data into the tidytacos format.

Lastly, it would be beneficial to get R CMD check passing without warnings on the CI. There are currently documentation-related warnings, which are ignored.

Minor Points on the Revised Article

The comparison with phyloseq in your getting started guide is quite compelling! Consider adding some of the points about the disadvantages of phyloseq's design to the paper and/or the homepage? https://lebeerlab.github.io/tidytacos/articles/getting-started.html

Other Notes

diazrenata commented 3 months ago

@kelly-sovacool Just checking in on your review here?

kelly-sovacool commented 3 months ago

Review summary

tidytacos is a compelling new R package for microbial ecology and microbiome analysis. The paper is well-written and makes a great case for the need the package fills in this space and its usefulness for novices and advanced users alike.

I found the package was easy to install (although there were minor warnings), and the vignettes are overall helpful for demonstrating the package's capabilities. However, there is a critical lack of detail on how users can import their own raw data into the package, as the getting started vignette links users to the source code directly. This is a significant barrier to entry for novice users who wish to analyze their own data. Additionally, as noted by @david-barnett, the documentation lacks examples for many of the functions.

While conducting my review of the package I opened up several issues. I separated these into lists of "Blocking issues" and "Suggestions" below. I leave it up to the discretion of the authors to decide whether to address the suggestions, but in my view the blocking issues do need to be addressed before publication.

Overall, this is a great package that will be a valuable addition to the field, it just needs minor updates before it will be fully ready for publication.

📝 Checklist

Blocking issues

Suggestions

wsmets commented 3 months ago

Thank you both so much for taking the time to look into this package more thoroughly! Your feedback is very valuable in improving our package, and we'll let you know when we got through all your points.

diazrenata commented 3 months ago

Hi @wsmets - just checking in on how revisions are going?

wsmets commented 3 months ago

Hi @diazrenata, it's going well, but slow, since we have a bunch of other work on our plates. We hope to be done in a week or two.

wsmets commented 3 months ago

@editorialbot generate pdf

editorialbot commented 3 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

wsmets commented 3 months ago

Thanks again for diving into the package, @david-barnett and @kelly-sovacool and giving us constructive feedback. Also, we appreciate your appreciation. We have looked into all your points of advice. All GitHub issues were addressed and closed. In short we:

We hope this satisfies everyone and, if not, are happy to receive any feedback.

editorialbot commented 2 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

editorialbot commented 2 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

TheOafidian commented 2 months ago

@editorialbot generate pdf

editorialbot commented 2 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

TheOafidian commented 2 months ago

@editorialbot generate pdf

editorialbot commented 2 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

wsmets commented 2 months ago

Dear @diazrenata , @david-barnett and @kelly-sovacool,

a friendly reminder that we have addressed all comments. We anticipate it will take substantially less time to check these compared to the deep dive into the package you did previously, which hopefully makes it easier to plan in.

Kind regards, Wenke

david-barnett commented 1 month ago

thanks for the reminder! I'll try to take a look this week

david-barnett commented 1 month ago

Hello all,

I've completed my 3rd round of review:

  1. I took a look through my previous issues, and I can confirm that most appear satisfactorily resolved - with a couple of relatively minor exceptions (see below)
  2. I was now able to tick off every item on the 📝 checklist

I think that my currently outstanding issues are now relatively minor, and should not block publication, but I would still recommend the authors take a look.

My previous issues that I can confirm are resolved:

My previous issues that were not fully resolved:

Link to my previous review summary: https://github.com/openjournals/joss-reviews/issues/6313#issuecomment-2252987538

TheOafidian commented 1 month ago

Hi David, many thanks for the in depth reviews as always!

I've tackled your two raised issues and have made some more references to the expected taxa ranks for using tidytacos in the intended way.

diazrenata commented 1 month ago

@kelly-sovacool I believe this is ready for you to revisit when you have the time! If you are satisfied with the authors' revisions, please let us know!

kelly-sovacool commented 1 month ago

Hi all,

A couple of my issues have not been resolved. I added additional comments in the respective issue threads to explain more in depth what I am looking for. In my view, these issues should be straightforward to resolve and would help tidytacos have robust documentation to meet the JOSS requirement for example usage. Thus, I would highly encourage the authors to address them before publishing the package.

The example usage requirement is the last unchecked box on my checklist. Once these are resolved I can give my enthusiastic approval! tidytacos will be a very useful resource for the microbial ecology research community.

wsmets commented 3 weeks ago

Thanks, @kelly-sovacool and @david-barnett for your insights! We know this takes you quite some effort and truly appreciate it!We have made the adaptations as you suggested, and await @diazrenata further instructions.

wsmets commented 2 weeks ago

@diazrenata, we adressed all points as the reviewers wished and were hoping you could give your final approval?