search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
699 stars 36 forks source link

[REVIEW]: OSTIR: open source translation initiation rate prediction #3362

Closed whedon closed 3 years ago

whedon commented 3 years ago

Submitting author: @jeffreybarrick (Jeffrey Barrick) Repository: https://github.com/barricklab/ostir Version: v1.0.6 Editor: @csoneson Reviewer: @ayush9pandey, @standage Archive: 10.5281/zenodo.5227845

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/43d02b32408a161e608b886f63e753c1"><img src="https://joss.theoj.org/papers/43d02b32408a161e608b886f63e753c1/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/43d02b32408a161e608b886f63e753c1/status.svg)](https://joss.theoj.org/papers/43d02b32408a161e608b886f63e753c1)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@ayush9pandey & @standage, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @csoneson know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @ayush9pandey

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @standage

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 3 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @ayush9pandey, @standage it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 3 years ago 
Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.05 s (211.8 files/s, 56546.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                           5            405            229           1417
Markdown                         3             54              0            238
R                                1             49             55            129
TeX                              1              0              1             93
-------------------------------------------------------------------------------
SUM:                            10            508            285           1877
-------------------------------------------------------------------------------

Statistical information for the repository '7dcc6e14fdc7923dad707d8c' was
gathered on 2021/06/11.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Cameron                          7            95             59            1.33
Cameron Roots                   15           554            711           10.91
Jeffrey Barrick                 11           654            391            9.01
croots                          21          2274           3492           49.73
jeffreybarrick                   4          3244            120           29.02

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Cameron                     199          209.5          2.6                6.53
Cameron Roots               140           25.3          4.6                8.57
Jeffrey Barrick             559           85.5          1.4               11.63
croots                     1153           50.7          6.4               10.32
whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1038/nbt.1568 is OK
- 10.1016/B978-0-12-385120-8.00002-4 is OK
- 10.1093/nar/gkt1139 is OK
- 10.1021/acssynbio.0c00394 is OK
- 10.1093/bioinformatics/btq458 is OK
- 10.1186/1748-7188-6-26 is OK
- 10.1002/jcc.21596 is OK
- 10.1038/s41592-018-0046-7 is OK

MISSING DOIs

- None

INVALID DOIs

- None
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

csoneson commented 3 years ago

👋🏼 @jeffreybarrick, @ayush9pandey, @standage - this is the review thread for the submission. All of our communications will happen here from now on.

Both reviewers have checklists at the top of this thread. As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues directly in the software repository. If you do so, please mention this thread so that a link is created (and I can keep an eye on what is happening). Please also feel free to comment and ask questions in this thread. It is often easier to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

Please feel free to ping me (@csoneson) if you have any questions or concerns. Thanks!

standage commented 3 years ago

Hi @csoneson, I am unable to update the review checklist.

danielskatz commented 3 years ago

@whedon re-invite @standage as reviewer

whedon commented 3 years ago

The reviewer already has a pending invite.

@standage please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

standage commented 3 years ago

Github claims that the invite has expired. This happened the first time I accepted the invite IIRC.

danielskatz commented 3 years ago

Ping @openjournals/dev ☝️

csoneson commented 3 years ago

@standage Was this resolved by the reassignment by whedon above (just to check before investigating further)?

standage commented 3 years ago

@csoneson Unfortunately no, I'm still unable to edit the checklist, either by clicking on the boxes or by editing the post/comment.

arfon commented 3 years ago

@whedon re-invite @standage as reviewer

whedon commented 3 years ago

OK, the reviewer has been re-invited.

@standage please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

arfon commented 3 years ago

I've just checked again that there's an invite and it's definitely showing up on GitHub https://github.com/openjournals/joss-reviews/invitations

Screenshot 2021-06-24 at 11 22 47

@standage - perhaps try logging in to GitHub with a different browser before clicking the invite link? I'm not sure what's going on here sorry.

standage commented 3 years ago

I tried on a different machine and this time it worked! Thanks everyone.

csoneson commented 3 years ago

Oh, great! Thanks @danielskatz and @arfon for the help :)

whedon commented 3 years ago

:wave: @standage, please update us on how your review is going (this is an automated reminder).

whedon commented 3 years ago

:wave: @ayush9pandey, please update us on how your review is going (this is an automated reminder).

ayush9pandey commented 3 years ago

👋 @ayush9pandey, please update us on how your review is going (this is an automated reminder).

I am installing the package on my machine to verify the final few checklist items.

standage commented 3 years ago

I successfully installed the software earlier in the week. Review is ongoing.

ayush9pandey commented 3 years ago

I've been having issues with making the software work on my Windows machine due to a dependency (Vienna RNA) that I haven't been able to setup correctly. I have created an issue in their repository hoping for some help: https://github.com/ViennaRNA/ViennaRNA/issues/130 (posting here for tracking the review progress).

csoneson commented 3 years ago

👋🏻 Hi all - just wanted to check in on how the reviews are coming along. Thanks!

standage commented 3 years ago

The authors describe OSTIR, a Python application for predicting translation initiation rates for bacterial mRNAs. For each input sequence, OSTIR identifies putative start codons and reports the corresponding translation initiation rate and several other terms relating to the free energy of the putative binding event.

The calculations performed by OSTIR provide insight into a fundamental biochemical process, with a wide variety of applications in research and industry. It is released under an open-source license that, unlike alternative software packages, permits its integration into custom workflows without burdensome licensing or user registration. OSTIR appears to be well implemented and works as advertised in my hands.

This tool looks like a valuable contribution to the field. However, I have a concern about documentation and a recommendation about testing.

Concern: guidance for evaluating output

In the paper summary, the authors claim—rightly so in my opinion—that the capabilities provided by OSTIR are of potential interest to a broad community and have applications in many areas of research and industry. However, the software documentation is very minimal and suitable only for a very narrow audience; that is, folks well-versed in biochemistry and models of RNA thermodynamics. For e.g. a newcomer to the field or a potentially interested user from an adjacent field, the documentation provides little to help the user interpret and evaluate the results of OSTIR output.

To be clear, a comprehensive exposition of RNA thermodynamics is out of scope for this submission, and a rigorous evaluation of the corresponding models is out of scope for this review. But if OSTIR is to be truly accessible to the broad audience the authors describe in their paper, it's not unreasonable to expect that the authors provide at least minimal guidance for interpreting the output of the software. Doing so would certainly be in the spirit of the open source community ethos embraced by JOSS.

As part of my evaluation of OSTIR, I grabbed a handful of bacterial mRNA sequences (coding sequence + some flanking sequence to approximate UTRs on both sides) from published genomes and compared the OSTIR output to output for random sequences of 100 [ACGT] characters. There is little in the documentation that would help me answer the following questions.

The answers to these questions are no doubt obvious to the authors—and to a narrow audience with an intuitive understanding of RNA thermodynamics—but not as obvious to many in the broader genome informatics and bioinformatics communities. Providing some basic guidance for interpreting the OSTIR output would be a valuable improvement to this submission.

Recommendation: configure a continuous integration environment

The OSTIR source code distribution includes an automated test suite that is presumably run routinely by the authors. I'd recommend that the authors configure a continuous integration (CI) environment that automatically executes the test suite on a frequent basis.

CI can be configured to:

  1. run on a schedule (i.e. a monthly cron job); or
  2. run on an event trigger (e.g. whenever the repository is updated); or
  3. both

However the CI is configured, each CI build starts with a fresh VM and builds the environment from scratch using instructions provided by the authors. This has numerous benefits.

Numerous CI providers are available for free to open source software projects: GitHub Actions is an obvious choice built right into the GitHub platform, but third-party providers such as Drone CI and Circle CI are also available.

csoneson commented 3 years ago

Thanks @standage!

@ayush9pandey - did you manage to solve the installation issues?

ayush9pandey commented 3 years ago

The authors present an open source software package to predict translation initiation rates of mRNAs in bacteria. The primary goal of this package is to provide an open source and command line tool that can be used for translation rate prediction since other tools with similar features are not open source yet. In particular, the widely used software RBS calculator is proprietary and can only be used through a webpage. Towards that end, OSTIR opens up new and interesting possibilities as it can become a part of a bigger pipeline to analyze/design biological systems.

Some comments:

  1. There are two suggested approaches to install OSTIR - one using bioconda and another using PyPi distribution. The installation using bioconda does not require the user to install ViennaRNA (an important OSTIR dependency) manually. However, to use the PyPi distribution of OSTIR, the user needs to install ViennaRNA before installing OSTIR. For Windows machines, I was not able to get ViennaRNA to run successfully and had to switch to a linux machine to get it all running. So, I would suggest that the authors recommend the bioconda installation method in their README for the package. Also, if possible, detailed instructions for Windows installations could help as well.
  2. The GitHub does not provide a contribution guidelines document/page and the issues page could include a few labeled topics (such as help-wanted, enhancement, etc.) that may help a new open source developer contribute to the project.
  3. For a scientist who is a potential user of OSTIR, the README documentation is a good start but it could be improved with a dedicated documentation page (either using GitHub wiki or another open source documentation web service).
  4. A detailed API documentation would also help any developers include OSTIR as a part of their software pipelines. Since a major contribution of OSTIR is that it provides an open source code that can predict translation rate initiation given the mRNA sequences, I think various tools could use the OSTIR API. For example, a software designed for forward design of synthetic biological circuits could massively benefit from OSTIR outputs.
csoneson commented 3 years ago

👋🏻 @jeffreybarrick - just checking in. Let us know if you have any questions about the reviews provided above!

jeffreybarrick commented 3 years ago

@csoneson Thanks for checking in. We are working on these recommendations.

jeffreybarrick commented 3 years ago

Thank you @standage and @ayush9pandey for the helpful feedback! And thanks @csoneson for organizing the review process.

Here's a run-down of the updates we made to improve OSTIR in response to each of your comments. Let us know if we missed anything or if you have additional comments/suggestions.

As part of this effort, we re-organized the online documentation, which had been a single README.md, into a set of GitHub wiki pages. The main README.md is now a shorter Quickstart guide that links to the more detailed documentation on the Wiki.

Reviewer 1 (@standage)

Concern: guidance for evaluating output

Great suggestion. We added a Background Page to the Wiki that has the following information:

The last bullet point is especially relevant to the reviewer's questions about interpreting the predicted expression values. It shows that a true start codon (one that begins an annotated gene) gives a prediction of roughly 1000 on average. Most "random" potential start codons elsewhere in the genome give scores below 10.

Recommendation: configure a continuous integration environment

Thanks for the suggestion to get this set up at the outset to make long-term maintenance easier.

We set up GitHub Actions for continuous integration. The build and test action performs linting and Linux/MacOSX installation and consistency tests. We have it configured as both a triggered and scheduled (monthly) task and added a corresponding passing/failed badge to the main OSTIR GitHub page.

Reviewer 2 (@ayush9pandey)

1. Installation instructions/issues

We updated the installation instructions as suggested to recommend using bioconda. However, there isn't a Windows conda package for the ViennaRNA requirement, so we also kept the previous pypi directions for a Windows install with a proviso that it is an 'expert option'.

We're not sure about your difficulty installing on Windows. Our best guess is that the installation issue was a $PATH setup issue where the terminal was not finding the ViennaRNA executables on the reviewer's machine as described here ViennaRNA/ViennaRNA#130.

One of us (@croots) is a Windows user and has tried hard to maintain compatibility and consistency on this platform throughout OSTIR development, including by submitting issues to the ViennaRNA team when we found problems with earlier versions of ViennaRNA on Windows. We hope to maintain Windows functionality going forward, but recognize that this will be more of a challenge than it is for Linux/MacOS.

2. Labeled topics for encouraging contributions

Great idea. We added a couple of ideas for enhancements and a help-wanted item to kick things off on the GitHub Issues Tracker for the project. We also added a short Developer Information Wiki page that includes a code of conduct.

3. Dedicated documentation

We expanded the documentation into a set of GitHub wiki pages and changed the README.md to a Quickstart guide. Much of the content here is the same as it was in the original README.md (just re-organized into different pages), but we did add entirely new Background and Developer Information pages.

4. API Documentation

OSTIR exposes the Python function run_ostir() as an entry point as described on the updated Python Module Usage Wiki page. At this point, we think this is the full extent of an API that will be useful to users/developers of OSTIR.

standage commented 3 years ago

Thanks for these responses @jeffreybarrick. The new background page in the wiki is indeed insightful, and the CI builds will hopefully ensure that OSTIR remains a valuable contribution to the community for many years requiring minimal maintenance.

@csoneson, I have no further concerns and recommend this submission for publication.

csoneson commented 3 years ago

Thank you @standage! Could you please tick the last box in the checklist above as well, so that everything is documented?

ayush9pandey commented 3 years ago

Thanks for all the updates to the software @jeffreybarrick! @csoneson I have completed all checkpoints for my review and can recommend this software for publication in JOSS.

csoneson commented 3 years ago

Thank you @ayush9pandey!

@jeffreybarrick - I will take a quick look as well and get back to you shortly with the next steps.

csoneson commented 3 years ago

@whedon generate pdf

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

csoneson commented 3 years ago

@whedon check references

whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1038/nbt.1568 is OK
- 10.1016/B978-0-12-385120-8.00002-4 is OK
- 10.1093/nar/gkt1139 is OK
- 10.1021/acssynbio.0c00394 is OK
- 10.1093/bioinformatics/btq458 is OK
- 10.1186/1748-7188-6-26 is OK
- 10.1002/jcc.21596 is OK
- 10.1038/s41592-018-0046-7 is OK

MISSING DOIs

- None

INVALID DOIs

- None
csoneson commented 3 years ago

👋🏻 @jeffreybarrick - I have looked through the submission and overall this looks good to me. A couple of things I noticed in the paper:

Could you please fix these details (you can then ask whedon to generate a new pdf proof)? The next steps would then be to:

I can then move forward with accepting the submission.

jeffreybarrick commented 3 years ago

We think we've fixed all of these issues in the paper:

jeffreybarrick commented 3 years ago

@whedon generate pdf

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

jeffreybarrick commented 3 years ago

Version tag for archived version: v.1.0.6 https://github.com/barricklab/ostir/releases/tag/v1.0.6

DOI for Archived Version: 10.5281/zenodo.5227845 https://zenodo.org/record/5227845#.YSABb9NKjm0

Thanks @csoneson for walking us through the steps! Let us know if there's anything else to correct.

csoneson commented 3 years ago

@whedon set 10.5281/zenodo.5227845 as archive

whedon commented 3 years ago

OK. 10.5281/zenodo.5227845 is the archive.

csoneson commented 3 years ago

@whedon set v1.0.6 as version

whedon commented 3 years ago

OK. v1.0.6 is the version.

csoneson commented 3 years ago

Looks good! I'm handing over now to the associate EiC on rotation for the final steps.

csoneson commented 3 years ago

@whedon recommend-accept

whedon commented 3 years ago 
Attempting dry run of processing paper acceptance...
whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1038/nbt.1568 is OK
- 10.1016/B978-0-12-385120-8.00002-4 is OK
- 10.1093/nar/gkt1139 is OK
- 10.1021/acssynbio.0c00394 is OK
- 10.1093/bioinformatics/btq458 is OK
- 10.1186/1748-7188-6-26 is OK
- 10.1002/jcc.21596 is OK
- 10.1038/s41592-018-0046-7 is OK

MISSING DOIs

- None

INVALID DOIs

- None