search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
721 stars 38 forks source link

[REVIEW]: BioProv - A provenance library for bioinformatics workflows #3622

Closed whedon closed 3 years ago

whedon commented 3 years ago

Submitting author: @vinisalazar (Vinícius Salazar) Repository: https://github.com/vinisalazar/BioProv/ Version: v0.1.24 Editor: @jmschrei Reviewer: @maximtrp, @Zethson Archive: 10.5281/zenodo.5550568

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/3614862df6bf7936838eaa786664deea"><img src="https://joss.theoj.org/papers/3614862df6bf7936838eaa786664deea/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/3614862df6bf7936838eaa786664deea/status.svg)](https://joss.theoj.org/papers/3614862df6bf7936838eaa786664deea)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@maximtrp & @Zethson, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @jmschrei know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @maximtrp

✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @Zethson

✨ Important: Please do not use the Convert to issue functionality when working through this checklist, instead, please open any new issues associated with your review in the software repository associated with the submission. ✨

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 3 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @maximtrp, @Zethson it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 3 years ago 
Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.13 s (383.0 files/s, 62889.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          26           1008           1283           3809
Markdown                         5            108              0            333
YAML                             7             17             16            244
reStructuredText                 7             60            110             76
Jupyter Notebook                 2              0            860             67
DOS Batch                        1              8              1             26
make                             1              4              7              9
-------------------------------------------------------------------------------
SUM:                            49           1205           2277           4564
-------------------------------------------------------------------------------

Statistical information for the repository '915916be15cac6761b6442ac' was
gathered on 2021/08/16.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Vini Salazar                   519         12583           6617           95.88
jvfe                            23           622            202            4.12

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Vini Salazar               5640           44.8          6.6               10.71
jvfe                        446           71.7          5.4                4.04
whedon commented 3 years ago

PDF failed to compile for issue #3622 with the following error: 

 Can't find any papers to compile :-(
jmschrei commented 3 years ago

@whedon generate pdf from branch joss-paper

whedon commented 3 years ago 
Attempting PDF compilation from custom branch joss-paper. Reticulating splines etc...
whedon commented 3 years ago

PDF failed to compile for issue #3622 with the following error: 

 /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-b63fc70cc085/lib/whedon/author.rb:72:in `block in build_affiliation_string': Problem with affiliations for Vinícius Salazar, perhaps the affiliations index need quoting? (RuntimeError)
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-b63fc70cc085/lib/whedon/author.rb:71:in `each'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-b63fc70cc085/lib/whedon/author.rb:71:in `build_affiliation_string'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-b63fc70cc085/lib/whedon/author.rb:17:in `initialize'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-b63fc70cc085/lib/whedon.rb:205:in `new'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-b63fc70cc085/lib/whedon.rb:205:in `block in parse_authors'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-b63fc70cc085/lib/whedon.rb:202:in `each'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-b63fc70cc085/lib/whedon.rb:202:in `parse_authors'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-b63fc70cc085/lib/whedon.rb:93:in `initialize'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-b63fc70cc085/lib/whedon/processor.rb:38:in `new'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-b63fc70cc085/lib/whedon/processor.rb:38:in `set_paper'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-b63fc70cc085/bin/whedon:58:in `prepare'
    from /app/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
    from /app/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
    from /app/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
    from /app/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
    from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-b63fc70cc085/bin/whedon:131:in `<top (required)>'
    from /app/vendor/bundle/ruby/2.6.0/bin/whedon:23:in `load'
    from /app/vendor/bundle/ruby/2.6.0/bin/whedon:23:in `<main>'
jmschrei commented 3 years ago

@vinisalazar the paper still doesn't appear to be working. Would you mind looking into that?

jmschrei commented 3 years ago

Howdy @maximtrp and @Zethson

Thanks for agreeing to review this submission.

The process for conducting a review is outlined above in the checklist from @whedon. Please check the boxes during your review to keep track, as well as making comments in this thread or opening issues in the repository itself to point out issues you encounter. Keep in mind that our aim is to improve the submission to the point where it is of high enough quality to be accepted, rather than to provide a yes/no decision, and so having a conversation with the authors is encouraged rather than providing a single review post at the end of the process.

Here are the review guidelines: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html And here is a checklist, similar to above: https://joss.readthedocs.io/en/latest/review_checklist.html

Please let me know if you encounter any issues or need any help during the review process, and thanks for contributing your time to JOSS and the open source community!

vinisalazar commented 3 years ago

@jmschrei thank you for letting me know. Both the Whedon web preview service and the Dockerfile compilation are working for me, but I'll investigate.

vinisalazar commented 3 years ago

@whedon generate pdf from branch joss-paper

whedon commented 3 years ago 
Attempting PDF compilation from custom branch joss-paper. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

vinisalazar commented 3 years ago

@jmschrei there was a subtle formatting mistake but I believe it is now fixed. Again, thank you for your work.

jmschrei commented 3 years ago

Great, thank you so much!

whedon commented 3 years ago

:wave: @Zethson, please update us on how your review is going (this is an automated reminder).

whedon commented 3 years ago

:wave: @maximtrp, please update us on how your review is going (this is an automated reminder).

jmschrei commented 3 years ago

@Zethson @maximtrp can you please provide an update on how your review is going? Thanks!

Zethson commented 3 years ago

Planned for next week. Still on holiday. Friday, 10 September 2021, 09:50PM +02:00 from Jacob Schreiber @.*** :

@.*** @maximtrp can you please provide an update on how your review is going? Thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or unsubscribe .

maximtrp commented 3 years ago

@jmschrei I have done the general checks and now testing the package.

Zethson commented 3 years ago

@jmschrei I cannot edit the checklists although this is not my first review. The invite in the OP has expired already. Got any idea?

danielskatz commented 3 years ago

@whedon re-invite @Zethson as reviewer

whedon commented 3 years ago

OK, the reviewer has been re-invited.

@zethson please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

Zethson commented 3 years ago

@danielskatz thank you. Works.

Zethson commented 3 years ago

Dear @vinisalazar,

thank you very much for your submission. I had quite some fun reading it and think that your approach to the outlined issue is very interesting. I come more from a classical bioinformatics workflow background and found your take with a "Project" centric view a novel view on the problem.

I have a couple of points/questions that I would be very happy to see addressed:

  1. What is your reasoning behind making the Project be executable with BioProv? I personally think that traditional workflow engines such as Nextflow or Snakemake already do this but with many many more features (inherent parallelization, container support, cloud support, [...]).
  2. BioProv is supposed to improve reproducibility, but the lacking container support and computational portability make this difficult. The workflow engines also track execution time, execution place etc. What novelty does the W3C-Prov report introduce to improve reproducibility that the workflow engines do not yet cover?
  3. I personally think that the USP of BioProv should be the generation of W3C-Prov compatible reports. What do you think about introducing a Nextflow DSL2 module or Snakemake module to facilitate the generation of such reports for all pipelines automatically?

Thank you very much. Really looking forward to your answers. - Lukas

vinisalazar commented 3 years ago

Dear @Zethson,

Thank you for your comments. Here are our responses:

  1. The main rationale behind this is to be able to consume processes' start and end time, standard output and error, and other relevant information from within the main BioProv process. We aimed to provide a simple framework that does this using Python standard library modules. I agree that there are excellent open-source workflow engines and I use them daily, but we wanted BioProv to be able to run programs natively (and not to depend on third-party tools), even with limited functionality in comparison to more established frameworks. We have added these arguments to the text to make it clearer.

  2. The main novelty of the BioProv report in comparison to other engines is representing a workflow instance (a project) as a W3C-PROV compatible object, with support for both prospective (i.e. information previous to a project's execution) and retrospective provenance. The W3C-PROV recommendation is a W3C standard that has extensive technical reasoning behind it. Our package aims to introduce how this standard can be utilized and disseminated in bioinformatics, to improve the integration of bioinformatic analyses with other knowledge bases.

  3. I would be delighted to see a Nextflow or Snakemake plugin offering generation of W3C-PROV reports. I believe that the latter is more feasible since BioProv and Snakemake are both written in Python. However, we did want to introduce a standalone library as a proof of concept on how to fit bioinformatics workflows into the existing W3C-PROV framework. We hope to attract contributors to our open source code to potentially develop these plugins.

Please let us know if there are any other questions.

Best wishes,

Vinícius, on behalf of the BioProv team

Zethson commented 3 years ago

Dear @vinisalazar,

thank you very much for your answers. I agree with your statements. Nevertheless, I still think that that a plugin for a workflow language could have good impact and hope that you will revisit this post publication (if no one from the OS community volunteers).

I found a typo in line 170: should be "containing"

@jmschrei I consider the paper to be well written with a novel view on the outlined issue. The paper is accompanied with useful tutorials and examples enabling a quick start.

I would like to recommend it for publication.

Cheers

vinisalazar commented 3 years ago

Thank you very much for your review. I have fixed the typo that was mentioned.

jmschrei commented 3 years ago

Great, thanks so much for your review @Zethson !

jmschrei commented 3 years ago

@maximtrp can you provide an update on how your testing is going? Thanks!

maximtrp commented 3 years ago

@jmschrei I have almost finished it. Will be ready in a few days.

jmschrei commented 3 years ago

@maximtrp can you provide an update on how it's going? Thanks!

maximtrp commented 3 years ago

@jmschrei I have finished the review.

jmschrei commented 3 years ago

@vinisalazar can you please provide an update on how your response to the review is going?

vinisalazar commented 3 years ago

@jmschrei thank you for the message, and @maximtrp thank you for the review.

I have addressed and closed the issues raised. Please let me know if there are any pendencies.

Best, Vinícius

jmschrei commented 3 years ago

Great, thanks so much! @maximtrp and @Zethson, can you confirm that the response has satisfied your concerns?

Zethson commented 3 years ago

Yes.

maximtrp commented 3 years ago

@jmschrei Yes

jmschrei commented 3 years ago

@whedon generate pdf from branch joss-paper

whedon commented 3 years ago 
Attempting PDF compilation from custom branch joss-paper. Reticulating splines etc...
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

jmschrei commented 3 years ago

@whedon check references

jmschrei commented 3 years ago

The paper looks good to me. @vinisalazar can you make a tagged release and upload it to Zenodo to get a doi?

vinisalazar commented 3 years ago

Hello @jmschrei, thank you for conducting the editing and review process. Is it okay if I upload the latest tag, v0.1.24? That's the current version that's available on PyPI, with all the changes requested from the reviews.

jmschrei commented 3 years ago

Yes, that should be fine as long as it contains all the changes that the reviewers asked for.

jmschrei commented 3 years ago

Can you post the doi here once it's been uploaded?

vinisalazar commented 3 years ago

Apologies, I hadn't realised that had already been done by my CI setup. Here it is: https://doi.org/10.5281/zenodo.5550568

This version contains all changes requested by the reviewers. Please note that the current tag (v0.1.24) is different from the originally submitted version (v0.1.23).

Please let me know if anything else is needed.

Best wishes, Vinícius

jmschrei commented 3 years ago

@whedon set 10.5281/zenodo.5550568 as archive

whedon commented 3 years ago

OK. 10.5281/zenodo.5550568 is the archive.

jmschrei commented 3 years ago

@whedon set v0.1.24 as version

whedon commented 3 years ago

OK. v0.1.24 is the version.