search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
722 stars 38 forks source link

[REVIEW]: kiwiPy: Robust, high-volume, messaging for big-data and computational science workflows #2351

Closed whedon closed 4 years ago

whedon commented 4 years ago

Submitting author: @muhrin (Martin Uhrin) Repository: https://github.com/aiidateam/kiwipy Version: v0.6.1 Editor: @danielskatz Reviewer: @dghoshal-lbl, @uellue Archive: 10.5281/zenodo.3970461

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/9e125b72f716db5303d3f4437896a157"><img src="https://joss.theoj.org/papers/9e125b72f716db5303d3f4437896a157/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/9e125b72f716db5303d3f4437896a157/status.svg)](https://joss.theoj.org/papers/9e125b72f716db5303d3f4437896a157)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@dghoshal-lbl & @uellue, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @danielskatz know.

✨ Please try and complete your review in the next six weeks ✨

Review checklist for @dghoshal-lbl

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @uellue

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 4 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @dghoshal-lbl, @uellue it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 4 years ago 
Reference check summary:

OK DOIs

- 10.1063/1.4812323 is OK
- 10.1007/s11837-013-0755-4 is OK
- 10.1016/j.commatsci.2012.02.005 is OK
- 10.1088/2515-7639/ab13bb is OK
- 10.1109/MCSE.2012.16 is OK
- 10.1002/cpe.3505 is OK
- 10.1016/j.commatsci.2015.09.013 is OK
- 10.21105/joss.01844 is OK
- 10.21105/joss.01844 is OK

MISSING DOIs

- None

INVALID DOIs

- None
whedon commented 4 years ago

:point_right: Check article proof :page_facing_up: :point_left:

danielskatz commented 4 years ago

👋 @dghoshal-lbl & @uellue - thanks for agreeing to review this submission. Please be sure to read the comments above, and let me know if you have any questions. Basically, your job is to check the article proof and repository and check items off your checklist above. (And note that you will have to accept the invitation before you will be able to do so.)

If you see small problems that need to be discussed, feel free to discuss them here. But if you can, create a new issue in the target repository and link to this review thread in that issue to create a corresponding breadcrumb trail here.

I look forward to seeing how this review goes

uellue commented 4 years ago

Regarding "State of the field":

Comparison with Pika

After a quick side-by-side comparison between the RabbitMQ documentation and kiwiPy, it is not clear to me what the difference between the default Pika Python API and the kiwiPy API is. The code of the Pika-based RabbitMQ examples and the kiwiPy examles are quite similar, and at least to my eyes the kiwiPy version is not a noticeable improvement, different from what is claimed in the paper.

(edit: source code formatting)

Pika https://www.rabbitmq.com/tutorials/tutorial-two-python.html ```python #!/usr/bin/env python import time import pika connection = pika.BlockingConnection( pika.ConnectionParameters(host='localhost')) channel = connection.channel() channel.queue_declare(queue='hello') def callback(ch, method, properties, body): print(" [x] Received %r" % body) time.sleep(body.count(b'.')) print(" [x] Done") channel.basic_consume( queue='hello', on_message_callback=callback, auto_ack=True) print(' [*] Waiting for messages. To exit press CTRL+C') channel.start_consuming() ```
kiwiPy https://github.com/aiidateam/kiwipy/blob/develop/README.rst ```python # -*- coding: utf-8 -*- import time import threading import kiwipy print(' [*] Waiting for messages. To exit press CTRL+C') def callback(_comm, task): print((' [x] Received %r' % task)) time.sleep(task.count('.')) print(' [x] Done') return task with kiwipy.connect('amqp://127.0.0.1/') as communicator: communicator.add_task_subscriber(callback) try: threading.Event().wait() except KeyboardInterrupt: pass ```

Comparison with other work distribution and messaging systems

A quick list of major distributed work scheduling systems with Python API that might deserve a comparison:

The paper and documentation should make much clearer what makes kiwiPy unique and for what particular use cases it is advantageous. Perhaps it can also reference the appropriate parts of RabbitMQ documentation since it is essentially a Python API for it, as far as I understood.

uellue commented 4 years ago

Regarding "References": A reference to the default "Pika" Python API for RabbitMQ https://pika.readthedocs.io/en/stable/ seems to be missing.

uellue commented 4 years ago

Regarding "Community guidelines": https://github.com/aiidateam/kiwipy/issues/65

uellue commented 4 years ago

@danielskatz @muhrin This concludes my first round of review. Looking forward to your responses!

muhrin commented 4 years ago

@uellue, many thanks for the helpful review. I'll coordinate with my coauthor and address the points you've raised.

danielskatz commented 4 years ago

👋 @muhrin - any update at this point?

muhrin commented 4 years ago

We've addressed @uellue issue regarding contributor guidelines by adding wiki entry with is now linked to in the docs.
I've yet to find the time to address the other points raise but should find the time in the coming days. I'll report back as soon as I can.

muhrin commented 4 years ago

Dear @danielskatz,

we once again thank @uellue for his thorough review which has helped to improve both the documentation and the paper itself. We have made the following changes to address the comments made:

This concludes our changes in response to the review.

We look forward to the continuation of the review process.

uellue commented 4 years ago

Dear @muhrin,

many thanks for the update! The new content is a great help for readers to understand the use cases and strengths of kiwiPy. The comparison with Pika and other systems sounds fair and objective. :+1: This fully addresses my previous comments and from my perspective the paper is good to go. 🚀 Congratulations!

CC @danielskatz

muhrin commented 4 years ago

Thanks @uellue . Indeed, the library has certainly benefited from these additions. Thanks again for your review.

uellue commented 4 years ago

Happy to hear that you found it helpful! :-)

danielskatz commented 4 years ago

👋 @dghoshal-lbl - how is your review coming along?

dghoshal-lbl commented 4 years ago

I finished my first round of review for the software and the paper. Some of the paper related concerns have already been addressed. I have a couple of more questions regarding the paper:

i) Has the library been tested to show any significant performance overheads? Mainly because this adds another level of abstraction to RabbitMQ.

ii) Since one of the use-cases for kiwiPy is to restart partially completed workflows and HPC jobs, I was wondering if there are any specific configurations in kiwiPy that enables this or if this is a default behavior?

Regarding the software, I was able to install it but not test its functionality properly:

i) I could not find any module or scripts that would let me test the library.

ii) When I wrote my own client and server following the examples, I ran into the following error: ImportError: cannot import name 'AsyncContextManager'

I also see that there are a few examples in the source tree. I can dig deeper into why I am getting this error, but I think it will be good to document the steps to quickly test the examples or provide scripts for validating the tests.

danielskatz commented 4 years ago

👋 @muhrin - I think we're waiting for your response to the issues @dghoshal-lbl has raised...

muhrin commented 4 years ago

Thanks @danielskatz.

Thank @dhhagan for your review. To answer your points:

1.i) I'm running some benchmarks comparing kiwiPy with pika (the 'default' python library for interacting with RabbitMQ). I'll post back with results shortly.

1.ii) Yes, this is the default. KiwiPy (in the language of RMQ) uses persistent messages and durable queues/exchanges so it's like this out of the box. We tried to highlight this in the paper by stating:

"In kiwiPy, all messages are saved to disk by RabbitMQ, meaning that any or all systems involved in a workflow, including the broker, can be shut down (abruptly or gracefully), and the previous state can be recreated allowing the workflow to continue when the necessary resources are brought back online."

If you feel that this could benefit from some specific clarification we would be happy to add it.

In an HPC setting this persistence can be exploited by submitting a python script to the scheduler that works on one or more kiwiPy tasks. If the scheduler job is terminated the running task is automatically re-inserted back into kiwiPy's queue and will be picked up the next time that job script is submitted. We have deliberately tried to keep kiwiPy's documentation somewhat use-case agnostic as HPC is just one setting where it can be useful, however there is a detailed description in an upcoming AiiDA paper of a similar usage scenario.

2.i) As far as testing kiwiPy, thanks for pointing out that this isn't documented. I will add it to the installation instructions shortly. Briefly, for someone that has cloned the source three it is just a matter of:

pip install -e .[rmq,tests]
pytest test

2.ii) Regarding the AsyncContextManager exception, would you be so kind as to open an issue here with your client/server code and I will have a look?

muhrin commented 4 years ago

Dear @dghoshal-lbl ,

I'm writing back to share changes addressing the points you've raised.

I hope this helps to sufficiently addresses the points you've raised.

We look forward to the continuation of the review process.

dghoshal-lbl commented 4 years ago

@muhrin Thanks for the update and adding the tests. Your answers, benchmark results and tests address most of my questions.

The only thing that remains is the import error for AsyncContextManager, for which I have created an issue and added the details. I received the same error while running the tests you provided. Let me know if you need anything else.

muhrin commented 4 years ago

@danielskatz , just a quick update to say that I'm liasinging with @dghoshal-lbl to figure out the source of the exception they are experiencing. The issue is being dealt with here https://github.com/aiidateam/kiwipy/issues/75.

dghoshal-lbl commented 4 years ago

@muhrin I was able to run the test following our discussion in the github issue thread. All my questions have been addressed and I think the paper is good to go. Congratulations!

Just a couple of minor comments, which might be worth fixing: i) It would be good to mention the required python version. ii) For the test suite, it might be worth mentioning what all services should be running prior to running the tests. For example, the tests that use rabbitmq fail if the server is not running.

danielskatz commented 4 years ago

@muhrin - are you working on the last couple of things from @dghoshal-lbl ? If so, can you let us know when they are done?

danielskatz commented 4 years ago

@dghoshal-lbl & @uellue - I think everything is done from your point of view (other than the minor comments from @dghoshal-lbl two comments above this) - please confirm

muhrin commented 4 years ago

Thanks @dghoshal-lbl !

I've created a PR to address the final points, this should get merged soon: https://github.com/aiidateam/kiwipy/pull/76.

uellue commented 4 years ago

@danielskatz Good to go from my perspective! :-)

dghoshal-lbl commented 4 years ago

@danielskatz It's all good from my side too.

danielskatz commented 4 years ago

@muhrin - please go ahead and merge your remaining PR.

Then, make sure the paper is complete and accurate, and

I can then move forward with accepting the submission, which include me proofreading the paper again.

muhrin commented 4 years ago

Thanks @danielskatz . So, here it is:

Tagged version is: https://github.com/aiidateam/kiwipy/tree/v0.6.1

Zenodo archive can be found here: https://zenodo.org/record/3970461.

Metadata matches that of submission.

DOI is: 10.5281/zenodo.3970461

Let us know if you need anything else to proceed.

danielskatz commented 4 years ago

@whedon set 10.5281/zenodo.3970461 as archive

whedon commented 4 years ago

OK. 10.5281/zenodo.3970461 is the archive.

danielskatz commented 4 years ago

@whedon set v0.6.1 as version

whedon commented 4 years ago

OK. v0.6.1 is the version.

danielskatz commented 4 years ago

@whedon accept

whedon commented 4 years ago 
Attempting dry run of processing paper acceptance...
whedon commented 4 years ago 
Reference check summary:

OK DOIs

- 10.1063/1.4812323 is OK
- 10.1007/s11837-013-0755-4 is OK
- 10.1016/j.commatsci.2012.02.005 is OK
- 10.1088/2515-7639/ab13bb is OK
- 10.1109/MCSE.2012.16 is OK
- 10.1002/cpe.3505 is OK
- 10.1016/j.commatsci.2015.09.013 is OK
- 10.21105/joss.01844 is OK
- 10.21105/joss.01844 is OK

MISSING DOIs

- None

INVALID DOIs

- None
whedon commented 4 years ago

:wave: @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof :point_right: https://github.com/openjournals/joss-papers/pull/1610

If the paper PDF and Crossref deposit XML look good in https://github.com/openjournals/joss-papers/pull/1610, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true
danielskatz commented 4 years ago

@muhrin - There are 2 PRs for small issues in the paper and the bib

Please let me know when these are merged, or if you want to discuss them

danielskatz commented 4 years ago

@whedon accept

whedon commented 4 years ago 
Attempting dry run of processing paper acceptance...
whedon commented 4 years ago 
Reference check summary:

OK DOIs

- 10.1063/1.4812323 is OK
- 10.1007/s11837-013-0755-4 is OK
- 10.1016/j.commatsci.2012.02.005 is OK
- 10.1088/2515-7639/ab13bb is OK
- 10.1109/MCSE.2012.16 is OK
- 10.1002/cpe.3505 is OK
- 10.1016/j.commatsci.2015.09.013 is OK
- 10.21105/joss.01844 is OK
- 10.21105/joss.01844 is OK

MISSING DOIs

- None

INVALID DOIs

- None
whedon commented 4 years ago

:wave: @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof :point_right: https://github.com/openjournals/joss-papers/pull/1616

If the paper PDF and Crossref deposit XML look good in https://github.com/openjournals/joss-papers/pull/1616, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true
danielskatz commented 4 years ago

@whedon accept deposit=true

whedon commented 4 years ago 
Doing it live! Attempting automated processing of paper acceptance...
whedon commented 4 years ago

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

whedon commented 4 years ago

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

  1. Check final PDF and Crossref metadata that was deposited :point_right: https://github.com/openjournals/joss-papers/pull/1617
  2. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/joss.02351
  3. If everything looks good, then close this review issue.

  4. Party like you just published a paper! 🎉🌈🦄💃👻🤘

    Any issues? Notify your editorial technical team...

danielskatz commented 4 years ago

Thanks to @dghoshal-lbl & @uellue for reviewing!

Congratulations to @muhrin (Martin Uhrin) and co-author!

danielskatz commented 4 years ago

@openjournals/dev - the DOI works but the PDF doesn't seem to load - can you check?

danielskatz commented 4 years ago

now it's there.