[REVIEW]: Pigeons.jl: Distributed sampling from intractable distributions

whedon commented 7 months ago

Submitting author: !--author-handle-->@nikola-sur@pitsianis<!--end-editor-- Reviewers: @nsailor, @georgebisbas Archive: Pending

Status

Status badge code:

HTML: <a href="https://proceedings.juliacon.org/papers/9c511e84865bf6eae357436c5a38be7d"><img src="https://proceedings.juliacon.org/papers/9c511e84865bf6eae357436c5a38be7d/status.svg"></a>
Markdown: [![status](https://proceedings.juliacon.org/papers/9c511e84865bf6eae357436c5a38be7d/status.svg)](https://proceedings.juliacon.org/papers/9c511e84865bf6eae357436c5a38be7d)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@nsailor & @georgebisbas, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

Make sure you're logged in to your GitHub account
Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @pitsianis know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @nsailor

Conflict of interest

[x] As the reviewer I confirm that I have read the JuliaCon conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

[x] I confirm that I read and will adhere to the JuliaCon code of conduct.

General checks

[ ] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Authorship: Has the submitting author (@nikola-sur) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[ ] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
[ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Paper format

[x] Authors: Does the paper.tex file include a list of authors with their affiliations?
[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[ ] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
[x] Page limit: Is the page limit for full papers respected by the submitted document?

Content

[x] Context: is the scientific context motivating the work correctly presented?
[ ] Methodology: is the approach taken in the work justified, presented with enough details and reference to reproduce it?
[ ] Results: are the results presented and compared to approaches with similar goals?

Review checklist for @georgebisbas

Conflict of interest
[x] As the reviewer I confirm that I have read the JuliaCon conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

[x] I confirm that I read and will adhere to the JuliaCon code of conduct.

General checks

[ ] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Authorship: Has the submitting author (@nikola-sur) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[ ] Functionality: Have the functional claims of the software been confirmed?
[ ] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
[ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Paper format

[x] Authors: Does the paper.tex file include a list of authors with their affiliations?
[ ] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[ ] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
[ ] Page limit: Is the page limit for full papers respected by the submitted document?

Content

[x] Context: is the scientific context motivating the work correctly presented?
[ ] Methodology: is the approach taken in the work justified, presented with enough details and reference to reproduce it?
[ ] Results: are the results presented and compared to approaches with similar goals?

whedon commented 7 months ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @nsailor, @georgebisbas it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/JuliaCon/proceedings-review) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

Set yourself as 'Not watching' https://github.com/JuliaCon/proceedings-review:

watching

You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

whedon commented 7 months ago

Failed to discover a Statement of need section in paper

whedon commented 7 months ago

Wordcount for paper.tex is 308

whedon commented 7 months ago

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.10 s (503.0 files/s, 350305.9 lines/s)
---------------------------------------------------------------------------------------
Language                             files          blank        comment           code
---------------------------------------------------------------------------------------
TeX                                     37           1011           1624          30073
Windows Module Definition                1              0              0           1585
Julia                                   10             59             50            330
Ruby                                     1              8              4             45
YAML                                     1              0              0             30
---------------------------------------------------------------------------------------
SUM:                                    50           1078           1678          32063
---------------------------------------------------------------------------------------

Statistical information for the repository '301fc6098cae505466f5f2a7' was
gathered on 2023/11/18.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Nikola Surjanovic                1            57              0          100.00

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Nikola Surjanovic            57          100.0          0.0                7.02

whedon commented 7 months ago

Failed to discover a valid open source license.

whedon commented 7 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 7 months ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1137/141000671 is OK
- 10.1007/s11222-012-9328-6 is OK
- 10.2307/2348941 is OK
- 10.1093/sysbio/syt022 is OK
- 10.1016/j.cma.2021.114264 is OK
- 10.1615/int.j.uncertaintyquantification.2022038478 is OK
- 10.1038/s41467-022-31830-2 is OK
- 10.1145/3314221.3314642 is OK
- 10.2307/2684568 is OK
- 10.21105/joss.02741 is OK
- 10.1186/s12918-017-0433-1 is OK
- 10.1103/physrevlett.57.2607 is OK
- 10.1111/rssb.12464 is OK
- 10.1093/bioinformatics/17.8.754 is OK
- 10.1093/sysbio/sys029 is OK
- 10.1109/wisp.2007.4447579 is OK
- 10.1007/s11222-008-9110-y is OK
- 10.3390/galaxies11010006 is OK
- 10.3847/1538-4357/ac09ee is OK
- 10.21105/joss.04457 is OK
- 10.1198/106186006x100579 is OK
- 10.1198/jcgs.2010.10039 is OK
- 10.1198/jcgs.2011.10167 is OK
- 10.1073/pnas.1408184111 is OK
- 10.4324/9781003289173-2 is OK
- 10.1111/rssb.12336 is OK
- 10.1145/62959.62969 is OK
- 10.1145/2714064.2660195 is OK
- 10.1080/01621459.1990.10474918 is OK
- 10.1093/sysbio/syq085 is OK
- 10.21105/joss.01816 is OK
- 10.1145/3551624.3555300 is OK

MISSING DOIs

- 10.1111/rssb.12464 may be a valid DOI for title: Non-reversible parallel tempering: a scalable highly parallel MCMC scheme

INVALID DOIs

- None

whedon commented 6 months ago

:wave: @georgebisbas, please update us on how your review is going (this is an automated reminder).

whedon commented 6 months ago

:wave: @nsailor, please update us on how your review is going (this is an automated reminder).

nsailor commented 6 months ago

This work is an implementation of the Parallel Tempering method in Julia, an improvement over MCMC for obtaining samples form otherwise difficult to sample probability distributions. The package supports parallelization using both Julia's built-in multi-threading features, as well as MPI.jl, a Julia wrapper for MPI.

I greatly appreciate the attention given to what the authors term "strong parallelism invariance" (SPI), roughly meaning that the result should not depend on the degree of parallelism (especially considering the complexities of finite-precision floating-point arithmetic). This package also builds upon a Julia translation of a Java library called Splittable Random previously published by the authors, allowing deterministic random number generation across multiple threads.

Overall, I think this work is a great contribution to the Julia ecosystem, notwithstanding the following minor points:

Although definitely not a requirement, I think it would be very interesting to see the scaling characteristics of this implementation, especially with respect to the other algorithm parameters, for instance, the number of chains. Additionally, the paper mentions Pigeons being able to run on "thousands of MPI-communicating machines". It may be worth clarifying if this is something that has been tested or a possibility given the package's design. Having said that, I was glad to find this issue so it seems this is something on the authors' radar.
Unfortuantely, I was not able to get a speedup with the toy MVN example using multiple threads (see this issue in the project repository). In general, it would be very helpful to have some hints in the package's documentation for tuning the degree of parallelism for a given problem, especially if the speedup from parallelism is not linear.
It would be great to have additional examples, as described in this issue

In writing the above, it is possible that I have misunderstood some part of this work, in which case, please feel free to correct me.

nikola-sur commented 6 months ago

Thank you for your comments! We will work on updating the manuscript, package, and documentation to account for your feedback.

miguelbiron commented 5 months ago

Dear @pitsianis -- we were wondering what the expected timeline for the reviews is? Should we perhaps just respond to @nsailor and not wait for @georgebisbas, or should we wait until we have both sets of comments? Thank you.

pitsianis commented 5 months ago

Please answer any outstanding issues; you don't need to do this sequentially. This exchange will also remind @georgebisbas to wrap up his review.

miguelbiron commented 5 months ago

Thanks for the clarification.

georgebisbas commented 5 months ago

Dear @miguelbiron and @pitsianis, my apologies for the delay here. Feel free to respond to the outstanding issues; I aim to prioritize this review in my task list.

miguelbiron commented 5 months ago

No worries @georgebisbas, thank you for prioritizing

georgebisbas commented 4 months ago

Hi again, apologies for the delay, I have partly written a draft, aiming to complete my review in the next days.

georgebisbas commented 4 months ago

First, I would like to thank the authors for their patience over the last few months.

Pigeons.jl offers a high-level API to leverage shared- and distributed-memory parallelism via Julia's built-in multi-threading features and MPI.jl, a Julia wrapper for MPI. Overall, this is a great work suitable for JuliaCon proceedings. My review below lists a few weaknesses/questions that could help clarify my understanding of the work in a few places or act as constructive feedback to improve this work.

Strengths :

Automated deployment to MPI DMP without necessary user experience
Strong parallelism invariance is guaranteed.
Applications are discussed in the Introduction.
Smooth transition for the reader, problem formulation, and running examples help in that.
Compatibility with other targets is a plus
Section 2.2.1 helps to describe Figure 2
I very much like how code is accompanied by text describing the symbolic math process

Weaknesses/Questions: I understand and like that it "works out of the box." It feels that authors focus more on correctness rather than performance. A few questions I have, are:

What are the memory requirements of a "typical large enough" problem to be tackled?
What are the advantages of DMP versus SMP only?
Why are any strong scaling graphs not included?
After reading the paper, I feel that MPI works correctly, but with no clue on why it is needed and what performance benefits it brings to the table

Minor:

Have any of the additional targets been implemented in the meantime? Could the paper be updated there?

nikola-sur commented 4 months ago

Thank you to both reviewers for their comments! Now that all reviews are in, we will get back to you shortly with updates and responses to the questions raised.

nikola-sur commented 4 weeks ago

We thank the reviewers for their insightful comments! Our responses to each reviewer are presented below.

Reviewer 1 (Jasan Barmparesos @nsailor) It would be very interesting to see the scaling characteristics of this implementation, especially with respect to the other algorithm parameters, for instance, the number of chains.

Thank you for this comment. We have added weak scaling plots in the paper in the new section on weak scaling of PT (Section 3.4). In this context, the problem size is the number of PT chains. As the problem size increases, we also increase the number of processors for the weak scaling plots. For strong scaling plots, where the problem size (number of chains) is fixed, we propose a future addition to the Pigeons software package: “parallel parallel tempering”. Here, more than one copy of PT is run while keeping the total number of chains across all copies fixed. This parallel PT approach has been previously studied in the PT literature (Syed et al., 2022), but is not yet available in the current version of Pigeons.

Additionally, the paper mentions Pigeons being able to run on "thousands of MPI-communicating machines". It may be worth clarifying if this is something that has been tested or a possibility given the package's design.

This has been tested (with up to 1000 MPI-communicating machines) on the computer clusters available at our institution. We have clarified this in the updated version of the paper.

I was not able to get a speedup with the toy MVN example using multiple threads (see this issue in the project repository). In general, it would be very helpful to have some hints in the package's documentation for tuning the degree of parallelism for a given problem, especially if the speedup from parallelism is not linear.

Thank you for raising this point. This issue has been closed as completed in the meantime. The reason for this is that the toy_mvn_target() has a special explorer performing iid sampling. We use this when we need very fast running tests (e.g. in CI) but in terms of performance profile this is atypical. Since the i.i.d. multivariate normal explorer is so quick the overhead of multithreading is not worth it. Typical explorers use algorithms such as slice sampling or gradient informed samplers, and for those algorithms we see improvements once the problem is large enough (and for small problems with fast explorers, absolute time spent will be small anyway, so relative speedup is less important). We have included new plots in the paper (added here for reference) comparing the speedup obtained with Pigeons and multi-threading in realistic scenarios.

It would be great to have additional examples, as described in this issue

We agree that this is a great addition. In fact, we have added a link in the paper to several real-life example use cases of Pigeons with these targets. In the meantime Pigeons has been applied to problems in physics, economics, astrostatistics, psychology, and more. Some examples are made publicly available on InferHub (https://julia-tempering.github.io/InferHub/), our new website dedicated to showcasing examples of applications of Pigeons. We plan to continue to grow this repository of examples over time.

Reviewer 2 (George Bisbas @georgebisbas)

What are the memory requirements of a "typical large enough" problem to be tackled?

We emphasize that the memory requirements of a “difficult” sampling task are problem dependent. However, Pigeons has facilities to support cases requiring large memory. For instance, we allow for the computation of online statistics (https://pigeons.run/stable/output-online/), as well as off-memory sample storage (https://pigeons.run/stable/output-off-memory/). These have been commented on in the updated version of the manuscript.
As a second remark, we note that Pigeons is intended for use with “complex” sampling problems. This does not necessarily mean that the memory requirements are large. For instance, a 100-dimensional real-valued distribution may have a very complicated structure, but only has a memory requirement of storing a floating point vector with 100 entries (800 bytes). Nevertheless, due to the complex nature of the problem, it may be necessary to use parallel tempering and distribute computation on up to several machines. In practice, the memory requirement is usually O(d), where d is the dimension of the target distribution. In the Bayesian setting, we may need to store n data points in order to evaluate the likelihood, in which case the memory requirement is O(d+n). Therefore, the memory requirements of a “difficult” sampling task are very much problem dependent.

What are the advantages of DMP versus SMP only?

Thank you for this question. We note that Pigeons supports both shared memory parallelism (SMP) and distributed memory parallelism (DMP). In certain cases, it might be useful to run parallel tempering with very many chains (e.g., 100) or to run several instances of parallel tempering. In these cases, it is not reasonable to expect all users to have access to a shared memory resource capable of executing 100 processes in parallel. Our implementation in Pigeons ensures that the communication cost in the DMP setup is minimal. In fact, in most cases, the communication cost is negligible compared to other computations performed on each process. We have clarified these points in the paper. Additionally, DMP can be useful when the user's code is not thread-safe. In such cases, concurrency can still be achieved via DMP.

Why are any strong scaling graphs not included?

Thank you for this comment. We have added weak scaling plots in the paper in the new section on weak scaling of PT (Section 3.4). In this context, the problem size is the number of PT chains. As the problem size increases, we also increase the number of processors for the weak scaling plots. For strong scaling plots, where the problem size (number of chains) is fixed, we propose a future addition to the Pigeons software package: “parallel parallel tempering”. Here, more than one copy of PT is run while keeping the total number of chains across all copies fixed. This parallel PT approach has been previously studied in the PT literature (Syed et al., 2022), but is not yet available in the current version of Pigeons. We are planning to implement this feature in the near future (https://github.com/Julia-Tempering/Pigeons.jl/issues/188).

After reading the paper, I feel that MPI works correctly, but with no clue on why it is needed and what performance benefits it brings to the table.

Thank you for raising this point. We have added to the paper comments in a new section (Section 3.5) that emphasize that Pigeons allows for both DMP and SMP. In cases where the sampling problem is particularly complex, it can be advantageous to request many processes, in which case a distributed memory setup (e.g., on a cluster) is the most logical when it cannot be expected from the user to have access to a machine with, e.g., 100-1000 processors. MPI is the go-to interface for DMP settings that require communication between workers. Another notable example of the benefit of MPI is for the multi-GPU use case. For instance, in some situations the likelihood evaluation of a single chain requires its own GPU (e.g., for high-resolution black hole imaging, which relies on running a Fast Fourier Transform on a GPU).

Have any of the additional targets been implemented in the meantime? Could the paper be updated there?

Yes, there are new targets, which are made publicly available on InferHub (https://julia-tempering.github.io/InferHub/), our new website dedicated to showcasing examples of applications of Pigeons. We plan to continue to grow this repository of examples over time. We have added a link in the paper to InferHub. In the meantime, Pigeons has also been applied to problems in physics, economics, astrostatistics, psychology, and more.

References Syed, S., Bouchard-Côté, A., Deligiannidis, G., & Doucet, A. (2022). Non-reversible parallel tempering: a scalable highly parallel MCMC scheme. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(2), 321-350.

editorialbot commented 2 weeks ago

My name is now @editorialbot

JuliaCon / proceedings-review