[REVIEW]: gLBM: A GPU enabled Lattice Boltzmann Method Library

whedon commented 4 years ago

Submitting author: @rclipp (Rachel Clipp) Repository: https://gitlab.kitware.com/LBM/lattice-boltzmann-solver Version: JOSS Editor: @eloisabentivegna Reviewer: @ianhinder, @gouarin Archive: 10.5281/zenodo.6076998

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/ef96137ff28121d1d3bb0a36187bd9f9"><img src="https://joss.theoj.org/papers/ef96137ff28121d1d3bb0a36187bd9f9/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/ef96137ff28121d1d3bb0a36187bd9f9/status.svg)](https://joss.theoj.org/papers/ef96137ff28121d1d3bb0a36187bd9f9)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@ianhinder & @gouarin, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

Make sure you're logged in to your GitHub account
Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @eloisabentivegna know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @ianhinder

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@rclipp) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @gouarin

Conflict of interest

[x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

[x] Repository: Is the source code for this software available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
[x] Contribution and authorship: Has the submitting author (@rclipp) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
[x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

[x] Installation: Does installation proceed as outlined in the documentation?
[x] Functionality: Have the functional claims of the software been confirmed?
[x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
[x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
[x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
[x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

[x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
[x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
[x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
[x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
[x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

whedon commented 4 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @ianhinder, @gouarin it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

whedon commented 4 years ago

Reference check summary:

OK DOIs

- None

MISSING DOIs

- https://doi.org/10.1103/physreve.77.056703 may be missing for title: Straight velocity boundaries in the lattice Boltzmann method
- https://doi.org/10.1103/physreve.81.016311 may be missing for title: Three ways to lattice Boltzmann: a unified time-marching picture
- https://doi.org/10.1109/embc.2018.8513672 may be missing for title: An Interactive, Patient-Specific Virtual Surgical Planning System for Upper Airway Obstruction Treatments
- https://doi.org/10.3389/fninf.2014.00013 may be missing for title: ITK: enabling reproducible research and open science

INVALID DOIs

- None

whedon commented 4 years ago

:point_right: Check article proof :page_facing_up: :point_left:

eloisabentivegna commented 3 years ago

@ianhinder @gouarin, could you provide an update on the status of your reviews?

gouarin commented 3 years ago

@eloisabentivegna could you assign me to this review ?

eloisabentivegna commented 3 years ago

@eloisabentivegna could you assign me to this review ?

@gouarin, you are assigned to this review issue. Are you having trouble editing/accessing something?

gouarin commented 3 years ago

@eloisabentivegna It is the first time I make a JOSS review. So, I don't know exactly how it should work. But I can't check the boxes. I try to click on "accept the invite" but the link is expired. I think this is my issue.

Could you send me a new invite?

eloisabentivegna commented 3 years ago

@whedon re-invite @gouarin as reviewer.

Certainly! Let me know if things work now.

whedon commented 3 years ago

OK, the reviewer has been re-invited.

@gouarin please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

gouarin commented 3 years ago

Now it works. Thanks! I will make my review next week.

eloisabentivegna commented 3 years ago

Thanks, @gouarin.

ianhinder commented 3 years ago

@ianhinder @gouarin, could you provide an update on the status of your reviews?

@eloisabentivegna, sorry there's not been much progress apart from the initial look over the project. I'll devote some time to it, and as with @gouarin, will aim to complete the review next week.

eloisabentivegna commented 3 years ago

Thanks @ianhinder. I look forward to your reviews.

gouarin commented 3 years ago

Hello @rclipp, @eloisabentivegna,

here is my review of the paper:

There is a typo in the first paragraph: properites -> properties
Links to the references don't work for me. Is it the same for you?
Could you explain more precisely why your software is unique compared to others (Palabos, openLB, ...)? Because, for me, rather than the GPU implementation, they can offer what you offer.
Could you write explicitly what kind of equations you can solve? Is it just Navier-Stokes?
What kind of LBM schemes are implemented in your software? And what kind of LBM boundary conditions?
Is it 2D and 3D or just 3D?
It might be interesting to give an example of a configuration file to better understand how to run a simulation.
Do you have studied the performance of your software in MLUPS or other metrics? If so, it could be interesting to have a few words about it.
In the automated verification and validation section, I don't understand if the validation is done by plotting the result and see if it is correct or if it is an automated process. Do you have developed your tools for this part? or do you use available test software?
In the future directions, could you explain more precisely what the next functionalities will be?
It might be interesting to have an overview of the software structure.

ianhinder commented 3 years ago

I've been working on installing and running the code. It was quite difficult; I have a number of issues to report, and will make issues in gitlab and post back here with a list.

ianhinder commented 3 years ago

I have opened 16 issues relating to installation and installation documentation. See https://gitlab.kitware.com/LBM/lattice-boltzmann-solver/-/issues?scope=all&utf8=✓&state=opened&author_username=ianhinder. Not all of these are vital.

I will continue the review next week.

eloisabentivegna commented 3 years ago

@rclipp, have you had a chance to check out the reviewers' comments?

eloisabentivegna commented 3 years ago

@rclipp, have you had a chance to check out the reviewers' comments?

Followed up via email.

eloisabentivegna commented 3 years ago

@gouarin, @ianhinder, have you finished posting your comments on this submission? If so, could you explicitly state your recommendation?

ianhinder commented 3 years ago

Contribution and authorship:

The contributing author has made a substantial contribution, but I cannot evaluate the appropriateness of the authorship of the authors Sorin Mitran, PhD and Andinet Enquobahrie, PhD. Please could this be clarified?

ianhinder commented 3 years ago

Installation:

I have posted 16 issues to the gitlab issue tracker of the code; it was quite complicated to install and understand the instructions. I think it would be best if this was substantially improved before acceptance.
I understand that writing instructions for installing scientific software like this is very hard; you have my sympathies! People will want to install it on many different systems, and dependencies are always a nightmare.
I was eventually able to get the code to install and run, but I think this would be very hard for a lot of people

ianhinder commented 3 years ago

Functionality:

The software claims to implement the Lattice Boltzmann Method for solving hydrodynamical systems.
When I ran the provided verification tests, they mostly reported "Baseline data not found" (see https://gitlab.kitware.com/LBM/lattice-boltzmann-solver/-/issues/15) and were not able to report a pass or fail. So I cannot tell whether it passes its own tests.
Ideally, I would like to see an example for a standard case, e.g. a Poiseuille flow, that I can run, see the output data, and compare to the analytical result. This comparison is shown in the paper, but there are no instructions for how I can replicate this myself.

ianhinder commented 3 years ago

Performance:

The main selling point of this code is probably that it runs on GPUs, implying that performance is an explicit goal. This is implied by the statement

The combination of speed (GPU implementation), flexibility, permissive licensing, and integrated verification and validation make this library unique when compared to other available libraries of the LBM [@palabos][@openLB][@sailfish].

found in the paper.
However, there are no performance measurements anywhere that I can find. As @gouarin requested, I would like to see measurements comparing the traditional MLUPS (million lattice updates per second) achieved with this code vs some other codes. It is relatively straightforward to write a correct GPU LBM code. The effort comes in taking advantage of the complicated characteristics of the GPU to achieve the high speeds promised.

ianhinder commented 3 years ago

Installation instructions:

See my comment above about my experiences with installation. I believe the installation instructions need to be significantly improved. Dependencies are listed satisfactorily.

ianhinder commented 3 years ago

Example usage:

I cannot find in the README (or elsewhere) examples of how to use the software to solve real-world (or any) problems, beyond a single commented configuration file referred to in passing at the end of the README.
It is not even clear what the main code is, or how to run it. There is a "driver" called "LBMITKDriver", but I cannot tell if this is something needed to use this "ITK" library, or whether this is in fact the main gLBM code that one would run to solve a problem.

ianhinder commented 3 years ago

Functionality documentation:

Beyond a single commented configuration file, there is no real documentation for how the code works, or how to use it.

ianhinder commented 3 years ago

Automated tests:

There are impressive-looking automated verification tests
However, these do not work for me, as they cannot find "baseline data".

ianhinder commented 3 years ago

Community guidelines:

There is a well-written contribution document
There is no explicit guide for how to report issues or ask for help; one would assume that creating an issue in the gitlab repo is the way to go, but this is not stated.

ianhinder commented 3 years ago

State of the field:

There is no in-depth comparison of how the software compares to other packages in the field
A quick internet search for "open source GPU LBM code" yields several hits, for example this one. The paper points to three existing LBM packages as examples, but there are more available. It would be good to see a more in-depth review, e.g. a couple of sentences for each package, explaining why this new code has an advantage over each existing package. The list of competitor packages should also be longer and include existing open source GPU LBM codes.

ianhinder commented 3 years ago

Quality of writing:

The paper is quite well written; it would benefit from a spell-checker here and there.

ianhinder commented 3 years ago

General comments:

The review criteria state that the paper should only contain the sections listed in bullet points at the top of the page. The paper contains sections not listed (Numerical Benchmarking, Automated Verification and Validation, Future Directions). I find those sections to be useful and informative, and would leave them in, but I mention it here due to the wording in the review criteria.

ianhinder commented 3 years ago

That's all! I've put each separate issue into a separate comment to make it easier to reply.

Current grade: (3) Requires major revisions

eloisabentivegna commented 3 years ago

@rclipp, did you have a chance to look at the reviewers' recommendations? Could you post an update?

rclipp commented 3 years ago

@eloisabentivegna, We have reviewd the recommendations and we are making changes now to both the paper and repository. We hope to have it finished early next week or sooner. Some of the library comparisons will be difficult due to the 1000 word limit, but we will address as many issues as we can. Thanks!

eloisabentivegna commented 3 years ago

@eloisabentivegna, We have reviewd the recommendations and we are making changes now to both the paper and repository. We hope to have it finished early next week or sooner.

Dear @rclipp, could you post an update?

aaron-bray commented 3 years ago

I have addressed all the issues provided and we should get an update to the paper very soon.

I hope the read me is better structured and has a better level of description for building and running our code

Thanks!

eloisabentivegna commented 3 years ago

@aaron-bray, @rclipp, are you happy for the reviewers to take a look at your new version now?

aaron-bray commented 3 years ago

From a code perspective, yes I believe that it should easier to build and run now.

On Sat, Jan 30, 2021, 2:42 PM Eloisa Bentivegna notifications@github.com wrote:

@aaron-bray https://github.com/aaron-bray, @rclipp https://github.com/rclipp, are you happy for the reviewers to take a look at your new version now?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openjournals/joss-reviews/issues/2555#issuecomment-770269913, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGTUWHZSFG73ZYQMKRMTHQTS4ROL5ANCNFSM4PW6FQBQ .

eloisabentivegna commented 3 years ago

@ianhinder, @gouarin, are you satisfied with the current version?

ianhinder commented 3 years ago

I'll take another look at it this week. Thanks, @aaron-bray, for addressing the issues!

gouarin commented 3 years ago

Thanks @aaron-bray for this update. I'll try to give my feedback next week.

eloisabentivegna commented 3 years ago

@ianhinder, @gouarin, do you have updates to report?

eloisabentivegna commented 3 years ago

@ianhinder, @gouarin, will you have the chance to complete your reviews shortly? You only seem to have a few unchecked items each, so hopefully this won't require too much longer.

gouarin commented 3 years ago

@eloisabentivegna it seems that the paper has not been modified since my first remarks and it is therefore impossible for me to continue the review procedure as long as my remarks have not been taken into account. When it will be done, I will check the last items.

eloisabentivegna commented 3 years ago

it seems that the paper has not been modified since my first remarks

Thanks, @gouarin. There are some changes to the package itself, if not the paper, that address your points. Could you see if any of those allow you to make progress on the checklist?

@rclipp, @aaron-bray, could you incorporate all suggested changes to your paper? Some are just typo fixes and other easy aspects to correct. It should not take too much work.

eloisabentivegna commented 3 years ago

@whedon generate pdf from branch joss

whedon commented 3 years ago

Attempting PDF compilation from custom branch joss. Reticulating splines etc...

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

rclipp commented 3 years ago

@gouarin @eloisabentivegna @ianhinder Sorry for the paper updates. I am working on them now and should be able to address them today. Thanks.

rclipp commented 3 years ago

@gouarin @eloisabentivegna @ianhinder We have addressed the paper comments. A quick summary for each comment is below.

There is a typo in the first paragraph: properites -> properties - CORRECTED

Links to the references don't work for me. Is it the same for you? - I am not sure. The references are at the end of the paper and look correct, but if they are supposed to be linked within the paper, then no they are not for us either.

Could you explain more precisely why your software is unique compared to others (Palabos, openLB, ...)? Because, for me, rather than the GPU implementation, they can offer what you offer. - The main advantages are the GPU implementation and the Apache 2.0 license, which enables proprietary development of applications. We have updated this is the paper.

Could you write explicitly what kind of equations you can solve? Is it just Navier-Stokes? - We do not solve the Navier-Stokes, I have added the equations.

What kind of LBM schemes are implemented in your software? And what kind of LBM boundary conditions? This has been added in the paper.

Is it 2D and 3D or just 3D? 3D, and this is added to the paper.

It might be interesting to give an example of a configuration file to better understand how to run a simulation. - We have added significantly more detail about this is the readme and instructions. But we are over the paper limit as is, so I did not add this to the paper.

Do you have studied the performance of your software in MLUPS or other metrics? If so, it could be interesting to have a few words about it. - We have not yet done this.

In the automated verification and validation section, I don't understand if the validation is done by plotting the result and see if it is correct or if it is an automated process. Do you have developed your tools for this part? or do you use available test software? We have written our own software to automatically plot and calculate the error. This has been clarified in the paper.

In the future directions, could you explain more precisely what the next functionalities will be? I have clarified this to some degree. as we are going to focus on improve the accuracy at high mach numbers.

It might be interesting to have an overview of the software structure. - Due to space constraints, I did not go into this.

openjournals / joss-reviews