[PRE REVIEW]: gpu-ISTL - Extending OPM Flow with GPU Linear Solvers

[editorialbot]

Submitting author: !--author-handle-->@multitalentloes<!--end-author-handle-- (Tobias Meyer Andersen) Repository: https://github.com/OPM/opm-simulators Branch with paper.md (empty if default branch): joss_gpuistl_paper Version: 2024.10 Editor: Pending Reviewers: Pending Managing EiC: Daniel S. Katz

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/8c35a29762fc508215904c0e1dd16013"><img src="https://joss.theoj.org/papers/8c35a29762fc508215904c0e1dd16013/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/8c35a29762fc508215904c0e1dd16013/status.svg)](https://joss.theoj.org/papers/8c35a29762fc508215904c0e1dd16013)

Author instructions

Thanks for submitting your paper to JOSS @multitalentloes. Currently, there isn't a JOSS editor assigned to your paper.

@multitalentloes if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). You can search the list of people that have already agreed to review and may be suitable for this submission.

Editor instructions

The JOSS submission bot @editorialbot is here to help you find and assign reviewers and start the main review. To find out what @editorialbot can do for you type:

@editorialbot commands

[editorialbot]

Hello human, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.1137/15M1026419 is OK
- 10.1016/j.parco.2021.102831 is OK
- 10.1016/j.camwa.2020.05.014 is OK
- 10.1016/j.camwa.2020.06.007 is OK
- 10.1016/j.camwa.2020.02.012 is OK
- 10.2118/218015-pa is OK
- 10.3390/mca24030070 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: Computer Architecture, Sixth Edition: A Quantitati...
- No DOI given, and none found for title: An Introduction to Reservoir Simulation Using MATL...
- No DOI given, and none found for title: CUDA, release: 12.2.r12.2
- No DOI given, and none found for title: hipify-perl
- No DOI given, and none found for title: Heterogeneous-computing Interface for Portability
- No DOI given, and none found for title: Bandicoot: C++ Library for GPU Linear Algebra and ...

❌ MISSING DOIs

- 10.1007/978-3-540-75755-9_82 may be a valid DOI for title: The Iterative Solver Template Library

❌ INVALID DOIs

- --- is INVALID

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.90  T=0.70 s (1694.9 files/s, 400720.8 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C/C++ Header                   617          24852          39348          92073
C++                            437          16384          12267          73855
SVG                              5              4             23           4704
CMake                           16            483            285           4298
TeX                             27            617            130           4024
CUDA                            12            375            286           2202
Bourne Shell                    25            261            119           1377
OpenCL                          23            124             97           1011
Python                           9             53            112            344
Markdown                         5            101              0            340
JSON                             3              0              0            124
YAML                             2              4              0             40
MATLAB                           1              8              0             20
Dockerfile                       1              2              3             13
awk                              1              1             24             10
TOML                             1              0              0              7
INI                              1              0              0              3
-------------------------------------------------------------------------------
SUM:                          1186          43269          52694         184445
-------------------------------------------------------------------------------

Commit count by author:

  3007  Arne Morten Kvarving
  2216  Atgeirr Flø Rasmussen
  1444  Andreas Lauser
  1255  Kai Bao
  1206  Tor Harald Sandve
  1024  Markus Blatt
   894  Bård Skaflestad
   796  Joakim Hove
   320  Liu Ming
   226  Robert Kloefkorn
   185  Roland Kaufmann
   184  Kjetil Olsen Lye
   176  Xavier Raynaud
   174  hnil
   170  Tong Dong Qiu
   146  babrodtk
   143  Håkon Hægland
   102  Tobias Meyer Andersen
   100  Vegard Kippe
    89  Robert K
    82  Jørgen Kvalsvik
    74  osae
    73  Paul Egberts
    69  T.D. (Tongdong) Qiu
    67  Alf Birger Rustad
    67  Bernd Flemisch
    63  Stein Krogstad
    52  Lisa Julia Nebel
    45  Kristian Flikka
    36  Antonella Ritorto
    36  goncalvesmachadoc
    34  OPMUSER
    34  Pål Grønås Drange
    33  Benjamin Faigle
    31  Steinar Foss
    30  Philipp Nuske
    30  Razvan Nane
    29  Markus Wolff
    28  Cintia Goncalves Machado
    28  Halvor Møll Nilsen
    28  jakobtorben
    26  Klaus Mosthaf
    26  kel85uk
    25  Melanie Darcis
    24  Jose Eduardo Bueno
    24  Ove Sævareid
    24  Svenn Tveit
    23  David Landa Marban
    23  Rohith Nair
    22  Elyes Ahmed
    20  Torbjørn Skille
    19  André R. Brodtkorb
    19  David Werner
    19  Franz G. Fuchs
    19  Paul
    19  chflo
    17  Christoph Grueninger
    17  tqiu
    16  josh bowden
    12  Josh Bowden
    11  Alf B. Rustad
    10  Halvor M. Nilsen
    10  Jostein Alvestad
    10  Jostein R. Natvig
    10  andrthu
     9  Kjell W. Kongsvik
     8  Katherina Baber
     7  jilocode
     6  Andrea T. Lonn
     6  Michael Sargado
     6  Trine Mykkeltvedt
     6  WesselDeZeeuw
     5  Edvin Brudevoll
     5  Franz Georg Fuchs
     4  Anders Matheson
     4  Erik Hide Sæternes
     4  Fredrik Gundersen
     4  Júlio Hoffimann
     4  Nicolas Schwenck
     4  Peter Verveer
     4  Trine S. Mykkeltvedt
     4  dependabot[bot]
     3  Alexander Kissinger
     3  Jens Olav Nygaard
     3  Justin Chang
     3  Jørgen Kvalsvk
     3  Karin Erbertseder
     3  rube051
     2  Andreas Thune
     2  Erik Hide Saeternes
     2  Giacomo Marchiori
     2  Joshua Bowden
     2  Kristian Hole-Drabløs
     2  Lars Petter Øren Hauge
     2  Tom Hogervorst
     2  Tor Harald Sanve
     2  Trine S Mykkeltvedt
     2  daavid00
     2  locture
     1  Andrea Tronstad Lønn (ST MSU DYN)
     1  Baurzhan Muftakhidinov
     1  Bjørn Spjelkavik
     1  Eduardo Bueno
     1  Georg Zitzlsberger
     1  Holger Class
     1  Jakob Torben
     1  Jean-Paul Balabanian
     1  Jochen Fritz
     1  Matthew Goodfield
     1  Timo Koch
     1  Viktor Szukitsch
     1  Vishal Jambhekar
     1  WesselZ
     1  Zeeuw
     1  kristinf
     1  nairr
     1  paean
     1  steink

[editorialbot]

Paper file info:

📄 Wordcount for paper.md is 1521

✅ The paper includes a Statement of need section

[editorialbot]

License info:

🟡 License found: GNU General Public License v3.0 (Check here for OSI approval)

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[editorialbot]

Five most similar historical JOSS papers:

Ginkgo: A high performance numerical linear algebra library Submitting author: @pratikvn Handling editor: @mjsottile (Retired) Reviewers: @adam-m-jcbs, @zeroset Similarity score: 0.6914

SPbLA: The Library of GPGPU-powered Sparse Boolean Linear Algebra Operations Submitting author: @EgorOrachyov Handling editor: @Nikoleta-v3 (Active) Reviewers: @mlxd, @bencardoen Similarity score: 0.6856

libCEED: Fast algebra for high-order element-based discretizations Submitting author: @jedbrown Handling editor: @diehlpk (Active) Reviewers: @thelfer, @FreddieWitherden, Similarity score: 0.6804

iharm3D: Vectorized General Relativistic Magnetohydrodynamics Submitting author: @bprather Handling editor: @eloisabentivegna (Retired) Reviewers: @bgiacoma, @cpalenzuela Similarity score: 0.6789

The 2DECOMP&FFT library: an update with new CPU/GPU capabilities Submitting author: @rfj82982 Handling editor: @diehlpk (Active) Reviewers: @CeciliaCoelho, @p-costa Similarity score: 0.6763

⚠️ Note to editors: If these papers look like they might be a good match, click through to the review issue for that paper and invite one or more of the authors before considering asking the reviewers of these papers to review again for JOSS.

[danielskatz]

@multitalentloes - Our apologies for the delay in getting this started. Something went wrong internally, but it's now fixed.

[danielskatz]

I note that this is a very large repo. It appears that you are submitted a part of this for review, as an element of the full repo, but this is not completely clear.

Can you identify the branch/files/etc that is the submitted contribution?

Additionally, the paper is about 50% longer than recommended for a JOSS submission. I wonder if there's part of this that could be place in the documentation or repo, then pointed to there, rather than being in the paper? The performance case study perhaps?

[multitalentloes]

Hello Daniel, thank you very much for initiating the pre-review.

The contributions described in this article predominantly reside in the gpuistl folder. Redoing the work of the bot with cloc on that folder yields something like this:

$ cloc opm-simulators/opm/simulators/linalg/gpuistl/
      59 text files.
      59 unique files.                              
       0 files ignored.

github.com/AlDanial/cloc v 1.90  T=0.02 s (2756.2 files/s, 507055.6 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C/C++ Header                    42            804           2616           3193
C++                             12            316            270           2136
CUDA                             5            154            133           1232
-------------------------------------------------------------------------------
SUM:                            59           1274           3019           6561
-------------------------------------------------------------------------------

There are other minor parts of the repository which have been added to support the gpu-ISTL framework, such as adjustments in:

• GraphColoring.hpp
• FlexibleSolver_impl.hpp
• PreconditionerFactory_impl.hpp
• CMakeLists.txt
• CMakeLists_files.txt

Regarding the length of the article; I think the section demonstrating that the current framework can yield significant speedup over conventional processors when running OPM Flow makes an important point, so I would like to keep it in some shape or form. We can maybe move Appendix A, and maybe into the repository somewhere, or just remove it? Additionally we can replace the section about the build system supporting both AMD and Nvidia with a few sentences in another section. These two changes would probably remove a few hundred words. Would that be properly in line with the typical JOSS format?

Edit: I forgot to mention that we also have a folder with tests specifically for gpu-ISTL functionality located in tests/gpuistl, all files in that folder (a few thousand lines) are also a part of this submission.

[danielskatz]

Thanks. And your suggestions for reducing the paper's word count seem reasonable.

Please feel free to make changes, then use the command @editorialbot generate pdf to make a new PDF.

editorialbot commands need to be the first entry in a new comment.

You can also use editorialbot check repository to check a bunch of things, including the word count for the paper.

[multitalentloes]

@editorialbot check repository

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.90  T=0.71 s (1664.8 files/s, 393679.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C/C++ Header                   617          24852          39348          92073
C++                            437          16384          12267          73855
SVG                              5              4             23           4704
CMake                           16            483            285           4298
TeX                             27            617            130           4024
CUDA                            12            375            286           2202
Bourne Shell                    25            261            119           1377
OpenCL                          23            124             97           1011
XML                              1              0              0            520
Python                           9             53            112            344
Markdown                         6            100              0            339
JSON                             3              0              0            124
YAML                             2              4              0             40
MATLAB                           1              8              0             20
Dockerfile                       1              2              3             13
awk                              1              1             24             10
TOML                             1              0              0              7
INI                              1              0              0              3
-------------------------------------------------------------------------------
SUM:                          1188          43268          52694         184964
-------------------------------------------------------------------------------

Commit count by author:

  3007  Arne Morten Kvarving
  2216  Atgeirr Flø Rasmussen
  1444  Andreas Lauser
  1255  Kai Bao
  1206  Tor Harald Sandve
  1024  Markus Blatt
   894  Bård Skaflestad
   796  Joakim Hove
   320  Liu Ming
   226  Robert Kloefkorn
   185  Roland Kaufmann
   184  Kjetil Olsen Lye
   176  Xavier Raynaud
   174  hnil
   170  Tong Dong Qiu
   146  babrodtk
   143  Håkon Hægland
   103  Tobias Meyer Andersen
   100  Vegard Kippe
    89  Robert K
    82  Jørgen Kvalsvik
    74  osae
    73  Paul Egberts
    69  T.D. (Tongdong) Qiu
    67  Alf Birger Rustad
    67  Bernd Flemisch
    63  Stein Krogstad
    52  Lisa Julia Nebel
    45  Kristian Flikka
    36  Antonella Ritorto
    36  goncalvesmachadoc
    34  OPMUSER
    34  Pål Grønås Drange
    33  Benjamin Faigle
    31  Steinar Foss
    30  Philipp Nuske
    30  Razvan Nane
    29  Markus Wolff
    28  Cintia Goncalves Machado
    28  Halvor Møll Nilsen
    28  jakobtorben
    26  Klaus Mosthaf
    26  kel85uk
    25  Melanie Darcis
    24  Jose Eduardo Bueno
    24  Ove Sævareid
    24  Svenn Tveit
    23  David Landa Marban
    23  Rohith Nair
    22  Elyes Ahmed
    20  Torbjørn Skille
    19  André R. Brodtkorb
    19  David Werner
    19  Franz G. Fuchs
    19  Paul
    19  chflo
    17  Christoph Grueninger
    17  tqiu
    16  josh bowden
    12  Josh Bowden
    11  Alf B. Rustad
    10  Halvor M. Nilsen
    10  Jostein Alvestad
    10  Jostein R. Natvig
    10  andrthu
     9  Kjell W. Kongsvik
     8  Katherina Baber
     7  jilocode
     6  Andrea T. Lonn
     6  Michael Sargado
     6  Trine Mykkeltvedt
     6  WesselDeZeeuw
     5  Edvin Brudevoll
     5  Franz Georg Fuchs
     4  Anders Matheson
     4  Erik Hide Sæternes
     4  Fredrik Gundersen
     4  Júlio Hoffimann
     4  Nicolas Schwenck
     4  Peter Verveer
     4  Trine S. Mykkeltvedt
     4  dependabot[bot]
     3  Alexander Kissinger
     3  Jens Olav Nygaard
     3  Justin Chang
     3  Jørgen Kvalsvk
     3  Karin Erbertseder
     3  rube051
     2  Andreas Thune
     2  Erik Hide Saeternes
     2  Giacomo Marchiori
     2  Joshua Bowden
     2  Kristian Hole-Drabløs
     2  Lars Petter Øren Hauge
     2  Tom Hogervorst
     2  Tor Harald Sanve
     2  Trine S Mykkeltvedt
     2  daavid00
     2  locture
     1  Andrea Tronstad Lønn (ST MSU DYN)
     1  Baurzhan Muftakhidinov
     1  Bjørn Spjelkavik
     1  Eduardo Bueno
     1  Georg Zitzlsberger
     1  Holger Class
     1  Jakob Torben
     1  Jean-Paul Balabanian
     1  Jochen Fritz
     1  Matthew Goodfield
     1  Timo Koch
     1  Viktor Szukitsch
     1  Vishal Jambhekar
     1  WesselZ
     1  Zeeuw
     1  kristinf
     1  nairr
     1  paean
     1  steink

[editorialbot]

Paper file info:

📄 Wordcount for paper.md is 1228

✅ The paper includes a Statement of need section

[editorialbot]

License info:

🟡 License found: GNU General Public License v3.0 (Check here for OSI approval)

[multitalentloes]

Run configurations for the case study have been moved to a separate .md file living next to the paper in the branch for the paper. The paragraph on the build system has also been updated, is the current length acceptable?

[danielskatz]

Yes, thanks.

But we're currently managing a large backlog of submissions and the editors most appropriate for your area is already rather busy. For now, we will need to waitlist this paper and process it as the queue reduces. Thanks for your patience!