Phdf5 ioserver merge to upstream

[mnagaso]

Hello @danielpeter !

I finally cought up the current devel branch of upstream. So I want to merge my development for:

• parallel HDF5 support for decompose/meshfem/generate_database/specfem3d
• asynchronous IO

A technical and brief report on the modifications are: https://docs.google.com/document/d/1Wtoh_GCrpm6OuVOqIr1CASRXeGysxup-hxbjx3IVqYU/edit?usp=sharing

The hdf5+asyncronous IO will be only available when the code is configured with e.g.:
./configure FC=h5pfc MPIFC=h5pfc --with-mpi --with-hdf5

There are two additional parameters in Par_file below:

# HDF5 Database I/O flag which is useful for a massively parallelized calculation.
HDF5_ENABLED                    = .true.

# number of IO dedicated procs, available only with HDF5 output option
NIONOD                          = 0

Please feel free to ask me further explanation on the implementation. I can also provide some performance test result (please send a message to my gmail if necessary).

[buildbot-princeton]

The files changed do not require a buildbot test. The maintainers can commit these changes.

[buildbot-princeton]

The files changed do not require a buildbot test. The maintainers can commit these changes.

[buildbot-princeton]

The files changed do not require a buildbot test. The maintainers can commit these changes.

[mnagaso]

As this change is very big, maybe keeping this version as a different branch would be an option🤔

[buildbot-princeton]

You pushed to your local branch while your previous changes are waiting to be committed. We assume that you did not intend for these changes to be part of the currently opened pull-request and ignore those changes. If you want these changes to be committed, please open a new pull-request.

Note to maintainers: when you accept the original pull-request, the message from buildbot will say that it closed the pull-request, as opposed to the usual message that it merged the pull-request. In spite of this, the initial pull-request will be merged after you accept it.

[codecov-commenter]

Codecov Report

Patch coverage: 27.09% and project coverage change: -11.73 :warning:

Comparison is base (90c20df) 45.24% compared to head (f16cf95) 33.51%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

```diff @@ Coverage Diff @@ ## devel #1597 +/- ## =========================================== - Coverage 45.24% 33.51% -11.73% =========================================== Files 250 259 +9 Lines 56776 57212 +436 =========================================== - Hits 25688 19177 -6511 - Misses 31088 38035 +6947 ``` | [Impacted Files](https://codecov.io/gh/SPECFEM/specfem3d/pull/1597?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SPECFEM) | Coverage Δ | | |---|---|---| | [.../decompose\_mesh/write\_mesh\_databases\_hdf5\_stub.F90](https://codecov.io/gh/SPECFEM/specfem3d/pull/1597?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SPECFEM#diff-c3JjL2RlY29tcG9zZV9tZXNoL3dyaXRlX21lc2hfZGF0YWJhc2VzX2hkZjVfc3R1Yi5GOTA=) | `0.00% <0.00%> (ø)` | | | [...erate\_databases/read\_partition\_files\_hdf5\_stub.f90](https://codecov.io/gh/SPECFEM/specfem3d/pull/1597?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SPECFEM#diff-c3JjL2dlbmVyYXRlX2RhdGFiYXNlcy9yZWFkX3BhcnRpdGlvbl9maWxlc19oZGY1X3N0dWIuZjkw) | `0.00% <0.00%> (ø)` | | | [...enerate\_databases/save\_arrays\_solver\_hdf5\_stub.F90](https://codecov.io/gh/SPECFEM/specfem3d/pull/1597?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SPECFEM#diff-c3JjL2dlbmVyYXRlX2RhdGFiYXNlcy9zYXZlX2FycmF5c19zb2x2ZXJfaGRmNV9zdHViLkY5MA==) | `0.00% <0.00%> (ø)` | | | [src/meshfem3D/save\_databases\_hdf5\_stub.F90](https://codecov.io/gh/SPECFEM/specfem3d/pull/1597?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SPECFEM#diff-c3JjL21lc2hmZW0zRC9zYXZlX2RhdGFiYXNlc19oZGY1X3N0dWIuRjkw) | `0.00% <0.00%> (ø)` | | | [src/shared/phdf5\_utils\_stub.f90](https://codecov.io/gh/SPECFEM/specfem3d/pull/1597?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SPECFEM#diff-c3JjL3NoYXJlZC9waGRmNV91dGlsc19zdHViLmY5MA==) | `0.00% <0.00%> (ø)` | | | [src/shared/serial.f90](https://codecov.io/gh/SPECFEM/specfem3d/pull/1597?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SPECFEM#diff-c3JjL3NoYXJlZC9zZXJpYWwuZjkw) | `45.38% <0.00%> (-1.13%)` | :arrow_down: | | [src/specfem3D/detect\_mesh\_surfaces.f90](https://codecov.io/gh/SPECFEM/specfem3d/pull/1597?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SPECFEM#diff-c3JjL3NwZWNmZW0zRC9kZXRlY3RfbWVzaF9zdXJmYWNlcy5mOTA=) | `14.28% <0.00%> (-83.76%)` | :arrow_down: | | [src/specfem3D/finalize\_simulation.f90](https://codecov.io/gh/SPECFEM/specfem3d/pull/1597?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SPECFEM#diff-c3JjL3NwZWNmZW0zRC9maW5hbGl6ZV9zaW11bGF0aW9uLmY5MA==) | `92.68% <ø> (-4.88%)` | :arrow_down: | | [src/specfem3D/io\_server\_stub.f90](https://codecov.io/gh/SPECFEM/specfem3d/pull/1597?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SPECFEM#diff-c3JjL3NwZWNmZW0zRC9pb19zZXJ2ZXJfc3R1Yi5mOTA=) | `0.00% <0.00%> (ø)` | | | [src/specfem3D/prepare\_attenuation.f90](https://codecov.io/gh/SPECFEM/specfem3d/pull/1597?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SPECFEM#diff-c3JjL3NwZWNmZW0zRC9wcmVwYXJlX2F0dGVudWF0aW9uLmY5MA==) | `3.70% <0.00%> (-75.74%)` | :arrow_down: | | ... and [14 more](https://codecov.io/gh/SPECFEM/specfem3d/pull/1597?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SPECFEM) | | ... and [94 files with indirect coverage changes](https://codecov.io/gh/SPECFEM/specfem3d/pull/1597/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SPECFEM) Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SPECFEM). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=SPECFEM)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[danielpeter]

hi Masaru-san, great work!

there are a bunch of test problems with the current PR, so let me close this one for now. I'll put some comments in the code section just as info where I think we might change things a bit.

I would suggest that I'll take your fork version and include your modifications one by one. as far as I see, there are two main additions:

• HDF5 file format for database files
• I/O server with splitting MPI processes into compute and io tasks

let me start by including the first point, making sure that I don't break the examples and tests. then, I'll add the IO server modification in some follow-up PRs.

many thanks :)

[mnagaso]

Hi Daniel,

Thank you very much for checking my pull request !

Yes it seems that a auto-compilation test was not ok. I might write the rules.mk wrongly.

Also, I am sorry for putting two features in one PR, which I should have divided by myself. But even only taking HDF5 I/O (the first point) into the code would be much simplify the file handling and post-processings. (If possible, I would like to add this to SPECFEM2D as well🤔)

Please ask me anything and anytime about the parts which I wrote if you will have any questions🙇 I am very happy to be able to contribute on SPECFEM.