Driver-portable walker logging

[jtkrogel]

Proposed changes

This PR introduces a simplified version of the legacy TraceManager that focuses solely on writing per-walker data to HDF5 throughout the run (no streaming/listener functionality). The new implementation, WalkerLogManager, is compatible with and implemented in both the legacy and batched drivers.

This implementation extends the original to include the ability to write full data for the highest, lowest, and median energy walkers on each rank to disk at every step. The intention of writing this abbreviated data is to enable inexpensive access to data from problematic walkers (e.g. population explosions) in production runs.

Capabilities

• Writing per-walker scalar data (local energy, etc)
• (optional) writing per-walker particle data (positions, spins, gradient, laplacian)
• (optional) writing full data for min/max/median energy walkers

Below is an example generated from a short DMC run of diamond on 6 MPI ranks showing the energies of all walkers (blue), highest energy walkers (green), lowest energy walkers (red), median energy walkers (black), vs the mean energy from dmc.dat (magenta line):

Design

• Main classes: WalkerLogManager, WalkerLogCollector, WalkerLogBuffer
• WalkerLogManager
  • Driver-level resource
  • Responsible for HDF file I/O
  • Finds min/max/median energy walker data
  • Writes walker data buffers from each crowd to HDF at the end of each block
• WalkerLogCollector
  • Crowd-level resource
  • Collects per-walker data during MC steps and stores it in local data buffers
• WalkerLogBuffer
  • Acts as a dynamic buffer for walker data collected during a block
  • Resource allocation is first-touch

Tests

• Legacy and batched driver deterministic tests
• 1-mpi/1-thread and 4-mpi/4-thread versions
• VMC and DMC
• The tests pass if the walker logging data can be used to reconstruct the information outputted to scalar.dat (VMC) or dmc.dat (DMC)

What type(s) of changes does this code introduce?

• New feature

Does this introduce a breaking change?

• No

What systems has this change been tested on?

Laptop

Checklist

• Yes. This PR adds tests to cover any new code, or to catch a bug that is being fixed

Path out of WIP

• [x] Manual entry
• [x] In-line documentation
• [x] Batched driver naming (Peter)
• [x] Input section ownership at driver-level (Peter)
• [x] WalkerLogCollector::collect() folded into Crowd (Peter)
• [x] Use of const in WalkerLogCollector::collect (Ye)
• [x] Naming conventions/rideTheCamel (Ye)
• [x] WalkerLogBuffer and WalkerLogCollector in separate h/cpp files (Ye)
• [x] Vacuous class for types, enum for const size and index names (Ye)
• [x] WalkerLogCollector unit test, collect method (Ye)

Tasks for review round 3

• [x] Make step counter seem like a global data member
• [x] Long names for crowd member functions
• [x] Make more getters()
• [x] Inline default values for WalkerLogState
• [x] Try higher level api for hdf (results in nan)

Elements for reviewers to review

• WalkerLog* classes
• WalkerLogCollector unit test
• Driver integration

[jtkrogel]

NB: I will fix the failing unit test (complex estimators) in 30 min.

[jtkrogel]

Hold on merge for naming discussion for now. Will commit a naming change if the decision is made today.

[PDoakORNL]

There is a great deal to cover here. I'm just starting on the review. Please introduce one piece at a time from the most detailed up. I'd start with trace buffer and its unit tests. It needs a review on its own.

[jtkrogel]

Renaming applied. Ready to proceed.

[PDoakORNL]

That would be a big help to me as well. I have suggestions/changes needed for the lower level classes which would be better to address and test at that level. I.e. use of lower level hdf5 functions when there is already a way to easily pass containers.

Other issues relate the integration with existing higher level classes which are largely separate.

[jtkrogel]

See #5035

[ye-luo]

I have merged in develop and fixed cmake issues. Please pull before making further changes.

[jtkrogel]

I've addressed Peter's comment from #5035 requesting simplification of WalkerLogState.

[jtkrogel]

I've also tried Peter's suggestion to use f.writeEntry(buffer, "data") in WalkerLogBuffer::writeHDF. Making that replacement resulted in NaN's being written to the HDF files (picked up by integration test). Therefore I've kept the original implementation.

[jtkrogel]

Comments addressed. Ready to proceed.

[ye-luo]

I've also tried Peter's suggestion to use f.writeEntry(buffer, "data") in WalkerLogBuffer::writeHDF. Making that replacement resulted in NaN's being written to the HDF files (picked up by integration test). Therefore I've kept the original implementation.

You need h5d_append and we don't have a high-level API in hdf_archive right now. added issue #5038. WalkerLogBuffer::appendHDF can be a better name to avoid confusion.

[ye-luo]

Test this please

[jtkrogel]

Hi guys. I had thought this was going in given Ye's approval.

If there really is a short list of "must have" items still remaining for this PR after all these passes, please make a short, final list below and I will address them.

I need to get past this point and get back to other work, some of which depends on this code being in place.

[PDoakORNL]

Reverting the whitespace OperatorBase.cpp change to reduce the number files is the only must have.

[ye-luo]

Test this please