LDMX-Software / fire

Event-by-event processing framework using HDF5 and C++17
https://ldmx-software.github.io/fire/
GNU General Public License v3.0
1 stars 0 forks source link

schema evolution of event object types #48

Closed tomeichlersmith closed 2 years ago

tomeichlersmith commented 2 years ago

We achieve schema evolution by introducing two new features.

  1. Persisting of data "structure" where type and version attributes are given to every H5 object in the event object recursion tree so that the versioned type by which an object was serialized can later be read.
  2. Allow the objects being attached by a user class to either be only-loaded, only-saved, or both. This is the minimum structure necessary to rename variables on disk in order to update the schema or to calculate a variable in a different way.

To Do

tomeichlersmith commented 2 years ago

First round of benchmarking results shows no change when updating to handle schema evolution. It is important to note that the benchmarking does not test the speed of reading an old version with a new version; however, it does test the time necessary to serialize the extra type/version attributes.

recon_data production_data