Though HDF5 files can have arbitrary schemas, since we use GridLAB-D a lot its probably worth setting up a dedicated parsing path that totally re-works the DataFrame before giving it to the user. The name of the files produced by GridLAB-D are hard-coded so we should be able to easily detect when an HDF5 file is of that format. Features we should probably include:
Concatenating data from all days in the file into one table. Data is currently split up by day into unique tables "index1", "index2, ...
Making unique tables for each object in the table. Right now all objects are aggregated into one table which creates non-unique values in the table (all of the objects have a row with a given timestamp). Separating out each object into a table allows for indexing which is a good thing.
Index each table by the timestamp
Define data types for each column in the table. These are known as the output from GridLAB-D that creates this HDF5 is static (takes a source code change to adjust).
This will result in a lot of unique tables BUT they will all be easy to work with.
Though HDF5 files can have arbitrary schemas, since we use GridLAB-D a lot its probably worth setting up a dedicated parsing path that totally re-works the DataFrame before giving it to the user. The name of the files produced by GridLAB-D are hard-coded so we should be able to easily detect when an HDF5 file is of that format. Features we should probably include:
This will result in a lot of unique tables BUT they will all be easy to work with.