HDF5 Estimator Testing File Generation

Fixes #214

Purpose

Running a whole psim MATLAB sim for estimator testing is slow. To test an estimator you only need a small subset of sensor readings and to compare that with the true value you are trying to estimate. To speed up testing I am storing this data in HDF5 files from a psim MATLAB sim or python script, so it can be used over and over again to quickly test and tune estimators in real time. HDF5 is language and machine agnostic so if I upload it to GitHub theoretically anyone can quickly open and use the data in MATLAB, numpy or some other language.

Summary of changes

Python and Matlab scripts to generate HDF5 files for estimator testing.

HDF5 Files

HDF5 is a self-describing binary file format. Each file is composed of groups and datasets. A group is like a dictionary of other groups and datasets. A dataset is like a numpy array or MATLAB array. Each group or dataset can also have a dictionary attributes.

To look at what is in an HDF5 file use command h5dump.

h5dump estimatortest/test-files/gps-from-matlab-sim.hdf5 Will print out the entire file and the data.

Using the -n flag shows the just the main structure.

$ h5dump -n estimatortest/test-files/gps-from-matlab-sim.hdf5 
HDF5 "estimatortest/test-files/gps-from-matlab-sim.hdf5" {
FILE_CONTENTS {
 group      /
 dataset    /sensors
 dataset    /truth
 }
}

The files used for estimator testing just have the root group "/" with two datasets: "sensors" and "truth". These datasets contain the sensor data used for an estimator and the real values the estimator is estimating for each control cycle, indexed first by control cycle.

For example here sensors is an array of type H5T_COMPOUND With data for every control cycle for 24hr and an attribute "docs" with a simple doc string:

h5dump -d /sensors -A estimatortest/test-files/gps-from-matlab-sim.hdf5
HDF5 "estimatortest/test-files/gps-from-matlab-sim.hdf5" {
DATASET "/sensors" {
   DATATYPE  H5T_COMPOUND {
      H5T_STD_I64LE "t";
      H5T_ARRAY { [3] H5T_IEEE_F64LE } "r";
      H5T_ARRAY { [3] H5T_IEEE_F64LE } "v";
   }
   DATASPACE  SIMPLE { ( 720000 ) / ( 720000 ) }
   ATTRIBUTE "docs" {
      DATATYPE  H5T_STRING {
         STRSIZE 50;
         STRPAD H5T_STR_NULLTERM;
         CSET H5T_CSET_ASCII;
         CTYPE H5T_C_S1;
      }
      DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
      DATA {
      (0): "GPS sensor data in ECEF and GPS time (ns, m, m/s)"
      }
   }
}
}

pathfinder-for-autonomous-navigation / psim