Provide utility program to gather metadata of a big dataset

[shaomeng]

Proposal: Introduce a utility program that goes over a big data set (potentially many time steps and multiple petabytes) and save a variety of metadata/statistics to an individual file.

Intended use case: Invoke this utility program once in batch mode and save the useful metadata. Then when using VAPOR in interactive mode, the saved metadata could be imported and used for speeding up multiple operations and/or more informative GUI displays.

Metadata/statistics to calculate

• Specification on what exactly the data set is that this metadata is for. (If the data set is changed on disk, VAPOR should know.)
• Minimum and maximum of each variable on each time step.
• Bounding box of each variable of each time step.
• Are there really missing values in a particular variable? If so, what percentage?
• What percentage of values are zeros? Maybe all zeros?
• Potentially a histogram of each variable at each time step.
• And so on. (The metadata/statistics calculation shouldn't be too expensive.)

(Note: the metadata file format should be designed so that a new metadata field could be easily added without breaking backward compatibility.)

(Note 2: the generated metadata file should not be tied together with wavelet transformed formats; rather, it will point to any existing file formats that VAPOR supports, including many unstructured files.)

[clyne]

One approach here might be to incorporate the above information into a VDC master file, and allow a VDC master file to referenced non VDC data. I.e. currently a VDC master file provides an indirect reference to VDC data files that contain the actual data. The VDC master file could easily be extended to reference other types of file such as NetCDF CF files.