Investigate whether there are similar HDF5 data structures / interfaces

[wadawson]

We don't want to entirely reinvent the wheel with our data structure. I think the only astronomical survey currently using the HDF5 format is LOFAR

Plus there may be similar/existing data structures and tools that we want to be compatible with. From what I know and given a brief search there are a number of examples:

1) Perhaps we should have a dataset format that is easily comparable with python pandas. When you create an hdf5 file from within pandas it conveniently creates column names either based on user input or column names already associated with the pandas data frame. This does not appear to be so convenient for h5py or hdf5 in general.

• [x] Create a simple hdf5 file in pandas with column names and look how the hdf5 file is structured to see if this structure can easily be incorporated in what we are doing.

2) Similarly PyTables is something to investigate.

3) NASA Goddard has been using hdf5 datasets for representing some of their geo-gridded data related to their earth sciences programs. They even have a GUI interface call Panoply which has potential to be very useful for viewing our astronomical data.

4) Another NASA project using and visualizing HDF5 files is the Space Weather Explorer.

It is good that NASA is already using this on a few projects since we will want WFIRST to use this format.

5) One of the better graphical interface tools is Paraview.

There is a list of many of the HDF related software here. Note that I know this list is incomplete.

[mdschneider]

Is this all a wild territory of data structures, or is this some standard set by some active community or governing body? If not, is there a de facto NASA standard?

Some thoughts:

• Panoply looks specific to "geo-gridded" data. Perhaps too restrictive for us? Or will we always use sky coordinates in a similar way?
• PyTables is python specific and geared toward very large datasets.
• Pandas is convenient, but doesn't seem fully developed yet and doesn't work well as the datasets grow in size.

[wadawson]

I am think that there must be a de facto NASA standard since their Earth Observing System (EOS) project is largely the reason that HDF has become the de facto hierarchical data format (see e.g.).

• [x] Investigate NASA EOS HDF5 common structure.

[wadawson]

In the documentation section of this page there are some good documents describing the NASA HDF-EOS common structure. There are some aspects that probably won't be pertinent for LSST or other related surveys but it might be possible to shoehorn them in (probably not what we want). Regardless, let's just leave the NASA format documented at this stage and progress with our own ad hoc format to get things running on the development end.

[mdschneider]

I think we've settled this issue in our verbal discussions, so closing here.