search

gt-space / servo

Control server software that coordinates actions between all other systems.
0 stars 0 forks source link

HDF5 Data Exports #1

Closed AndrewCarlisleECE closed 7 months ago

AndrewCarlisleECE commented 7 months ago

Data exports need to be more robust. Currently we have support for CSV exports, which are decently usable but incredibly space-inefficient and somewhat difficult to parse with units. CSV data export support should remain, but it should be made more efficient, as the code to support exports was written quickly.

Data exports to HDF5 should be supported as well, so that the data is effectively labeled and in a form that conserves more space without losing any precision. Embedding units into the measurement data is also a must.

  1. Exports should go through the existing endpoint at /data/export.
  2. servo export should export to HDF5 by default, with explicit flags available for --csv and --hdf5
  3. Prepare for GUI support; an operator should be able to easily export from the GUI.

As with most tool actions, this kind of change requires two parts:

Changing the API route(s) to support this behavior.
Changing the arguments and behavior of the command line tool portion, which makes a request to the API.

I would recommend doing one of these entirely and then the other; specifically, you should probably do the changes to the export API route before working on integrating it with the tool, which should be comparatively simple.

AndrewCarlisleECE commented 7 months ago

Expanded on the current API for HDF5. Improvements on networking via switching to streaming / preventing timeouts on large scale exports will be moved to another issue.

Export type is also determined by input file type instead of flags per type.