search

sot / astromon

Astrometric accuracy (celestial location) monitor
0 stars 0 forks source link

Severely bloated HDF5 file #19

Closed taldcroft closed 2 years ago

taldcroft commented 2 years ago

The current astromon.h5 file is 550 Mb. Extracting the data from the files and re-writing separately (without the tables "mark" stuff for atomic transactions) gives a total of about 31 Mb. The 550 Mb is at a level where it becomes noticeable and annoying.

I'm not completely sure of the best strategy, but at 31 Mb there is no problem in copying the file off to the side, updating the original in place, and in the event of an exception mv'ing the copy back. Or something like that.

taldcroft commented 2 years ago

I'm wondering if the strategy of removing the node and making a new one is related to the large file size. Maybe a simpler strategy is to save each table in a separate file e.g. astromon_xray_src.h5. Then updating the table could be a matter of overwriting the entire file with Table.write().