search

BlueQuartzSoftware / DREAM3D

Data Analysis program and framework for materials science data analytics, based on the managing framework SIMPL framework.
http://dream3d.bluequartz.net
Other
167 stars 77 forks source link

gzip support in dream3d file format #790

Open Micket opened 6 years ago

Micket commented 6 years ago

We develop a tool that outputs synthetic microstructures directly as a *.dream3d data file, and after a bit of debugging, I realized that it was the HDF standard compression of datasets I started using a while back.

I see a drastic decrease in file size with a simple gzip (default compression) turned on (over 50 times smaller!), so it would be a shame if it could not be supported in DREAM3D.

imikejackson commented 6 years ago

We can look into supporting this in DREAM.3D. Historically HDF5 did not include all the libraries needed for portable compression. By that I mean that some systems had zlib included and some did not which created all sorts of problems when creating a redistributable package. It is probably time to take another look at enabling the compression in HDF5.