search

visit-dav / visit

VisIt - Visualization and Data Analysis for Mesh-based Scientific Data
https://visit.llnl.gov
BSD 3-Clause "New" or "Revised" License
428 stars 111 forks source link

Support XZ and ZFP compression in BOV plugin #17770

Open markcmiller86 opened 2 years ago

markcmiller86 commented 2 years ago

Is your feature request related to a problem?

XZ compression is often 2-3x better than gzip...the BOV default. Files compressed by XZ can be distinguished from files compressed with gzip using magic header bytes. So, there is no need to consider BOV metadata changes to support XZ.

I believe same is possible with ZFP...that is its possible to read the first handful of bytes from a file and detect if it is a ZFP compressed stream. We need to confirm with @lindstro.

We should add support to the BOV plugin for these. This would mean some additional options in the writer and if ZFP is selected, ZFP compression parameters.

For the reader, things should just transparently work though I think we might wanna report in File->information (via a database comment) what kind of compression the files are using and what the compression parameters are if ZFP is in use.

markcmiller86 commented 2 years ago

look at Facebook's Z Standard compression too.

Can probably detect compression based on filename extensions specified in bov header.