search

htrc / htrc-feature-reader

Tools for working with HTRC Feature Extraction files
37 stars 12 forks source link

Store parquet volume metadata on the parquet tokencounts file #40

Open bmschmidt opened 3 years ago

bmschmidt commented 3 years ago

I've been happily using feather the last year as a backend with volume metadata on the files.

But parquet is smaller than feather, and also supports volume and chunk-level metadata.

So rather than storing metadata in a separate .json file, I propose storing the data in the footer of the parquet file, and caching it when needed with pyarrow.parquet.read_metadata.

Reason:

  1. Fewer files. The big one for me--taking up 2x as many inodes is a problem in an HPC environment.
  2. More portable

Downsides:

  1. Requires a pyarrow dependency for parquet, rather than flexibility about using fastparquet instead.
  2. Requires either breaking exist pyarrow installations, or some term-unlimited multiple format support.
bmschmidt commented 3 years ago

Goes without saying I'm volunteering to do these tasks. Can push to the Denver repo first. I've decided to move my personal workflow away from feather from parquet because it's more space-efficient, but would need these cghanges to do it.

organisciak commented 3 years ago

I think that's fine. In the short term, I wouldn't want to make the json metadata unsupported - perhaps just discouraged - because it's too soon after the last changes.

bmschmidt commented 3 years ago

Hmm. Does raise an issue though. The current schema makes it sensible to store only, say, character counts as parquet, in which case you'd need to bind the metadata to all three... Rather than deprecate or discourage json metadata, I'll just allow it to be written, but also include the metadata in tokenlist and read from there first.