Open jcannelos opened 6 years ago
Hi J'son :)
Happy you've found Spectrify useful. Regarding your question -- I think that's the easiest way right now... if it's any comfort, it used to be significantly more difficult, so it's at least it's trending in the right direction!
The default is gzip
because in the benchmarks I performed:
Either/both of those may have been artifacts of our configuration, or may have changed since those tests (last October).
Ways forward:
SPECTRIFY_COMPRESSION=snappy
?Thanks, Colin
Hi all, this is a great product and has saved us a ton of time on our Redshift to Spectrum transition. Question: Is it possible to store the Parquet files in snappy format, rather than gzip? I can see in Writer._get_writer where it's being specified as gzip. Do I have to sub-class Writer, then CsvManifestConverter and ConcurrentManifestConverter in order to specify snappy or is there a simpler way?
Thanks!
Sincerely,
J'son