runt18 / google-bigquery

Automatically exported from code.google.com/p/google-bigquery
0 stars 0 forks source link

support lz4 compression when uploading data #320

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago
LZ4 is one of fastest known LZ77 compressors today and is especially efficient 
when decompressing data. Its compression ratio is not like of gzip but it's 
generally good on text data. It's so good that it can actually save user time 
when writing a file to a persistent disk compared to a raw write in IO scarce 
environments like Google cloud.

Also, since it's 10 times more efficient than gzip it can save lots (lots!) of 
CPU cycles to Google and its clients if you guys support lz4 for uploading the 
data in addition to gzip.

Original issue reported on code.google.com by roma...@gmail.com on 10 Sep 2015 at 1:42

GoogleCodeExporter commented 8 years ago
Thanks for the feature request!  I've submitted this to our internal tracker.

Original comment by thomasp...@google.com on 24 Nov 2015 at 9:02

GoogleCodeExporter commented 8 years ago
Update from the engineers who own our ingestion pipeline: it is looking 
unlikely that we will add support for additional compression formats for the 
foreseeable future.

Original comment by thomasp...@google.com on 7 Dec 2015 at 11:19