add option to request lz4 compressed ReadRowsResponse

GoogleCloudDataproc / spark-bigquery-connector

BigQuery data source for Apache Spark: Read data from BigQuery into DataFrames, write DataFrames into BigQuery tables.

Apache License 2.0

374 stars 196 forks source link

add option to request lz4 compressed ReadRowsResponse #1222

Closed yqiu2 closed 4 months ago

yqiu2 commented 5 months ago

This adds the ability to request compressed ReadRowsResponses. If the server decides to give back a ReadRowsResponse, we use the specified compression algorithm (LZ4) to decompress the results. This also adds two integration tests to test requesting compressed results for both arrow and avro.