add option to request lz4 compressed ReadRowsResponse

GoogleCloudDataproc / spark-bigquery-connector

BigQuery data source for Apache Spark: Read data from BigQuery into DataFrames, write DataFrames into BigQuery tables.

Apache License 2.0

378 stars 198 forks source link

add option to request lz4 compressed ReadRowsResponse #1222

Closed yqiu2 closed 6 months ago

yqiu2 commented 6 months ago

This adds the ability to request compressed ReadRowsResponses. If the server decides to give back a ReadRowsResponse, we use the specified compression algorithm (LZ4) to decompress the results. This also adds two integration tests to test requesting compressed results for both arrow and avro.