search

r-dbi / bigrquery

An interface to Google's BigQuery from R.
https://bigrquery.r-dbi.org
Other
517 stars 182 forks source link

Allow PARQUET format for uploading data. #609

Closed apalacio9502 closed 6 months ago

apalacio9502 commented 6 months ago

Hi @hadley,

This pull request contains the implementation for allowing the user to decide in which format they want to transmit the data (JSON or PARQUET) to BigQuery (For large amounts of data, loading data in JSON format is very time-consuming due to the size of the data that needs to be transmitted. To address this problem, BigQuery accepts other file formats, including Parquet). The most significant change enabling PARQUET data transmission is that the uploadType is no longer multipart; it is now resumable.

https://cloud.google.com/bigquery/docs/reference/api-uploads

Regards,

apalacio9502 commented 6 months ago

Hi @hadley,

Thank you for your review. I have taken into account all of your comments, and I hope I haven't missed any.

Regards,

apalacio9502 commented 6 months ago

Hi @hadley,

The implementation of Nanoparquet to replace Arrow has been completed. After several data loading tests, I believe it works very well.

I look forward to your comments.

Regards,

hadley commented 6 months ago

Thanks so much for working on this!