search

apache / incubator-xtable

Apache XTable (incubating) is a cross-table converter for lakehouse table formats that facilitates interoperability across data processing systems and query engines.
https://xtable.apache.org/
Apache License 2.0
925 stars 149 forks source link

Create BigQuery Sink #217

Open anoopj opened 1 year ago

anoopj commented 1 year ago

Build a BigQuery sync will allow customers to incrementally sync from table formats to BigLake Managed Tables (and BQ native tables).

The implementation of the sink will be different than other sinks:

  1. Instead of metadata translation, it will invoke a LOAD statement in BigQuery.
  2. It can only support added files and not removed files.

@the-other-tim-brown: Do you think this will be useful?

the-other-tim-brown commented 1 year ago

It's an interesting use case and could work with our existing framework. I'm curious if there are other tools that already do this though.

anoopj commented 1 year ago

I am not aware of any existing tooling. BigQuery LOAD jobs support only file formats (Parquet/ORC/Avro et al) - open table formats are not supported.