GoogleCloudDataproc / flink-bigquery-connector

BigQuery integration to Apache Flink's Table API
Apache License 2.0
23 stars 14 forks source link

Table API Implementation: `BigQuerySinkTableConfig` and `BigQueryConnectorOptions` #132

Closed prashastia closed 4 months ago

prashastia commented 5 months ago
  1. Adds DELIVERY_GUARANTEE in BigQueryConnectorOptions
  2. Adds builder to BigQuerySinkTableConfig
  3. Enables formation of BigQuerySinkTableConfig from BigQueryConnectorOptions
  4. Adds unbounded read mode to BigQueryDynamicTableSource and tests for the above...

The table Configuration for Read.

BigQueryTableConfig readTableConfig =
                BigQueryReadTableConfig.newBuilder()
                        .table(sourceTableName)
                        .project(sourceGcpProjectName)
                        .dataset(sourceDatasetName)
                        .testMode(false)
                        .partitionDiscoveryInterval(partitionDiscoveryInterval)
                        .boundedness(Boundedness.CONTINUOUS_UNBOUNDED)
                        .build();

The table Configuration for Sink.

BigQueryTableConfig sinkTableConfig =
                BigQuerySinkTableConfig.newBuilder()
                        .table(destTableName)
                        .project(destGcpProjectName)
                        .dataset(destDatasetName)
                        .testMode(false)
                        .build();

Please Note: (In Progress)

  1. BIGNUMERIC read will not work (error is values written)
  2. TIME type read will not work (connector breaks)

/gcbrun

prashastia commented 4 months ago

@clmccart @yerramalli Please review this PR. Thanks!