confluentinc / kafka-connect-bigquery

A Kafka Connect BigQuery sink connector
Apache License 2.0
3 stars 2 forks source link

Storage Write API: Adds conversion for Bytes schema type #323

Closed b-goyal closed 1 year ago

b-goyal commented 1 year ago

This PR adds logic to support Bytes in BigQuery table schema. Storage Api support ByteString for a bytes type. (You may find the related google support ticket in below mentioned JIRA ticket)

This PR contains additional logic to covert map to JSONObject as new JSONObject(map) does not support complex types. ByteString was getting converted to {empty: false} JSONObject due to JSOnObject's internal implementation.

After making these changes, records following below schema has been tested with both old and new API.

Screenshot 2023-05-09 at 12 27 29 PM Screenshot 2023-05-09 at 12 27 49 PM

JIRA: CC-20287 EPIC: CC-19644