Please Review this PR After PR #130 is merged. Then proceed to PR #132.
AvroToRowDataConvertors is a pre-existing implementation, however it does not take into account microsecond precision for TIMESTAMP and DATETIME (TIMESTAMP_WITH_LOCAL_TIMEZONE in Flink's LogicalType) which is required for reading TIMESTAMP and DATETIME BigQuery fields.
Note: Problems in Reading TIME type:
While reading from BigQuery, TIME type is read as a long value indicating the number of microseconds since EPOCH. However, when the read Avro Rows are converted to RowData (via AvroToRowDataConvertors) we can convert it into LONG orINT type.
Ideally, the conversion should be toLONG type (since the RowDataToProtoSerializer is expecting a LONG value of TIME(6) type).
But RecordEmitter uses RowDataSerializer which converts TIME type to INT, See SS causing the error. Which is disabling (throws the above described error) the read of time type from bigquery. Given correct RowData TIME type, the sink would function as expected.
Please Review this PR After PR #130 is merged. Then proceed to PR #132.
AvroToRowDataConvertors is a pre-existing implementation, however it does not take into account microsecond precision for
TIMESTAMPandDATETIME(TIMESTAMP_WITH_LOCAL_TIMEZONEin Flink's LogicalType) which is required for readingTIMESTAMPandDATETIMEBigQuery fields.Note: Problems in Reading
TIMEtype:TIMEtype is read as a long value indicating the number of microseconds since EPOCH. However, when the read Avro Rows are converted to RowData (via AvroToRowDataConvertors) we can convert it intoLONGorINTtype.LONGtype (since theRowDataToProtoSerializeris expecting aLONGvalue of TIME(6) type)./gcbrun