Flink avro `timestamp_mapping.legacy` config is not supported w/ filesystem connector

DataSQRL / sqrl

Flexible development framework for building streaming data applications in SQL with Kafka, Flink, Postgres, GraphQL, and more.

https://www.datasqrl.com/

97 stars 14 forks source link

Flink avro `timestamp_mapping.legacy` config is not supported w/ filesystem connector #968

Closed henneberger closed 1 week ago

henneberger commented 2 weeks ago

The filesystem connector does not acknowledge the timestamp_mapping.legacy flag for the avro format so avro records will fail to deserialize certain data types such as timestamp-millis. There currently is no workaround other than to disable the timestamp_mapping.legacy option.

CREATE TABLE ...
) WITH (
  'format' = 'avro',
  'path' = '...',
  'connector' = 'filesystem',
  'avro.timestamp_mapping.legacy' = 'false',
  'avro.encoding' = 'json'
);

henneberger commented 1 week ago

Fixed in #969. We will use the new type mapping by default for all .avro schemas