search

DataSQRL / sqrl

Flexible development framework for building streaming data applications in SQL with Kafka, Flink, Postgres, GraphQL, and more.
https://www.datasqrl.com/
97 stars 14 forks source link

Timestamp on nested data causes serializer exception #649

Open mbroecheler opened 6 months ago

mbroecheler commented 6 months ago

When using flexible-json on nested data that has a timestamp inside the nested array, this exception is thrown:

 Caused by: org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Java 8 date/time type `java.time.Instant` not supported by default: add Module "org.apache.flink.shaded.jackson2.com.fasterxml.jackson.datatype:jackson-datatype-jsr310" to enable handling
flink-jobmanager-1     |    at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:77) ~[flink-dist-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.SerializerProvider.reportBadDefinition(SerializerProvider.java:1306) ~[flink-dist-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.impl.UnsupportedTypeSerializer.serialize(UnsupportedTypeSerializer.java:35) ~[flink-dist-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.SerializerProvider.defaultSerializeValue(SerializerProvider.java:1148) ~[flink-dist-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.POJONode.serialize(POJONode.java:115) ~[flink-dist-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.POJONode.serialize(POJONode.java:111) ~[flink-dist-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ObjectNode.serialize(ObjectNode.java:425) ~[flink-dist-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ArrayNode.serialize(ArrayNode.java:320) ~[flink-dist-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.std.SerializableSerializer.serialize(SerializableSerializer.java:39) ~[flink-dist-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.std.SerializableSerializer.serialize(SerializableSerializer.java:20) ~[flink-dist-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480) ~[flink-dist-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319) ~[flink-dist-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4624) ~[flink-dist-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3892) ~[flink-dist-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at com.datasqrl.json.FlinkJsonTypeSerializer.serialize(FlinkJsonTypeSerializer.java:41) ~[?:?]
flink-jobmanager-1     |    at com.datasqrl.json.FlinkJsonTypeSerializer.serialize(FlinkJsonTypeSerializer.java:11) ~[?:?]
flink-jobmanager-1     |    at org.apache.flink.util.InstantiationUtil.serializeToByteArray(InstantiationUtil.java:498) ~[flink-dist-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.table.data.binary.BinaryRawValueData.materialize(BinaryRawValueData.java:113) ~[flink-table-api-java-uber-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.table.data.binary.LazyBinaryFormat.ensureMaterialized(LazyBinaryFormat.java:126) ~[flink-table-api-java-uber-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.table.runtime.typeutils.RawValueDataSerializer.copy(RawValueDataSerializer.java:60) ~[flink-table-runtime-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.table.runtime.typeutils.RawValueDataSerializer.copy(RawValueDataSerializer.java:36) ~[flink-table-runtime-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.table.runtime.typeutils.RowDataSerializer.copyRowData(RowDataSerializer.java:170) ~[flink-table-runtime-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.table.runtime.typeutils.RowDataSerializer.copy(RowDataSerializer.java:131) ~[flink-table-runtime-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.table.runtime.typeutils.RowDataSerializer.copy(RowDataSerializer.java:48) ~[flink-table-runtime-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:74) ~[flink-dist-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50) ~[flink-dist-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29) ~[flink-dist-1.18.1.jar:1.18.1]
flink-jobmanager-1     |    at StreamExecCalc$595.processElement(Unknown Source) ~[?:?]
henneberger commented 1 month ago

The jackson object mapper should use the java time module