Using the Java binding with Apache Flink and Apache Spark

[omidb]

Hi I'm trying to use FDB and Flink. It seems that some of the Java classes are not serializable.

[error] Exception in thread "main" org.apache.flink.api.common.InvalidProgramException: com.apple.foundationdb.DatabaseOptions@2ef3eef9 is not serializable. The object probably contains or references non serializable fields.
[error]     at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:151)
[error]     at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:126)
[error]     at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:126)
[error]     at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:71)
[error]     at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.clean(StreamExecutionEnvironment.java:1821)
[error]     at org.apache.flink.streaming.api.datastream.DataStream.clean(DataStream.java:188)
[error]     at org.apache.flink.streaming.api.datastream.DataStream.addSink(DataStream.java:1328)
[error]     at org.apache.flink.streaming.api.scala.DataStream.addSink(DataStream.scala:1144)
[error]     at OlayFlink$.main(OlayFlink.scala:23)
[error]     at OlayFlink.main(OlayFlink.scala)
[error] Caused by: java.io.NotSerializableException: com.apple.foundationdb.DatabaseOptions
[error]     at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1185)
[error]     at java.base/java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:349)
[error]     at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:586)
[error]     at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:133)
[error]     ... 9 more

Many of the Java libraries need full serialization support. It's the same problem in Apache Spark. JVM based big data tools need full serializability so they can send the object to remote executors.

[ajbeamon]

https://forums.foundationdb.org/t/java-serilizability/2233