amperity / sparkplug

Clojure API bindings for Apache Spark
Other
28 stars 1 forks source link

Fix map serialization and add serialization tests #1

Closed brandonvin closed 5 years ago

brandonvin commented 5 years ago

Nice library! I played with it a bit and found that map serialization wasn't working. Turns out it was writing each map entry as a single object but reading two objects at a time into map entries. For example, {:a "b"} would deserialize as {[:a "b"] nil}, if running in a local Spark context, and if running in cluster would result in a Kryo buffer underflow error. Here's a fix and a generative test for roundtripping Clojure data. Good news, looks like all the other serializers work well! 🙂