mjakubowski84 / parquet4s

Read and write Parquet in Scala. Use Scala classes as schema. No need to start a cluster.
https://mjakubowski84.github.io/parquet4s/
MIT License
283 stars 65 forks source link

[question] is there a way to have a single parquet writer that writes for a class hierarchy? #298

Closed normana400 closed 1 year ago

normana400 commented 1 year ago

is there a way to get a single writer that writes a parquet file for a class hierarchy?

trait Animal( def color: String def legs: Int ) case class Cat extends Animal(color: String, legs: Int, isPetable: Boolean) case class Dog extends Animal(color: String, legs: Int, isGoodBoy: Boolean)

val cat = Cat("gray", 4, false) val dog = Dog("black", 4, true)

val writer = writer.of[Animal] writer.write(Seq(cat,dog))

mjakubowski84 commented 1 year ago

What Parquet requires to write a file is a schema. Parquet4s derives a schema (and an encoder from Scala types to Parquet types) from a class. This derivation is done by means of type classes. If you are able to provide a custom type class for Animal then it will work. Please read the documentation: https://mjakubowski84.github.io/parquet4s/docs/records_and_schema/