search

vincenzobaz / spark-scala3

Apache License 2.0
89 stars 15 forks source link

Support Spark 3.2.2+ #20

Closed megri closed 1 year ago

megri commented 2 years ago

The field isDeterministic: Boolean = true was added to the case class Invoke somewhere between version 3.2.0 -> 3.2.2 (commit where this happened: https://github.com/apache/spark/commit/3860ac5d2313d438c25e5c46ed4e2b8b2b5227e3)

This causes the derivation to fail during compile time.

It's probably not very nice to support this across several versions of spark 🤔

vincenzobaz commented 2 years ago

Thank you for reporting this @megri This is a development and experimentation library, so it makes sense to support the latest Spark version only. It would amazing if you could open a PR which upgrades the Spark version and supports the new field!

In the meantime you could also have a look at Iskra where the derivation of encoders is done in a completely different way. Unfortunately nowadays I do not have a lot of time to maintain this library, but let me know if you cannot open the PR and I will see if I can spare some time for it.

megri commented 2 years ago

Seems like all that was needed was to recompile the project with the newer dependency. 😄