hbutani / icebergSQL

Integration of Iceberg table management into Spark SQL
Apache License 2.0
11 stars 4 forks source link

DelegatedMethod0 is not serializable #5

Closed yujiantao closed 4 years ago

yujiantao commented 4 years ago

This project is very helpful for us to deploy iceberg with spark sql support. When we test join with icebergSQL, we found serialization issue as below. After we make DelegatedMethod0 extend Serializable, the problem is still there, but this time the unserializable method is java.lang.reflect.Method, which is unchangeable. Is there any way to workaround?

select * from store_sales_out a join store_sales b on a.ss_item_sk=b.ss_item_sk where a.ss_item_sk=17908

Caused by: java.io.NotSerializableException: org.apache.spark.sql.iceberg.utils.DelegationUtils$DelegatedMethod0 Serialization stack:

After we make DelegatedMethod0 extend Serializable

Caused by: java.io.NotSerializableException: java.lang.reflect.Method Serialization stack:

hbutani commented 4 years ago

Oh good catch; busy today, tomorrow; will take a look over the weekend.

hbutani commented 4 years ago

Fixed in commit 63c26ae00bb03bb7d05bfbeac4fe6787b019d505