search

bkvarda / kudusparklyr

A Kudu extension for Sparklyr
11 stars 5 forks source link

create kudu table errors #5

Closed krb16704 closed 7 years ago

krb16704 commented 7 years ago

create_kudu_table results in:

Error: java.lang.Exception: No matched method found for class org.apache.kudu.spark.kudu.KuduContext.createTable

It seems like the function is just a wrapper for the spark calls, however, it seems like it is broken. Running cdh 5.12 with kudu 1.4 and spark 2.2

bkvarda commented 7 years ago

Yes it seems there are some issues with Spark 2.2 integration now - I'm still investigating.

bkvarda commented 7 years ago

@krb16704 this is caused by the sdf_schema() function shipped with core sparklyr returning something that is fundamentally different than what sdf_schema() returned in this package before I removed it. create_kudu_table expects the schema to be a schema object as opposed to a printed representation of the schema. I have re-added sdf_schema() to this package which resolved the issue in my testing.

krb16704 commented 7 years ago

Seems like it's functioning properly now. I'm closing the issue out.