Type change of HiveWarehouseSession from interface to class in HDP 3.1.4

hortonworks-spark / spark-llap

Apache License 2.0

101 stars 68 forks source link

Type change of HiveWarehouseSession from interface to class in HDP 3.1.4 #277

Open derSascha opened 4 years ago

derSascha commented 4 years ago

Hi,

we wrote Spark code that works on HDP 3.x using the HiveWarehouseSession. In the last version (HDP 3.1.4) it fails with:

java.lang.IncompatibleClassChangeError: Found class com.hortonworks.hwc.HiveWarehouseSession, but interface was expected

This repository does not contain any commits on that, but the files here and the jar files in HDP 3.1.4 contains an abstract class instead of an interface.

Why was that changed? This breaks all compiled code... Is there a known workaround to be compatible with HDP 3.x and HDP 3.1.4+?

derSascha commented 4 years ago

As a workaround, parts of the HiveWarehouseSession can be used directly:

import com.hortonworks.hwc.HiveWarehouseSession;
...
HiveWarehouseSession session = HiveWarehouseSession.session(sparkSession);

becomes

import com.hortonworks.spark.sql.hive.llap.HiveWarehouseBuilder;
import com.hortonworks.spark.sql.hive.llap.HiveWarehouseSession;
...
HiveWarehouseSession session = HiveWarehouseBuilder.session(sparkSession);

SalNedia commented 3 years ago

I used in this way: import com.hortonworks.spark.sql.hive.llap.HiveWarehouseBuilder; import com.hortonworks.spark.sql.hive.llap.HiveWarehouseSession; ... HiveWarehouseSession session = HiveWarehouseBuilder.session(sparkSession); HiveWarehouseSessionImpl hive = session.build(); ... hive.setDatabase(db_name); ...

and i get:

User class threw exception: java.lang.NoSuchMethodError: com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.sql(Ljava/lang/String;)Lorg/apache/spark/sql/Dataset;

Any suggestion?