abh1sh2k
commented
5 years ago I have created a pr https://github.com/abh1sh2k/spark-llap/pull/1/files here . if anybody can look . It is working fine for spark 2.4. Would love to make pr in this repo? currently it seems I don't have permission to even push to a branch .
carloslima
Currently method like executeQuery does not work when running this library with spark 2.4.0 because in spark 2.4.0 some classes have been removed and some classes have been renamed 1) DataReaderFactory have been renamed to InputPartition 2)DataReader have been renamed to InputPartitionReader 3)SupportsWriteInternalRow have been removed
Exception while running executeQuery
java.lang.NoClassDefFoundError: org/apache/spark/sql/sources/v2/reader/DataReaderFactory at com.hortonworks.spark.sql.hive.llap.HiveWarehouseConnector.getDataSourceReader(HiveWarehouseConnector.java:64) at com.hortonworks.spark.sql.hive.llap.HiveWarehouseConnector.createReader(HiveWarehouseConnector.java:37) at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation$SourceHelpers.createReader(DataSourceV2Relation.scala:155) at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation$.create(DataSourceV2Relation.scala:172) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:204) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167) at com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.executeQuery(HiveWarehouseSessionImpl.java:62) ... 49 elided Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.sources.v2.reader.DataReaderFactory at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 56 more