search

hortonworks-spark / spark-llap

Apache License 2.0
101 stars 68 forks source link

Add LLAP ReadSupport #237

Closed EricWohlstadter closed 6 years ago

EricWohlstadter commented 6 years ago

What changes were proposed in this pull request?

Add impls. for the ReadSupport interfaces: DataSourceV2, DataSourceReader, DataSourceReaderFactory, and DataReader.

Refactor some code into Util classes.

Move some non-static methods in LlapRelation classes to DataSourceV2 classes.

Add some Mock classes to support UT.

How was this patch tested?

UT

EricWohlstadter commented 6 years ago

Build failures are expected. Tests pass on my local with Hive branch-3.

dongjoon-hyun commented 6 years ago

For the build failure, which Hive Apache JIRA is adding org.apache.hadoop.hive.ql.io.arrow then? Should we use Apache Hive snapshot instead of internal repo?

EricWohlstadter commented 6 years ago

@dongjoon-hyun

To build, we need:

  1. HIVE-19306 (introduces ArrowWrapperWritable and RootAllocatorFactory)
  2. HIVE-19495 (refactors HIVE-19306)
  3. HIVE-19308 (introduces LlapArrowBatchRecordReader and modifies LlapBaseInputFormat)
EricWohlstadter commented 6 years ago

@dongjoon-hyun

Can we build against Hive 3.1.0-SNAPSHOT? I didn't know it was being published anywhere. If it is, that should work.

dongjoon-hyun commented 6 years ago

Could you build, verify, and merge this manually, @HyukjinKwon ?

HyukjinKwon commented 6 years ago

Sure.

HyukjinKwon commented 6 years ago

@EricWohlstadter, are you able to build this against Hive 3.1.0-SNAPSHOT? Seems I can't even with locally installed Hive one.

EricWohlstadter commented 6 years ago

@HyukjinKwon

What build error are you seeing? I am able to pass build/test locally after doing a local mvn install of Hive master (or branch-3). I'm using:

build/sbt clean assembly -Drepourl=http://nexus-private.hortonworks.com/nexus/content/groups/public/ -Dhadoop.version=3.0.0.3.0.0.0-SNAPSHOT -Dhive.version=3.1.0-SNAPSHOT
HyukjinKwon commented 6 years ago

@EricWohlstadter, before merging this, mind if I ask to fix some style nits by running build/scalastyle?