This project is not actively maintained. Proceed at your own risk!
A Hive Input/Output Library in Java.
The Hive project is built to be used through command line HQL.
As such it does not have a good interface to connect to it through code directly.
This project fixes that problem - it adds support for input/output with Hive in code directly.
Currently we have:
This project uses Maven for its build.
To build the code yourself clone this repository and run mvn clean install
To use this library in your project with maven add this to you pom.xml:
<dependency>
<groupId>com.facebook.hiveio</groupId>
<artifactId>hive-io-exp-core</artifactId>
<version>0.10</version>
</dependency>
You can also browse the released builds and download the jars directly.
The project's maven site
is generated with each release.
API information is available here:
JavaDocs.
If you just want to quickly read or write a Hive table from a Java process, these are for you.
The input API is simply:
public class HiveInput {
public static Iterable<HiveReadableRecord> readTable(HiveInputDescription inputDesc);
}
And the output API is:
public class HiveOutput {
public static void writeTable(
HiveOutputDescription outputDesc,
Iterable<HiveWritableRecord> records)
}
For example usages of these take a look at the cmdline apps and the tests in the code.
The input and output classes have a notion of profiles. A profile is tagged by a name and describes the input or output you wish to connect to. Profiles are serialized into the Hadoop Configuration so that they can be passed around easily. MapReduce takes classes you configure it with and instantiates them using reflection on many hosts.
This sort of model makes it hard to configure the classes beyond just setting which to use. The approach we have chosen is to have the user create input / output descriptions, and set them with a profile. This writes it to the configuration which gets shipped to every worker by MapReduce. See example code mentioned below for more information.
The Hive IO library conforms to Hadoop's InputFormat API. The class that implements this API is HiveApiInputFormat. MapReduce creates the InputFormat using reflection. It will call getSplits() to generate splits on the hive tables to read. Each split is then sent to some worker, which then calls createRecordReader(split). This creates a RecordReaderImpl which is used to read each record.
The Hive IO library conforms to Hadoop's
OutputFormat API.
The class that implements this API is
HiveApiOutputFormat.
MapReduce creates the OutputFormat using reflection.
It will call checkOutputSpecs() to verify that the output can be written.
Then it will call createRecordWriter() on each map task host.
These record writers will get passed each record to write.
Finally it will call getOutputCommitter() and finalize the output.
Because of its use of reflection, to make use of this library you will need to extend HiveApiOutputFormat (and potentially HiveApiRecordWriter).
The best way to get started with output is to have a look at the example mapreduce
code.
This is a simple example of using the library that should be used as a starting point for projects.
Using Hive IO we were able to read from Hive at 140 MB/s. For comparison, using the hive command line on the same partition reads at around 35 MB/s. This benchmark is an evolving piece of work and we still have performance tweaks to make. Code for the benchmark is located here.
This library comes with a hive reader, called hivetail, that can be used to stream through Hive data.
It reads from a Hive table / partition and dumps rows to stdout.
It is also a good starting point for implementing input with this library.
The code is located
here.
To use it first build the code or download the jar from
Maven Central.
Then on your machine run java -jar hive-io-exp-cmdline-0.10-jar-with-dependencies.jar help tail
.
The cmdline
module contains a suite of command line tools using HiveIO.
The jar-with-dependencies that it builds is completely standalone.
It includes hive jars and everything else it needs so that it can be run directly on the command line.
Moreover it does not use mapreduce, but rather executes completely in a single JVM.
This package contains:
Additionally there is a mapreduce
module that contains a suite of tools for MapReduce jobs with HiveIO.
Currently it contains an example MR job that writes to a Hive table.
Giraph uses Hive IO to read and write graphs from Hive.
It is another good example that can be used for starter code.
It is a more complicated use case than the examples mentioned above as it converts records to graph vertex and edge objects.
It has scaled to terabytes of data and is running in production at large companies (e.g. Facebook).
The code is
here.
Specifically you should look at
vertex-input,
edge-input,
and output.
Note especially how the vertex and edge inputs use different profiles to allow reading from multiple Hive tables.