search

jghoman / haivvreo

Hive + Avro. Serde for working with Avro in Hive
Apache License 2.0
59 stars 27 forks source link

Update to Avro 1.5.2 #3

Closed tomwhite closed 13 years ago

tomwhite commented 13 years ago

Unit tests pass and I managed to successfully create an Avro-backed Hive table and perform a simple query on it.

tomwhite commented 13 years ago

I just updated the message as you suggested, which should be sufficient until HIVE-1850 can be used.

tomwhite commented 13 years ago

Here's the change to support compression (such as Snappy) in Avro files written by Hive.

I tried it out by running test_serializer.sql with the following lines added just before the "insert overwrite" line:

SET hive.exec.compress.output=true; SET avro.output.codec=snappy;

I then manually verified that the resulting Avro file was snappy compressed (by looking at the raw bytes), and that it could be read using Avro tools.

I also needed to copy the snappy-java jar to Hive's lib directory before running the test.