Parquet Complex Data Support for Big Decimals and INT96

[gna-phetsarath]

Motivation

The primitive type support was extended to support:

• fixed_len_byte_array(L) DECIMAL(p,s) with BigDecimal and BigInteger conversion
• INT96 Timestamp

Supported Complex Data Type Conversions

Parquet Data Type	Converted Scalding Type
fixed_len_byte_array(L) field_name (DECIMAL(p,s))	BigDecimal(p,s) => String if s == 0, and p <= 10 => Int if s == 0 and p <= 18 => Long
INT96	Date to String formatted - Timestamp.toString "2011-01-01 00:00:00.123456789" "yyyy-mm-dd hh:mm:ss.SSSSSSSSS"

Changes

• Modified
  • ParquetTupleConverter
• Added
  • Conversion Utils for Decimal and ParquetTimestamps
  • Unit Tests

Tests

• Added new unit tests
• imports org.apache.hive | hive-exec for test utils for Timestamp to NanoTime conversions

Tested

sbt clean test

[CLAassistant]

All committers have signed the CLA.

[gna-phetsarath]

@johnynek and @non: Would it be possible for you to review? Thank you.