Hive 3

[findepi]

Support Hive 3.1+ (HDP 3.1)

• [x] Create product-test environments with Hive 3.1 https://github.com/prestosql/presto/issues/190
• [x] Make product tests pass
  • [x] Make product tests run on HDP 3 https://github.com/prestosql/presto/pull/1615
• [ ] Use HDP 3.1 defaults in HDP 3.1 product test environment https://github.com/prestosql/presto/issues/1841
• [x] Run product tests on HDP 3 by default https://github.com/prestosql/presto/pull/1239
• Support reading Hive managed Transactional/ACID tables https://github.com/prestosql/presto/issues/576
  • [x] insert-only tables
  • [x] ORC ACID (#2068), (#1591)
  • [x] Support ORC ACID with base in raw format (https://github.com/prestosql/presto/issues/2292)
  • [x] Support reading ACID/Transactional tables with "original files" (https://github.com/prestosql/presto/issues/2293)
• Allow creation of and writing to Hive transactional/ACID tables https://github.com/prestosql/presto/issues/1956
  • [x] Allow creation of transactional/ACID tables
  • [x] Allow writes to transactional/ACID tables
• Accommodate to Hive timestamp changes https://issues.apache.org/jira/browse/HIVE-12192 "Hive should carry out timestamp computations in UTC" and related
  • [x] Fix ORC reader (io.prestosql.plugin.hive.AbstractTestHive#testTypesOrc)
  • [x] Fix Parquet reader (io.prestosql.plugin.hive.AbstractTestHive#testTypesParquet) (requires Hive 3.1.2, https://issues.apache.org/jira/browse/HIVE-21002, https://issues.apache.org/jira/browse/HIVE-21290)
    • [ ] proper defaults https://github.com/prestosql/presto/issues/5139
  • [x] Fix io.prestosql.plugin.hive.AbstractTestHive#testTypesRcBinary (requires https://issues.apache.org/jira/browse/HIVE-22167)
  • [ ] Add similar test for Avro (https://github.com/prestosql/presto/issues/5144; may require Hive 3.1.2, https://issues.apache.org/jira/browse/HIVE-21002, https://issues.apache.org/jira/browse/HIVE-21291)
• [x] TIMESTAMP behaviour does not match sql standard https://github.com/prestosql/presto/issues/37 is related due to https://issues.apache.org/jira/browse/HIVE-12192 "Hive should carry out timestamp computations in UTC"
• [ ] Support Hive timestamp with local time zone type (https://github.com/prestosql/presto/issues/1240)
• [x] Hive bucket version 2 tables https://github.com/prestosql/presto/issues/1201, https://github.com/prestosql/presto/issues/538
  • [x] Hive bucket version 2 for timestamp https://github.com/prestosql/presto/issues/1706
• [x] Handle Hive's information_schema https://github.com/prestosql/presto/issues/1192 https://github.com/prestosql/presto/pull/3008
• [x] Handle Hive's sys schema https://github.com/prestosql/presto/pull/3008
• [ ] Handle Hive's new privileges https://github.com/prestosql/presto/issues/4253 (https://github.com/prestosql/presto/pull/4254, https://github.com/prestosql/presto/pull/4307)
• [ ] Support Hive's DEFAULT https://github.com/prestosql/presto/issues/2048
• [ ] Support Hive's NOT NULL https://github.com/prestosql/presto/issues/2923
• [ ] Support role as an owner of hive table https://github.com/prestosql/presto/issues/5706
• [x] Document capabilities & limitations https://github.com/prestosql/presto/pull/3013
• [ ] ...

Related:

• [x] https://github.com/prestosql/presto/issues/456

[RugratsJ]

@findepi, I installed Presto 332 and HDP 3.1, when connecting to Hive and run select queries, I got Original non-ACID files in transactional tables are not supported. What should I do to at lease be able to query (read-only) to Hive 3? Thanks.

[findepi]

This is a high-level roadmap issue. Let's keep the discussion high level here only. @RugratsJ Please ping me on our slack.

[jiegzhan]

Added hive-3 missing privileges and principal:

https://github.com/prestosql/presto/pull/4254

https://github.com/prestosql/presto/pull/4307

[dain]

Is HDP end of life (https://endoflife.software/applications/big-data/hortonworks-data-platform-hdp)? Should we update the title to just Hive3?