0.198 release notes

[martint]

Dain Sundstrom

• [x] 03-20 19:28:41 5429cc4e79 Categorize error creating Hive imput format
• [x] 03-21 19:41:26 23c70870bd Use simple non-partitioned output for scaled table writer
• [x] 03-21 19:59:35 97bba29608 Remove unnecessary calls to Page.getBlocks
• [x] 03-21 19:59:37 38dd559489 Change TopN to store Page instead of Block[]
• [x] 03-21 20:16:40 f545a6cab9 Add Page.appendColumn
• [x] 03-21 20:16:42 96e96f093a Change ExpressionInterpreter to operate on Page instead of Block[]
• [x] 03-21 20:16:42 b17bdeb6a6 Change index to not use Page.getBlocks
• [x] 03-21 20:16:43 17ee70af8d Remove unnecessary calls to Page.getBlocks in AggregationTestUtils
• [x] 03-21 20:16:43 61d932375f Remove Page.getBlocks from JoinFilterFunction
• [x] 03-21 20:16:43 abafe12eec Add instance overhead to Page.getRetainedSizeInBytes
• [x] 03-21 20:16:43 af1393492f Remove Page.getBlocks()
• [x] 03-28 10:25:19 6a533dd8d9 Remove old plan logic from QueryMonitor
• [x] 03-28 10:25:19 a6ef836612 Use explain plan in QueryCompletedEvent if possible
• [x] 03-28 10:25:19 d45566ed02 Change PlanPrinter to use FunctionRegistry instead of Metadata
• [x] 03-28 10:25:19 d581df0cc3 Do not hold on to config in QueryMonitorConfig
• [x] 03-28 17:30:50 a666f20f47 Convert MapBlock protected constructor to a factory method
• [x] 03-28 17:30:50 bef7519022 Share validation code between MapBlock factories
• [x] 03-28 17:30:50 e3a6325d1d Validate null supression in structural blocks

  David Phillips

• [x] 03-20 17:29:09 b3cf39330f Add profile to skip Accumulo tests
• [x] 03-22 17:23:16 fa6b2e3d54 Fix NPE in Raptor shard cleaner
• [x] 03-26 12:39:32 0e1cf99098 Rename DB resource groups test classes
• [x] 03-26 12:39:32 db750277c2 Replace query queues with resource groups
• [x] 03-26 17:09:37 349ece155a Do not require table creation for SHOW TABLES test
• [x] 03-26 20:42:16 0a0bc694f5 Allow listing missing schemas in Accumulo
• [x] 03-26 21:39:10 84be87ec8b Add support for Glue Hive metastore
• [x] 03-27 09:59:28 dd9f4db6a0 Don't reuse WKTReader across threads
• [x] 03-27 13:04:25 3f3ba3f7db Minor cleanup of Glue metastore
• [x] 03-27 13:04:25 fe290e3dc5 Fix thread safety in Glue metastore tests
• [x] 03-28 11:36:25 323b78f595 Remove --enable-authentication option from CLI
• [x] 03-28 11:37:54 56b14b4cf4 Enforce HTTPS for Kerberos authentication in CLI

  Grzegorz Kokosiński

• [x] 03-19 15:34:45 69a006879b Remove not used correlations in AddExchanges
• [x] 03-19 15:34:45 caf4c9aa14 Inline AddExchanges#Context
• [x] 03-22 13:37:50 067dd4cdfd Test that order of parameters in JMX table is ignored
• [x] 03-22 13:37:50 17fea35c42 Update JMX docs about wildcards
• [x] 03-22 13:37:50 21a33cb47a Make JmxTableHandle support multiple objectNames
• [x] 03-22 13:37:50 6e14e2143d Add support for wildcard patterns in JMX table name
• [x] 03-22 13:37:50 83432f66d8 Use Compartor in JmxMetadata
• [x] 03-22 13:37:50 a5fc80b4e6 Use ObjectName#getCanonicalName as JMX table name
• [x] 03-22 13:37:50 ec71cfc48f Use stream with ImmutableMap collector over plain for loop
• [x] 03-25 22:43:12 8606ff99e6 Tests for predicate pushdown on partition column
• [x] 03-25 22:43:12 b491271785 Optimize tableScan with layout with TupleDomain:none

  Haozhun Jin

• [x] 03-22 16:08:48 783fb1772b Add SignatureBinder test case for lambda that returns orderable
• [x] 03-26 17:17:58 82e4ab5017 Add default step for date variant of sequence function

  James Sun

• [x] 03-23 13:05:26 6c08d90042 Add string stats limit to ORC validation
• [x] 03-23 13:05:26 8041762b5f Fix null case for actualStringStatistics in ORC validation
• [x] 03-23 13:05:26 96d28a7ac9 Fix source slice in OrcOutputBuffer::writeBytes

  Jiexi Lin

• [x] 03-28 12:10:00 79f2f9ec93 Fix ordinal_position when adding new column

  Karol Sobczak

• [x] 03-20 01:40:44 eb64190dfb Make local exchange buffer size configurable
• [x] 03-20 04:08:13 0b4698cc8e Extract validateValueIsPowerOfTwo method
• [x] 03-20 04:08:50 00a340494d Make StreamPreferredProperties#constrainTo preserve partitioning columns order
• [x] 03-20 04:08:50 2436992471 Do not create redundant preferredChildProperties
• [x] 03-22 06:50:58 772a3cf653 Remove unused PlanRewriter
• [x] 03-23 01:50:44 abcf5303e6 Add default and S3 profile to hive-hadoop2 module
• [x] 03-26 03:00:23 013424f0df Extract common hive tests container methods to separate file
• [x] 03-26 03:00:23 351a20e99e Run Hive S3 tests via Travis
• [x] 03-26 03:00:23 45913d5427 Use writableBucket in AbstractTestHiveClientS3#testGetFileStatus
• [x] 03-26 03:00:23 bb8b6cbd4e Allow running product tests against S3
• [x] 03-26 03:00:23 f6a15cc907 Do not start Hive test in subprocess
• [x] 03-26 12:28:24 3e844d6f5a Improve local aggregation parallelism
• [x] 03-26 12:28:24 b07c5bfdf3 Extract hasSingleNodeExecutionPreference method into AggregationNode
• [x] 03-28 06:48:20 143f2cf2f9 Extract StreamPropertyDerivations#derivePropertiesRecursively method
• [x] 03-28 06:48:20 269244aaf9 Move null replication check from PropertyDerivation to AddExchanges
• [x] 03-28 06:48:20 3a9ea72e7c Add com.facebook.presto.util.Optionals utility class
• [x] 03-28 06:48:20 3d5df12a2f Extract PropertyDerivations#derivePropertiesRecursively method
• [x] 03-28 06:48:20 4b197eb526 Add validator for aggregations that produce default values
• [x] 03-28 06:48:20 7f4b4a52d5 Include ValidateAggregationsWithDefaultValues in PlanSanityChecker
• [x] 03-28 06:48:20 9a68bed81d Propagate single and coordinator partitionings through local exchange

  Maria Basmanova

• [ ] 03-20 08:41:55 5671e711fe Optimize serialization and deserialization of Geometry types
• [x] 03-20 08:48:39 10d24a9a72 Rename an argument in JoinFilterFunction for clarity
• [ ] 03-20 08:48:39 831823597f Fix row addressing in spatial index's filter
• [ ] 03-20 12:49:06 9ebf3bfb50 Add ST_IsValid and geometry_invalid_reason functions
• [ ] 03-20 13:59:55 239f726d90 Optimize ST_Intersects geometry function
• [x] 03-20 13:59:55 257c561134 Add maven flag to create tests jar with dependencies
• [ ] 03-20 13:59:55 425205b3e2 Improve GeometryBenchmarkUtils#loadPolygon
• [x] 03-20 13:59:55 c3742c5470 Pick Benchmark* files as potential TestNG tests
• [ ] 03-21 03:11:35 635d7f580c Add simplify_geometry function
• [ ] 03-22 08:21:29 b883ec5dc6 Apply assertEquals directly to geometry objects
• [ ] 03-22 08:21:29 c95cd79a8e Upgrade esri-geometry-api to 2.1.0 (from 1.2.1)
• [ ] 03-22 08:21:29 f01ca18748 Switch to ESRI's estimateMemorySize API
• [ ] 03-22 13:56:33 7467b00eeb Add great_circle_distance function
• [ ] 03-23 13:03:33 51e6a49647 Broadcast distance query

  Martin Traverso

• [x] 03-19 19:35:41 8c770dd2ac Move implementation of DISTINCT aggregates to optimizer rule
• [ ] 03-19 21:08:29 749de301e3 Analyze query before enqueuing
• [x] 03-20 15:28:32 fc2fa98f47 Fix incorrect parameter order
• [x] 03-26 21:45:36 0095d1d583 Fix bounds check in MultiChannelGroupByHash
• [x] 03-26 21:45:36 3d4d6ca3b6 Tighten check for splitting aggregation node
• [ ] 03-26 21:45:36 5d6072ed89 Add support for filtered + distinct aggregations
• [ ] 03-26 21:45:36 78f8146548 Implement DISTINCT in aggregation operator
• [x] 03-28 12:18:34 2328f43e12 Add missing parameters to exception message

  Nezih Yigitbasi

• [x] 03-20 12:49:45 34017617de Destroy buffers sooner when client buffers are destroyed
• [x] 03-20 14:57:42 04c5ad2ae4 Add trace token support to scheduler and exchange HTTP clients
• [x] 03-22 01:51:19 b010c183ca Construct the error message only on error
• [x] 03-28 17:01:40 cc37ca3a37 Add peak task total memory usage to PrismQueryCompletedEvent

  Piotr Findeisen

• [x] 03-19 15:50:33 4bbf9aad18 Fix epoch day to java.sql.Date conversion
• [x] 03-19 15:50:53 d2af24fa52 Construct proper literal for DECIMAL in LiteralInterpreter.toExpression
• [x] 03-19 15:51:12 5bcc5b8108 Make sure every Rule has a name
• [x] 03-19 15:51:12 998d200618 Fix grammar in PreconditionRules's method name
• [x] 03-19 15:52:44 51a80fc059 Fix CHAR comparisons in TPC-DS product tests
• [x] 03-19 15:52:44 52bfad6bef Fix CHAR comparisons in TPC-DS benchmarks
• [x] 03-19 15:52:44 582271b4b8 Remove redundant semicolons from product tests' TPC-DS queries
• [x] 03-19 15:52:44 b0928ddf2d Increase cast precision in q58 to support datasets larger than 100GB
• [x] 03-19 15:52:44 b120029051 Unquarantine TPC-DS q08
• [x] 03-19 15:52:44 cdba0a47f0 Unquarantine TPC-DS queries related to 'null IN subquery'
• [x] 03-20 07:50:39 15dff4c520 Parse decimal literals as DECIMAL by default
• [x] 03-21 03:17:54 873152dbe0 Fix current_date when session zone had gap at 1970-01-01 00:00:00
• [x] 03-21 09:09:32 308f7a25c1 Inline com.facebook.presto.matching.Util
• [x] 03-21 09:09:32 b82b1652a5 Simplify com.facebook.presto.matching.Util
• [x] 03-21 13:40:19 a3cb42abd5 Do not use Streams in Block/Pages implementations
• [x] 03-23 01:52:11 5b2201dd03 Add LikePredicate constructor accepting escape as Optional
• [x] 03-23 01:52:12 294eaf0964 Support ESCAPE in LIKE pattern of SHOW SCHEMAS and SHOW TABLES
• [x] 03-23 04:39:10 831ce5a070 Remove unused code in InCodeGenerator
• [x] 03-23 08:40:16 314cfe1aae Create Ordering in CostComparator
• [x] 03-23 14:44:01 250aa04e49 Add missing determinism declarations
• [x] 03-23 14:44:01 553299b103 Refactor TestSqlToRowExpressionTranslator
• [x] 03-23 14:44:01 7713581f71 Optimize DECIMAL constants for byte code generation
• [x] 03-23 14:44:01 a696c3a478 Require explicit determinism declaration in scalar builder
• [x] 03-23 14:44:01 be417dfa72 Remove redundant else
• [x] 03-23 14:44:01 d92855e9a5 Remove duplication in TestExpressionOptimizer
• [x] 03-28 05:01:36 1ced0b6a2f Drop tables at the end of testScaleWriters test

  Raghav Sethi

• [ ] 03-22 17:16:02 71a8f7c4d4 Change path to root to be separate method

  Rebecca Schlussel

• [x] 03-27 08:11:05 64f1b63d63 Increase max arguments to zip function to 5

  Wenlei Xie

• [x] 03-19 16:32:24 56cd056ca9 Revert "Use appropriate initial state object in generated aggregation code"
• [x] 03-22 13:20:01 e4e97d97a6 Remove unused preGeneratedExpressions variables
• [x] 03-27 22:24:09 71bcf96be7 Add doc for cast between JSON and ROW
• [x] 03-28 10:52:27 dd1fe3710e Support MIN/MAX over UNKNOWN type
• [x] 03-28 12:59:46 50f231df0d Add array_agg test to AbstractTestQueries
• [x] 03-28 12:59:46 ba4540c969 Optimize array_agg with flattened group state

  sr

• [x] 03-19 16:28:29 9778befa0e Minor Cleanup of Histogram and HistogramState

  Łukasz Osipiuk

• [x] 03-22 02:42:43 7fb3e463b1 Add explanation why we are filtering out empty WindowOperatorStats
• [x] 03-22 02:42:43 85692da89e Add basic testExplainAnalyzeVerbose test
• [x] 03-22 02:42:43 94bce3b833 Fix computing averages in WindowOperatorStats
• [x] 03-22 08:23:10 6aeeaf7a5c Accept system properties in RuleTester

[rschlussel]

General

• Increase the maximum number of arguments to zip from 4 to 5.

[nezihyigitbasi]

General

• Add trace token support to the scheduler and exchange HTTP clients. Each HTTP request sent by the scheduler and exchange HTTP clients will have a "trace token" (a unique ID) in their headers, which will be logged in the HTTP request logs. This information can be used to correlate the requests and responses during debugging.

[mbasmanova]

General

• Add ST_IsValid, geometry_invalid_reason, simplify_geometry and great_circle_distance functions
• Optimize the ST_Intersects function
• Optimize serialization of Geometry values
• Add support for spatial joins expressed via ST_Distance function
• Fix spatial joins with additional filters

[wenleix]

General

• Support MIN/MAX aggregation over UNKNOWN type
• Optimize array_agg to avoid excessive object overhead. It also fixes excessive native memory usage with G1 garbage collection.

[dain]

General
-------
* Improve performance of dynamic writer scaling.

SPI
---
* Add performance statistics to plan in ``QueryCompletedEvent``.
* Remove ``Page.getBlocks()``.  This call was rarely used and is performs an expensive copy.  Instead use ``Page.getBlock(channel)`` or the new helper ``Page.appendColumn()``.
* Improve validation ``MapBlock``, ``RowBlock`` and ``ArrayBlock`` during construction.

[haozhun]

General
-------

* Allow omission of step parameter in ``DATE`` variant of :func:`sequence` function
  when the desired step is 1/-1 day.

[kokosing]

General
 * Optimize empty table scan to empty `VALUES`.

JMX Connector
---------------
 * Add wildcard character * which allows to query several MBeans data with single query.

[highker]

Hive

• Fix a bug in the ORC writer that will write incorrect data of type VARCHAR or VARBINARY into files.

[findepi]

General
-------
* Parse decimal literals (e.g. ``42.0``) as ``DECIMAL``. Previously they were parsed as ``DOUBLE``.
* Fix ``current_date`` failure when session time zone had a "gap" at 1970-01-01 00:00:00, e.g. ``America/Bahia_Banderas``.
* Support ``ESCAPE`` for ``LIKE`` predicate in ``SHOW SCHEMAS`` and ``SHOW TABLES`` queries.
* Enable simplification and pushdown of operators on ``DECIMAL``s.

However I think current_date and fixing DECIMAL operators are so small a thing that they don't deserve release note's space. @martint ?

[martint]

The current_date change makes sense to document. What's the "simplification and pushdown of operators" for DECIMAL?

[electrum]

Resource Groups Changes
-----------------------

* Query Queues have been removed. Resource Groups are always enabled. The
  config property ``experimental.resource-groups-enabled`` has been removed.

CLI Changes
-----------

* The ``--enable-authentication`` option has been removed. Kerberos authentication
  is automatically enabled when ``--krb5-remote-service-name`` is specified.
* Kerberos authentication now requires HTTPS.

Hive Changes
------------

* Add support for using `AWS Glue <https://aws.amazon.com/glue/>`_ as the metastore.
  Enable it by setting the ``hive.metastore`` config property to ``glue``.

[findepi]

What's the "simplification and pushdown of operators" for DECIMAL?

@martint certain operations on decimal were registered as nondeterministic (https://github.com/prestodb/presto/commit/7713581f712f0ae3e50a3de7715f2f4f4f06cb51 + https://github.com/prestodb/presto/commit/250aa04e4906ffa8560a9ea0d7a735bf27c43c00). I observed this prevented them from being simplified by ExpressionOptimizer (https://github.com/prestodb/presto/blob/aad321268dcee7228748652ff276d479124e8aba/presto-main/src/main/java/com/facebook/presto/sql/relational/optimizer/ExpressionOptimizer.java#L173), but i thought we consider determinism in other places too.

However, this bullet may be actually wrong, since https://github.com/prestodb/presto/blob/aad321268dcee7228748652ff276d479124e8aba/presto-main/src/main/java/com/facebook/presto/sql/planner/ExpressionInterpreter.java#L771 and https://github.com/prestodb/presto/blob/aad321268dcee7228748652ff276d479124e8aba/presto-main/src/main/java/com/facebook/presto/sql/planner/ExpressionInterpreter.java#L1120 do not check determinism and DeterminismEvaluator checks determinism of function calls only. If indeed so, the bullet should be removed. ExpressionOptimizer misbehavior was actually a regression after Construct proper literal for DECIMAL in LiteralInterpreter.toExpression commit, which is also part of this release.

[sopel39]

General
-------
* Add configuration property ``task.max-local-exchange-buffer-size`` for setting local exchange buffer size
* Improve parallelism of queries with empty grouping set