Release notes for 0.208

[raghavsethi]

Andrii Rosa

• [x] all checked
• 08-02 05:54:03 03e2eb147d Collect column statistics on table write: Hive Connector
• 08-02 05:54:03 05fbe72b47 Implement AutoCloseableCloser
• 08-02 05:54:03 123f735241 Replace supportsColumnStatistics with getSupportedColumnStatistics
• 08-02 05:54:03 561d60cae8 Collect column statistics on table write: Planner
• 08-02 05:54:03 6df8ff2a71 Collect column statistics on table write: SPI
• 08-02 05:54:03 bfe113cb94 Collect column statistics on table write: Execution
• 08-02 05:54:04 34940483e7 Collect column statistics on table write: Product Tests
• 08-02 05:54:04 9dcf44d55b Collect column statistics on table write: Documentation
• 08-02 05:54:04 b6158fbc2f Collect column statistics on table write: Smoke Tests
• 08-02 05:54:04 c2675616e6 Add information about collected statistcs to explain output
• 08-02 05:54:04 eb6a32ce00 Add properties for column statistics collect
• 08-02 08:08:57 122015c7c3 Pull dataSize statistics from Hive for VARCHAR columns
• 08-02 08:08:57 212cd5c69a Fix assertion messages in Tpch, Tpcds stat tests
• 08-02 08:08:57 5817e55193 Check value is known as early as possible
• 08-02 08:08:57 899d6a608b Remove unnecessary condition
• 08-02 08:08:57 9011742cb8 Extract rowsPerPartition calculation
• 08-02 08:08:57 b69e78b6ef Update TPC-DS statistics with data size
• 08-02 08:08:57 bd4b70f0cd Add statistics test for table partitioned by varchar column
• 08-02 08:08:57 d694edc99e Remove redundant else
• 08-02 08:08:57 d817709ea3 Inline StatisticsEstimator#addPartitionStats's overload
• 08-02 08:08:57 daff49aecc Extract variable
• 08-02 08:08:57 e907dbe391 Fix NATION_PARTITIONED_BY_REGIONKEY data files
• 08-02 08:08:57 f673abd13a Rename parameter to match meaning
• 08-02 08:08:57 fc5419dab5 Update TPC-H statistics with data size
• 08-02 16:37:06 175ad301ad Fix SingleRowBlock#getSizeInBytes
• 08-02 16:37:06 2f90d7a316 Support hidden aggregation functions
• 08-02 16:37:06 47e208122d Add internal aggregations over estimated data size for stats
• 08-02 16:37:06 4cc73a1931 Implement Block#getEstimatedDataSizeForStats
• 08-06 07:10:51 ab23cd1d31 Add additional collectibe statistics
• 08-06 07:56:03 0f02316f3d Refactor HiveColumnStatistics
• 08-06 07:56:03 9b31a8c639 Collect column data size statistics in Hive connector

  Dain Sundstrom

• [x] all checked
• 08-03 16:41:38 68670b163b correcting documentation for percent_rank window function return type

  David Phillips

• [x] all checked
• 08-02 08:08:08 dec7153ca4 Remove size limit for writing Hive bucketed sorted tables
• 08-02 13:02:23 f93d5e5c1d Update to Airbase 82
• 08-03 09:01:57 11f35c48e6 Add Hadoop native libraries for Linux ppc64le
• 08-03 09:01:57 e9f01034a8 Refactor system requirement checks into separate methods
• 08-03 09:01:57 f04d2bbb74 Allow running Presto on Linux ppc64le
• 08-04 11:59:38 2d43bd8760 Report physical written bytes for Parquet writer
• 08-05 15:14:08 d3642ce131 Only bind plan optimizers on coordinator
• 08-06 09:37:52 85013d8a13 Add missing word in "too many stages" error code

  Gerlou Shyy

• [x] all checked
• 08-02 11:23:36 0284f2538b Add BLOCK_AND_POSITION calling convention to annotation framework
• 08-02 11:23:36 20a3276321 Rename ScalarImplementation to ParametricScalarImplementation
• 08-02 11:23:36 91e535db7d Fix specializedTypeParameters in ParametricScalarImplementation
• 08-02 11:23:36 d6f7548c24 Move @BlockPosition and @BlockIndex to SPI

  Grzegorz Kokosiński

• [x] all checked
• 08-02 00:36:48 93466634ab Remove unnecessary Lookup::resolve usage
• 08-02 00:37:52 19e423b370 Add support for setting default cli history path with an env variable
• 08-02 00:37:52 7acd5f2007 Static import isNullOrEmpty
• 08-02 00:38:07 f0c3b48a8d Expose debug port for Hadoop processes in product tests
• 08-02 00:38:19 f8eb7a474c Enable history for presto-cli.sh
• 08-02 00:39:19 c57bbb8237 Optimizing DependenciesChecker for TableScanNode
• 08-02 00:41:54 95440257d3 Optimizing DependenciesChecker for validating JoinNode
• 08-02 00:44:57 4af49fea49 Aggregation with probe symbols should not be pushed through outer join
• 08-03 01:56:11 bf7c988644 Migrate TransformCorrelatedSingleRowSubqueryToProject to rule
• 08-03 01:56:11 fc40f70eba Skip correlated projections in during single row subquery rewrite

  Haozhun Jin

• [x] all checked
• 08-02 13:06:12 a2c6c129fd Skip currentConstraint field when serializing TableScanNode
• 08-02 13:24:39 2730762662 Limit number of stages in a query
• 08-02 13:24:39 73d94c294e Reduce LocalQueryRunner constructor overload
• 08-02 13:24:39 7b33a46747 Make PlanFragmenter.createSubPlans not static
• 08-02 13:24:39 9899432eaf Add session property to ease rollout of stage count limit

  Jiexi Lin

• [x] all checked
• 08-02 10:04:38 36e701b567 Append original error message to thrift connector remote exception
• 08-03 14:21:09 c49cf2a9b7 Change java type of UNKNOWN from void to boolean

  Maria Basmanova

• [x] all checked
• 08-03 11:16:31 a669b1137a Upgrade to esri-geometry-api version 2.2.0

  Martin Traverso

• [x] all checked
• 08-02 11:39:46 5315bc5e6b Add SpookyHashV2 hash functions
• 08-06 09:32:30 1fc8c1be03 Preserve the name of column after unnest

  Nezih Yigitbasi

• [x] all checked
• 08-02 10:27:08 099f0e8923 Fix cli history initialization
• 08-02 10:27:57 1b823cb72f Remove unused SqlTaskManagerStats class
• 08-02 10:27:57 885c8913c7 Remove unused fields/parameters
• 08-02 10:27:57 9b46a053a6 Fix incorrect null check against optional
• 08-02 10:28:17 399ff7e4b3 Update units to 1.3
• 08-02 10:29:06 19d3d1c1b2 Make ClusterMemoryManager resilient to memory accounting leaks
• 08-02 10:29:06 3cf24e9e9b Rename queries to runningQueries in ClusterMemoryManager
• 08-02 10:29:06 4e100444ef Fix thread-safety of ClusterMemoryPool
• 08-02 10:29:06 6c54770f1f Minor refactor of ClusterMemoryManager
• 08-02 10:29:06 a0406bf04c Mark QueryStats::getEndTime() as Nullable
• 08-02 21:14:53 cca44dd05a Log the leaked queries at DEBUG level
• 08-03 14:07:21 509cdc2efa Make sure to close scheduled executor in various tests
• 08-03 22:43:18 33b6df19c8 Static import TaskStates
• 08-03 22:43:18 b520111c71 Add a JMX counter for failed tasks
• 08-06 13:32:49 2ac73e1047 Fix bottleneck in InternalResourceGroup::internalGenerateCpuQuota
• 08-06 13:39:19 40ad1c4891 Fix javadoc
• 08-06 13:39:19 53aceb04a7 Simplify Optional handling in LocalExecutionPlanner
• 08-06 13:39:19 60fb653e05 Remove unnecessary getClass call
• 08-06 13:39:19 80c0f03c7f Remove unused field
• 08-06 13:39:19 bc25fb75a3 Simplify expression with a method reference
• 08-06 13:39:19 dea856d24d Mark fields final
• 08-06 13:39:19 ec7ddf1be8 Remove unnecessary type parameter
• 08-06 13:39:19 f7b38582bf Simplify collection construction
• 08-06 13:39:19 fc2a66f2bb Replace statement lambda with expression lambda
• 08-06 19:54:59 53b7dce55b Make sure to free output buffer memory on task completion

  Wenlei Xie

• [x] all checked
• 08-02 16:13:13 50ed39f50f Cleanup legacy handling for TRY
• 08-06 18:11:30 9c0fe41281 Improve Stripe Size Estimation for SliceDictionaryColumnWriter

  Yi He

• [x] all checked
• 08-02 11:00:29 aa38ec45a0 Fix alter table drop column in raptor

  yingsu00

• [x] all checked
• 08-02 12:59:41 6dcd67fdff Inline only simple expressions in predicate pushdown
• 08-02 14:33:51 9c45d9a2c6 Grow the batch size fro OrcRecordReader from 1 row

[martint]

* Add :func:`spooky_hash_v2_32` and :func:`spooky_hash_v2_64` functions.

[martint]

* Preserve field names when unnesting columns of type ``ROW``.

[nezihyigitbasi]

General
-------
* Fix a bug that could cause ``PartitionedOutputOperator``s to hold onto significant amount of memory eventually causing full garbage collection pauses and out of memory exceptions.
* Add a cluster memory leak detector that logs the queries that have possibly leaked (e.g., has accounted for memory usage incorrectly) memory on some worker(s). 
* Make the cluster out of memory killer more resilient to memory accounting leaks. Previously, memory accounting leaks on the workers could effectively disable the out of memory killer.

CLI Changes
-----------
* Fix the creation of the cli history file on fresh installations. 

[wenleix]

Hive Connector Changes
----------------
* Improve stripe size estimation for the optimized ORC writer. This reduces the number of cases where tiny ORC stripes will be written.

[tdcmeehan]

Please add:

a669b1137a16a3727d9e14420b048ed5ccc53376

Update esri-geometry-api to version 2.2.0 (from 2.1.0)

[arhimondr]

Hive Connector Changes
----------------
* Collect column level statistics on table write (INSERT, CREATE TABLE AS SELECT). Disabled by default. Can be enabled by setting the "hive.collect-column-statistics-on-write" connector property.
* Pull data size statistics from the metastore for VARCHAR, VARBINARY and CHAR columns

[haozhun]

General

* Limit the number of stages in a query to 100. The limit can be adjusted with 
  ``query.max-stage-count`` config and ``query_max_stage_count`` session property.

[electrum]

Thrift Connector Changes
------------------------

* Include error message from remote server in remote exceptions.

[electrum]

General Changes
---------------

* Add experimental support for running on Linux ppc64le.

Hive Connector Changes
----------------------

* Remove size limit for writing bucketed sorted tables.
* Allow using writer scaling with Parquet.

[kokosing]

General
---------
 * Improve planning time for queries over tables with high number of columns.
 * Extend support for correlated subqueries, including single row correlated subquery with complex expression over a correlated column reference.
 * Fix aggregation push down through left outer join optimization for aggregations over columns from the left side join relation.