Release notes for 0.209

dain commented 6 years ago

Andrii Rosa

[x] all checked
08-23 07:26:06 20cffc4ef6 Add wrapped Boolean benchmarks
08-23 07:26:06 a64563ec1f Implement equality comparisons projection benchmark
08-23 08:05:14 37aaf49785 Use local variable instead of dup instruction in InCodeGenerator
08-23 12:04:51 e16cb85cbf Change EQUAL and NOT_EQUAL implementation signatures
08-23 12:04:52 226c9541f5 Implement nullable EQUAL and NOT_EQUAL for ARRAY type
08-23 12:04:52 2afb4c27ab Change equals semantincs for null in MAP
08-23 12:04:52 9f4cd60522 Implement nullable EQUAL and NOT_EQUAL for ROW type
08-23 12:11:05 b6fb1451f8 Support IN predicate for complex type values with nulls
Dain Sundstrom
[x] all checked
08-09 11:30:06 19b272185b Add NoOpTransactionManager for workers
08-09 11:30:06 a627f4230f Extract TransactionManager interface
08-14 19:41:17 de4679af77 Use field names to map field to underlying data for structs in ORC
David Phillips
[x] all checked
08-13 12:40:17 b600533621 Fix formatting in PushAggregationThroughOuterJoin
08-13 12:50:18 9e674fb1f6 Update to Airbase 83
08-13 12:50:18 cf79df3966 Simplify annotations
08-13 12:50:18 dbc25b6343 Add fall through comments for switch statement
08-13 12:50:42 4a4ab61fe0 Move stats calculator binding to separate module
08-13 12:51:37 b7482630c7 Only bind stats calculator on coordinator
08-13 13:01:06 68283b0f67 Cleanup statement resource executors on shutdown
08-13 13:01:06 89dba434b9 Remove unused statement resource thread pools from workers
08-13 13:08:24 5d211f4f3c Cleanup binding of failure detector
08-13 13:24:37 c1287aa36c Only bind discovery server on coordinator
08-19 21:46:50 91524c2351 Update to ASM 6.2.1 and bytecode 1.1
08-20 08:25:27 25d1ecd0c4 Completely remove SHOW PARTITIONS
08-20 10:26:56 a28fcad239 Add workaround for Mac JDK IPv6 bug for CLI and JDBC
08-20 23:00:29 3911f79541 Fix data loss when writing sorted Hive tables
08-20 23:00:29 46a36da31b Cleanup all temporary files when writing sorted Hive tables
08-23 14:18:35 67992400e7 Allow alternate implementations of HiveMetadata
08-24 11:13:42 ec2f4db199 Fix handling of thread interruption in JDBC driver
Gerlou Shyy
[x] all checked
08-13 16:42:42 a06780d815 Add FunctionInvokerProvider for requesting invocation convention
08-13 16:49:39 fcf66b6134 Rename pre-existing FunctionInvoker to InterpretedFunctionInvoker
Grzegorz Kokosiński
[x] all checked
08-08 22:47:28 08e7c37df6 Remove unnecessary copies in PruneUnreferencedOutputs
08-14 22:17:34 ff115ce115 Add Kudu connector
08-18 12:47:49 384ae61c64 Remove CoefficientBasedStatsCalculator
08-20 03:49:08 fbd4cb2647 Simplify assertions in SHOW TABLES test
08-22 04:02:00 d7ccd894f8 Remove TableScanNode originalConstraint
Haozhun Jin
[x] all checked
08-09 17:29:08 8fcb2a393d Add UnknownType.writeBoolean because some logic relies on it
08-09 17:29:08 ca6cf5743e Remove special handling of UNKNOWN in min_by/max_by
08-14 11:07:39 28d830dc2c Rename SqlScalarFunctionBuilder to PolymorphicScalarFunctionBuilder
08-14 11:07:39 738fa51841 Remove unused predicate feature from PolymorphicScalarFunction
08-15 14:44:22 81f28fd75b Move TestBlockAndPositionNullConvention to scalar package
08-15 14:44:23 762a117a17 Fix handling of Object type in annotation-based scalar
08-15 14:44:23 a998082c81 Add constructor dependency sanity check to ParametricScalarImplementation
08-15 14:46:06 99545bf618 Fix delay in sending noMoreSplits for scan node in colocated join
08-16 10:55:20 c12f84e917 Fix EmptySplit handling during bucket-by-bucket execution
08-18 15:43:42 8ccfcacfbe Minor refactor in SqlTaskExecution
08-18 15:43:42 b35d611adc Improve abstraction of SourcePartitionedScheduler
08-18 15:43:42 e1fbcb01f5 Remove unused method in SourcePartitionedScheduler
08-19 00:12:46 f0f2730783 Determine grouped execution on an individual scan node basis
08-19 00:12:47 020b1ce004 Reduce method overloads in PlanPrinter
08-19 00:12:47 beee9e848f Print grouped execution decision for individual Scan node in EXPLAIN
08-20 20:46:40 a9ace9e65f Fix truncated print for ProjectionNode in PlanPrinter
Jiexi Lin
[x] all checked
08-24 15:31:14 7546f9198c Add per table column_ranges system table in Raptor
Karol Sobczak
[x] all checked
08-09 05:50:34 5053352096 Remove redundent is decimal check
08-09 05:50:34 92f05c6be9 Simplify decimal type test
08-17 08:27:26 20d222e2e9 Add WorkProcessor#flatten method
08-17 08:27:26 8dc337ebaa Add Transformation#process javadoc
08-17 08:27:26 8e1852839d Remove redundant "else"
08-21 02:12:55 2696592f92 Better estimate NDVs and range in subtractColumnStats
08-21 02:12:55 3834510653 Use lambda instead of expression
08-21 02:12:55 6c0c84bbf6 Make tests stats fields in TestFilterStatsCalculator
08-21 02:12:55 707871aa58 Prevent nulls fraction to be negative in subtractColumnStats
08-21 02:12:55 8d004628cc Mark subtractColumnStats as @Deprecated as it semantics is undefined
08-21 08:17:12 1dc4ecc122 Dispose resources in WorkProcessorUtils when they are no longer needed
08-21 08:17:12 e1ab5bf7d5 Add WorkProcessor#transformProcessor method
Maria Basmanova
[x] all checked
08-09 08:13:05 c8ebb19972 Add the geometry_union_agg spatial aggregation function
08-10 10:49:51 2c6ba08e37 Check MarkDistinctSymbols#distinctSymbols is not empty
08-10 10:49:51 575a9a6be5 Remove unnecessary remote exchange from scalar correlated subquery plan
08-15 17:29:50 731eb9a97c Add convex_hull_agg function
08-16 08:27:46 dbe03b43bd Document geometry_union_agg function
08-16 13:21:40 cff67783a7 Move BenchmarkSpatialJoin to presto-geospatial
08-16 16:48:33 7ec5210b77 Add bing_tiles_around variant taking radius in kilometers
Martin Traverso
[x] all checked
08-13 11:51:58 662af7a9f1 Remove explicit grouping sets list from aggregation node
08-13 12:06:20 4bafb48856 Parse grouping columns as generic expressions
08-13 17:10:01 6ae9e8cf4a Compute covariance, correlation and regression in numerically stable manner
08-17 14:19:46 f9de0bd032 Fix incorrect predicate pushdown through empty grouping set
08-22 11:28:36 14f2ee4e7d Tighten assertion for parsing failure in TestSqlEnvironmentConfig
08-22 11:28:36 20f5ec2963 Rename singleExpression to standaloneExpression
08-22 11:28:36 30d455ec98 Implement recursion-free ATN simulator
08-22 11:28:36 8e04402761 Log raised during error handling
08-22 11:28:36 999e0aa185 Improve parsing error message
08-22 11:28:36 f18d0e9c5c Handle end of stream at the top-level rule
08-22 11:28:36 f5790cec97 Update to antlr 4.7.1
08-22 11:28:36 f95ed89a10 Use EOF-terminated rule for path specification
Nezih Yigitbasi
[x] all checked
08-09 10:17:52 10efb40e09 Don't poll/update the coordinator memory pools if not necessary
08-09 11:33:46 d47b9dc24b Move tagged memory allocations on memory pool update
08-09 14:04:19 7c1842671b Make moveQuery package private
08-09 14:04:19 e2e98ebbb3 Update tagged memory allocation API to make tags required
08-10 14:54:37 c25ab366bd Fix the race between close and destroy in the Driver
08-10 14:55:22 7bb2f0d6d3 Fix information schema predicate pushdown for views
08-10 17:08:58 70abfd42bc Fix live plan and stage kill button URL
08-10 17:10:59 88e4c1e935 Fix various typos
08-16 14:57:35 049fdce8d5 Add MockConnectorFactory.Builder to simplify construction
08-16 15:52:26 b4bbeedac0 Update docs with new memory-related configs
Piotr Findeisen
[x] all checked
08-11 12:55:45 005bc1fbdc Remove duplicate method
08-11 13:09:18 0895e77536 Report SemanticException for invalid TIME literal
08-13 23:49:24 1f2182c4a1 Update documentation about link to query's JSON
08-16 07:39:35 0964662615 Use Optionals in KafkaTopicDescription
08-16 07:39:35 54df0a482e Add basic Avro decoder for Kafka
08-16 07:39:35 9045ce4ce0 Add data schema to the decoder params
08-16 07:39:35 b93d25ad89 Add tests for Avro schema evolution in Kafka connector
08-16 07:39:35 cbdd36f1a3 Add documentation for Avro support in Kafka connector
08-16 07:39:35 dcb6dd83f7 Add TestingEnvironment
08-16 07:39:35 edabfc58ad Add Avro decoder to Redis
08-16 07:39:35 ff38359a1a Add support for structural types in Avro decoder for Kafka
08-20 00:27:52 656c2fcd3e Respect X-Forwarded-Proto header in / endpoint
08-22 00:27:10 687baebdd5 Ensure access control classes implement all methods
08-23 03:27:18 bac5959f4f Make HiveClientConfig.resourceConfigFiles non-nullable
08-27 02:08:15 abd65e37cc Replace usage of deprecated TreeTraverser
Raghav Sethi
[x] all checked
08-21 17:23:33 afe04dfba3 Add ManagedQueryExecution as root execution interface
08-21 18:07:14 59d297913f Fix error message in QuerySessionSupplier
08-24 16:17:43 e8f962d304 Separate EXCEEDED_MEMORY_LIMIT error into local and global
Rebecca Schlussel
[x] all checked
08-22 13:55:45 65ea3ca9d7 Don't check column access for aliases
08-22 13:55:45 f79aa806cd Track origin column for fields
08-27 07:51:19 0d20887425 Simplify formatFragment in PlanPrinter
08-27 07:51:19 16ab0c36e4 Show cost estimates in QueryMonitor
08-27 07:51:19 1b47fb3df7 Add exchange type as a field to RemoteSourceNode
08-27 07:51:19 789753c218 Propagate stats/costs across fragments in EXPLAIN
Shixuan Fan
[x] all checked
08-14 16:16:05 60de647d8d Implement EXPLAIN (TYPE IO, FORMAT JSON) statement
08-15 14:12:50 b813033e0a Use catalog field instead of hardcoded catalog
Wenlei Xie
[x] all checked
08-10 16:06:51 b3d9e15503 Log queryId when a small stripe get MAX_BYTES flush
08-17 09:35:45 f8ee82d868 Add block compact test cases
08-22 11:40:03 cdd2f673c2 Raise requried Java version to 8u151
08-22 14:27:05 05b66c0419 Avoid flushing small stripes due to DICTIONARY_FULL
08-22 14:27:05 40af0e803c Log queryId when a small stripe get DICTIONARY_FULL flush
08-23 18:11:13 c92cfef154 Add Hive connector procedure call to create empty partition
08-24 15:34:37 51ebbd09b8 Fix Hive smoke test when running in parallel
08-26 17:28:34 47355c83b3 Fix test failures due to ExceededMemoryLimitException message
08-26 22:20:11 900b011531 Fix more test failures due to ExceededMemoryLimitException message
yingsu00
[x] all checked
08-13 12:10:44 21841870e3 Refactor BooleanInputStream
08-13 12:27:03 97552478c9 Add memory tracking for StreamReader local buffers

tdcmeehan commented 6 years ago

General
--------
* Add the geometry_union_agg spatial aggregation function to union the input geometries.

mbasmanova commented 6 years ago

General
-------
* Optimized scalar correlated subquery to eliminate unnecessary data shuffle

nezihyigitbasi commented 6 years ago

Web UI Changes
--------------
* Fix the kill query button in the live plan and stage performance UIs.

martint commented 6 years ago

* Improve numerical stability for :func:`covar_samp`, :func:`corr`, :func:`regr_slope` and :func:`regr_intercept`.

shixuan-fan commented 6 years ago

General
-------
* Support EXPLAIN (TYPE IO, FORMAT JSON) that shows input tables with constraints and output table in JSON format.

kokosing commented 6 years ago

Kudu Connector
----------------
 * Introduce Apache Kudu Connector

yuyongyang800 commented 6 years ago

General
---------
 * Add the convex_hull_agg spatial aggregation function to compute the convex hull of the input geometries.

anusudarsan commented 6 years ago

Kafka connector
----------------
* Support for Avro formatted Kafka messages.
* Support for backward compatible Avro schema evolution.

https://github.com/prestodb/presto/pull/11108

avinashidnani commented 6 years ago

General
---------
 * Add bing_tiles_around variant taking radius in kilometers

martint commented 6 years ago

* Fix incorrect predicate pushdown when grouping sets contain the empty grouping set (#11296).

findepi commented 6 years ago

* Fix ``X-Forwarded-Proto`` header handling for ``/`` path

https://github.com/prestodb/presto/pull/11290

shreyasjoshis commented 6 years ago

Just want to make sure this isn't missed:

Hive
----
* ORC struct columns are now mapped by name rather than ordinal. This correctly handles missing/extra struct fields in the ORC file.

cc @dain #11001

kokosing commented 6 years ago

General
--------
 * Remove table scan original constraint information from EXPLAIN queries

wenleix commented 6 years ago

General

Raise required Java version to 8u151. This avoids correctness issues for map to map cast when running under some lower version of JVM (e.g 8u92)

Hive

Add Hive connector procedure call to create empty partition
Reduces the number of cases where tiny ORC stripes will be written when some columns are highly dictionary compressed.

rschlussel commented 6 years ago


General
-------
* Fix AllowAllAccessControl and FileBasedAccessControl to default to allowing access for `checkCanCreateSchema`, `checkCanDropSchema`, and `checkCanRenameSchema`.
* Fix a bug where we would send column aliases in the list of columns whose access permissions we were checking.

raghavsethi commented 6 years ago

General

Change exceeded memory limit exceptions to have distinct error codes for global and per-node limits.

rschlussel commented 6 years ago

* Print statistic and cost estimates in EXPLAIN (TYPE DISTRIBUTED) and EXPLAIN ANALYZE plans

arhimondr commented 6 years ago

General
-------
* Support ROW/ARRAY/MAP values with NULLs in EQUAL and NOT_EQUAL operators

jessesleeping commented 6 years ago

Raptor
--------
* Every Raptor table now has a hidden system table which contains min/max values of every 
BIGINT, DATE, TIMESTAMP and BOOLEAN column. A table named `example` will have a 
column range system table named `example$column_ranges`.

nezihyigitbasi commented 6 years ago

Web UI Changes
--------------
* Fix kill query button in live plan and stage UI.

sopel39 commented 6 years ago

General
-------
* Improve statistics estimation for expressions that involve NOT/OR
* Fix potential negative nulls fraction estimates for expressions that involve NOT/OR

yingsu00 commented 6 years ago

Hive
-------
* Improve memory accounting when reading ORC files. Previously, local buffers 
and object overhead of StreamReader was not tracked.

haozhun commented 6 years ago


* Fix regression in 0.208 that results in execution failure when
  at least one of the arguments to :func:`min_by` and :func:`max_by`
  is a constant null.
* Fix execution failure of queries using bucket-by-bucket execution when
  some buckets are completely filtered out.
* Fix execution failure of queries due to planning deficiency involving
  complex nested joins where a join that isn't eligible for bucket-by-bucket execution
  feeds into the build side of a join that is eligible.

SPI
* Allow using Object as parameter type and return type of SQL functions
  when the correponding SQL type is an unbounded generic.
  Previously, Object can be used only in limited circumstances, and wasn't intentional.

prestodb / presto