[meta] 7.6 release

[russcam]

Preliminary release notes for 7.6.0

(Generated from asciidoc release notes on 2020-02-03)

Breaking changes

Mapping

• Add a cluster setting to disallow loading fielddata on _id field #49166 (issues: #26472, #43599)

Breaking Java changes

Security

• Support Client and RoleMapping in custom Realms #50534 (issue: #48369)

Deprecations

Analysis

• Deprecate and remove camel-case nGram and edgeNGram tokenizers #50862 (issue: #50561)

Authorization

• Deprecating kibana_user and kibana_dashboard_only_user roles #46456

Distributed

• Deprecate synced flush #50835 (issue: #50776)

• Deprecate indices without soft-deletes #50502

Features/Indices APIs

• Emit warnings when index templates have multiple mappings #50982

Infra/Core

• Deprecate the local parameter of /_cat/nodes #50499 (issue: #50088)

MULTIPLE AREA LABELS

• Ensure we emit a warning when using the deprecated template field. #50831 (issue: #49460)

• Add a deprecation warning regarding allocation awareness in search request #48351 (issue: #43453)

Reindex

• Deprecate sorting in reindex #49458 (issue: #47567)

Search

• Update the signature of vector script functions. #48604

• Deprecate the sparse_vector field type. #48315

New features

Aggregations

• New Histogram field mapper that supports percentiles aggregations. #48580 (issue: #48578)

• Implement stats aggregation for string terms #47468

Analysis

• Implement Lucene EstonianAnalyzer, Stemmer #49149 (issue: #48895)

Features/ILM+SLM

• ILM action to wait for SLM policy execution #50454 (issue: #45067)

• Add ILM histore store index #50287 (issue: #49180)

Features/Ingest

• CSV processor #49509 (issue: #49113)

MULTIPLE AREA LABELS

• Password Protected Keystore (Feature Branch) #49210

Machine Learning

• Implement precision and recall metrics for classification evaluation #49671 (issue: #48759)

• [ML] Explain data frame analytics API #49455

• [ML] ML Model Inference Ingest Processor #49052

• Implement accuracy metric for multiclass classification #47772 (issue: #48759)

Mapping

• Add per-field metadata. #49419 (issue: #33267)

Search

• Add fuzzy intervals source #49762 (issue: #49595)

• Add a listener to track the progress of a search request locally #49471 (issue: #49091)

Enhancements

Aggregations

• Add reusable HistogramValue object #49799 (issue: #49683)

• Optimize composite aggregation based on index sorting #48399 (issue: #48130)

Allocation

• Auto-expand indices according to allocation filtering rules #48974

• Do not cancel ongoing recovery for noop copy on broken node #48265 (issue: #47974)

• Quieter logging from the DiskThresholdMonitor #48115 (issue: #48038)

• Faster access to INITIALIZING/RELOCATING shards #47817 (issues: #46941, #48579)

Analysis

• Check for deprecations when analyzers are built #50908 (issue: #42349)

• Make Multiplexer inherit filter chains analysis mode #50662 (issue: #50554)

• Allow custom characters in token_chars of ngram tokenizers #49250 (issue: #25894)

Authentication

• Add Debug/Trace logging for authentication #49575 (issue: #49473)

Authorization

• Increase Size and lower TTL on DLS BitSet Cache #50535 (issues: #43669, #49260)

• Add monitor_snapshot cluster privilege #50489 (issue: #50210)

• Remove reserved roles for code search #50068 (issue: #49842)

• [Code] Remove code_admin/code_user roles #48164

• Resolve the role query and the number of docs lazily #48036

CCR

• Improve error message when pausing index #48915

• Use MultiFileTransfer in CCR remote recovery #44514 (issue: #44468)

CRUD

• print id detail when id is too long. #49433

• Add preflight check to dynamic mapping updates #48817 (issue: #35564)

Cluster Coordination

• Move metadata storage to Lucene #50907 (issue: #48701)

• Remove custom metadata tool #50813 (issue: #48701)

Distributed

• Use retention lease in peer recovery of closed indices #48430 (issue: #45136)

Engine

• Do not force refresh when write indexing buffer #50769

• Deleted docs disregarded for if_seq_no check #50526

• Allow realtime get to read from translog #48843

• Do not warm up searcher in engine constructor #48605 (issue: #47186)

• Add a new merge policy that interleaves old and new segments on force merge #48533 (issue: #37043)

• Refresh should not acquire readLock #48414 (issue: #47186)

Features/ILM+SLM

• Refresh cached phase policy definition if possible on new poli… #50820 (issue: #48431)

• Make the UpdateRolloverLifecycleDateStep retryable #50702 (issue: #48183)

• Make InitializePolicyContextStep retryable #50685 (issue: #48183)

• ILM retryable async action steps #50522 (issues: #44135, #48183)

• Make the TransportRolloverAction execute in one cluster state update #50388

• ILM open/close steps are noop if idx is open/close #48614

• ILM Make the check-rollover-ready step retryable #48256 (issue: #44135)

Features/Ingest

• Foreach processor - fork recursive call #50514

• Sync grok patterns with logstash patterns #50381

• Replace required pipeline with final pipeline #49470 (issue: #49247)

• Add templating support to enrich processor #49093

• Introduce on_failure_pipeline ingest metadata inside on_failure block #49076 (issue: #44920)

• Add templating support to pipeline processor. #49030 (issue: #39955)

• Add option to split processor for preserving trailing empty fields #48664 (issue: #48498)

• Change grok watch dog to be Matcher based instead of thread based. #48346 (issues: #43673, #47374)

• update ingest-user-agent regexes.yml #47807

Features/Java High Level REST Client

• Add remote info to the HLRC #49657 (issue: #47678)

• Add delete alias to the HLRC #48819 (issue: #47678)

Features/Monitoring

• Significantly Lower Monitoring HttpExport Memory Footprint #48854

• Validate proxy base path at parse time #47912 (issue: #47711)

• Validate index name time format setting at parse time #47911 (issue: #47711)

• Validate monitoring header overrides at parse time #47848 (issue: #47711)

• Validate monitoring username at parse time #47821 (issue: #47711)

• Validate monitoring password at parse time #47740 (issue: #47711)

Features/Watcher

• Log attachment generation failures #50080

• Don’t dump a stacktrace for invalid patterns when executing elasticse… #49744 (issue: #49642)

Geo

• "CONTAINS" support for BKD-backed geo_shape and shape fields #50141 (issue: #41204)

• Adds support for geo-bounds filtering in geogrid aggregations #50002

• Introduce faster approximate sinh/atan math functions #49009 (issue: #41166)

• Add IndexOrDocValuesQuery to GeoPolygonQueryBuilder #48449

Infra/Core

• Add "did you mean" to ObjectParser #50938

• Consistent case in CLI option descriptions #49635

• Improve resiliency to formatting JSON in server #48553 (issue: #48450)

• Don’t close stderr under --quiet #47208 (issue: #46900)

Infra/Packaging

• Respect ES_PATH_CONF on package install #50158

• Restrict support for CMS to pre-JDK 14 #49123 (issue: #46973)

• Remove parsed JVM settings from general settings in Windows service daemon manager #49061 (issue: #48796)

• Package the JDK into jdk.app on macOS #48765

• Add UBI-based Docker images #48710 (issue: #48429)

Infra/Plugins

• Report progress of multiple plugin installs #51001 (issue: #50924)

• Allow installing multiple plugins as a transaction #50924 (issue: #50443)

Infra/Scripting

• Scripting: ScriptFactory not required by compile #50344 (issue: #49466)

• Scripting: Cache script results if deterministic #50106 (issue: #49466)

• Scripting: Groundwork for caching script results #49895 (issue: #49466)

• Scripting: add available languages & contexts API #49652 (issue: #49463)

• Scripting: fill in get contexts REST API #48319 (issue: #47411)

• Scripting: get context names REST API #48026 (issue: #47411)

Infra/Settings

• Add parameter to make sure that log of updating IndexSetting be more detailed #49969 (issue: #49818)

• Enable dependent settings values to be validated #49942

• Do not reference values for filtered settings #48066

License

• Add max_resource_units to enterprise license #50735

• Add setting to restrict license types #49418 (issue: #48508)

• Support "enterprise" license types #49223 (issue: #48510)

MULTIPLE AREA LABELS

• Add ingest info to Cluster Stats #48485 (issue: #46146)

Machine Learning

• [ML] Add audit warning for 1000 categories found early in job #51146 (issue: #50749)

• [ML] Add num_top_feature_importance_values param to regression and classi… #50914

• [ML] Implement force deleting a data frame analytics job #50553 (issue: #48124)

• [ML] Delete unused data frame analytics state #50243

• Make each analysis report desired field mappings to be copied #50219 (issue: #50119)

• [ML] retry bulk indexing of state docs #50149 (issue: #50143)

• [ML] Persist/restore state for DFA classification #50040

• [ML] Introduce randomize_seed setting for regression and classification #49990

• Pass prediction_field_type to C++ analytics process #49861 (issue: #49796)

• [ML] Add optional source filtering during data frame reindexing #49690 (issue: #49531)

• [ML] Add default categorization analyzer definition to ML info #49545

• [ML] Add graceful retry for anomaly detector result indexing failures #49508 (issue: #45711)

• Lower minimum model memory limit value from 1MB to 1kB. #49227 (issue: #49168)

• Throw an exception when memory usage estimation endpoint encounters empty data frame. #49143 (issue: #49140)

• Change format of MulticlassConfusionMatrix result to be more self-explanatory #48174 (issue: #46735)

• Make num_top_classes parameter’s default value equal to 2 #48119 (issue: #46735)

• [ML] Improve model_memory_limit UX for data frame analytics jobs [ISSUE] #44699

Mapping

• Add telemetry for flattened fields. #48972

Network

• Add certutil http command #49827

• Do not load SSLService in plugin contructor #49667 (issue: #44536)

• Netty4: switch to composite cumulator #49478

• Add the simple strategy to cluster settings #49414 (issue: #49067)

• Deprecate misconfigured SSL server config #49280 (issue: #45892)

• Improved diagnostics for TLS trust failures #48911

Percolator

• Refactor percolator’s QueryAnalyzer to use QueryVisitors #49238 (issue: #45639)

Ranking

• Support search_type in Rank Evaluation API #48542 (issue: #48503)

Recovery

• Use peer recovery retention leases for indices without soft-deletes #50351 (issues: #45136, #46959)

• Recovery buffer size 16B smaller #50100

Reindex

• Reindex sort deprecation warning take 2 #49855 (issue: #49458)

SQL

• SQL: Add TRUNC alias for TRUNCATE #49571 (issue: #41195)

• SQL: binary communication implementation for drivers and the CLI #48261 (issue: #47785)

Search

• Add Validation for maxQueryTerms to be greater than 0 for MoreLikeThisQuery #49966 (issue: #49927)

• Optimize numeric sort on match_all queries #49717 (issue: #48804)

• Pre-sort shards based on the max/min value of the primary sort field #49092 (issue: #49091)

• Optimize sort on long field #48804

• Search optimisation - add canMatch early aborts for queries on "_index" field #48681 (issue: #48473)

• #48475 Pure disjunctions should rewrite to a MatchNoneQueryBuilder #48557

• Disable caching when queries are profiled #48195 (issue: #33298)

• BlendedTermQuery’s equals method should consider boosts #48193 (issue: #48184)

• Increase the number of vector dims to 2048 #46895

Security

• Make .async-search-* a restricted namespace #50294

• Security should not reload files that haven’t changed #50207 (issue: #50063)

Snapshot/Restore

• Use Cluster State to Track Repository Generation #49729 (issue: #49060)

• Track Repository Gen. in BlobStoreRepository #48944 (issues: #38941, #47520, #47834, #49048)

• Restore from Individual Shard Snapshot Files in Parallel #48110 (issue: #42791)

• Track Shard-Snapshot Index Generation at Repository Root #46250 (issues: #38941, #45736)

Store

• mmap dim files in HybridDirectory #49272 (issue: #48509)

Transform

• [Transform] Improve force stop robustness in case of an error #51072

• [Transform] add actual timeout in message #50140

• [Transform] automatic deletion of old checkpoints #49496

• [Transform] improve error handling of script errors #48887 (issue: #48467)

• [ML][Transforms] add wait_for_checkpoint flag to stop #47935 (issue: #45293)

Bug fixes

Aggregations

• Fix format problem in composite of unmapped #50869 (issue: #50600)

• SingleBucket aggs need to reduce their bucket’s pipelines first #50103 (issue: #50054)

• Avoid precision loss in DocValueFormat.RAW#parseLong #49063 (issue: #38692)

• Fix ignoring missing values in min/max aggregations #48970 (issue: #48905)

Allocation

• Collect shard sizes for closed indices #50645 (issue: #33888)

• Auto-expand replicated closed indices #48973

• Ignore dangling indices created in newer versions #48652 (issue: #34264)

• Handle negative free disk space in deciders #48392 (issue: #48380)

Analysis

• Throw Error on deprecated nGram and edgeNGram custom filters #50376 (issue: #50360)

• _analyze api does not correctly use normalizers when specified #48866 (issue: #48650)

Audit

• Audit log filter and marker #45456 (issue: #47251)

Authentication

• Don’t fallback to anonymous for tokens/apikeys #51042 (issue: #50171)

• Populate User metadata with OpenIDConnect collections #50521 (issue: #50250)

• Always return 401 for not valid tokens #49736 (issue: #38866)

• Fix iterate-from-1 bug in smart realm order #49473

• Remove unnecessary details logged for OIDC #48746

• Add owner flag parameter to the rest spec #48500 (issue: #48499)

Authorization

• Fix memory leak in DLS bitset cache #50635 (issue: #49261)

• Validate field permissions when creating a role #50212 (issues: #46275, #48108)

• Validate field permissions when creating a role #48108 (issue: #46275)

CCR

• CCR should auto-retry rejected execution exceptions #49213

CRUD

• Block too many concurrent mapping updates #51038 (issue: #50670)

• Ensure meta and document field maps are never null in GetResult #50112 (issue: #48215)

• Replicate write actions before fsyncing them #49746

• Do not mutate request on scripted upsert #49578 (issue: #48670)

• Fix Transport Stopped Exception #48930 (issue: #42612)

• Return consistent source in updates #48707

• Close query cache on index service creation failure #48230 (issue: #48186)

Cluster Coordination

• Import replicated closed dangling indices #50649

• Ignore metadata of deleted indices at start #48918

• Make elasticsearch-node tools custom metadata-aware #48390

Discovery-Plugins

• Make EC2 Discovery Cache Empty Seed Hosts List #50607 (issue: #50550)

• Make EC2 Discovery Plugin Retry Requests #50550 (issue: #50462)

Distributed

• Closed shard should never open new engine #47186 (issues: #45263, #47060)

Engine

• Account trimAboveSeqNo in committed translog generation #50205 (issue: #49970)

• Greedily advance safe commit on new global checkpoint #48559 (issue: #48532)

• Do not ignore exception when trim unreferenced readers #48470

Features/Features

• Fix X-Pack SchedulerEngine Shutdown #48951

Features/ILM+SLM

• Fix SLM check for restore in progress #50868

• Handle failure to retrieve ILM policy step better #49193 (issue: #49128)

• Don’t halt policy execution on policy trigger exception #49128

• Re-read policy phase JSON when using ILM’s move-to-step API #48827

• Don’t schedule SLM jobs when services have been stopped #48658 (issue: #47749)

• Ensure SLM stats does not block an in-place upgrade from 7.4 #48367

• Ensure SLM stats does not block an in-place upgrade from 7.4 #48361

• Add SLM support to xpack usage and info APIs #48096 (issue: #43663)

• Change policy_id to list type in slm.get_lifecycle #47766 (issue: #47765)

Features/Ingest

• Fix ignore_missing in CsvProcessor #51600

• Don’t overwrite target field with SetSecurityUserProcessor #51454 (issue: #51428)

• Fix ingest simulate response document order if processor executes async #50244

• Allow list of IPs in geoip ingest processor #49573 (issue: #46193)

• Do not wrap ingest processor exception with IAE #48816 (issue: #48810)

• Introduce dedicated ingest processor exception #48810 (issue: #48803)

Features/Java High Level REST Client

• Support es7 node http publish_address format #49279 (issue: #48950)

• Add slices to delete and update by query in HLRC #48420

• fix incorrect comparison #48208

• Fix HLRC parsing of CancelTasks response #47017

Features/Java Low Level REST Client

• Improve warning value extraction performance in Response #50208 (issue: #24114)

Features/Monitoring

• Validate exporter type is HTTP for HTTP exporter #49992 (issues: #47246, #47711, #49942)

Geo

• Guard against null geoBoundingBox #50506 (issue: #50505)

• Geo: Switch generated GeoJson type names to camel case (#50285) #50400 (issue: #49568)

• Geo: Switch generated WKT to upper case #50285 (issue: #49568)

• Fix typo when assigning null_value in GeoPointFieldMapper #49645

• Fix handling of circles in legacy geo_shape queries #49410 (issue: #49296)

• GEO: intersects search for geo_shape return wrong result [ISSUE] #49017

• Geo: improve handling of out of bounds points in linestrings #47939 (issue: #43916)

Highlighting

• Fix invalid break iterator highlighting on keyword field #49566

Infra/Core

• Ignore virtual ethernet devices that disappear #51581 (issue: #49914)

• Guess root cause support unwrap #50525 (issue: #50417)

• Allow parsing timezone without fully provided time #50178 (issue: #49351)

• [Java.time] Retain prefixed date pattern in formatter #48703 (issue: #48698)

• Don’t drop user’s MaxDirectMemorySize flag on jdk8/windows #48657 (issues: #44174, #48365)

• Warn when MaxDirectMemorySize may be incorrect (Windows/JDK8 only issue) #48365 (issue: #47384)

• [Java.time] Calculate week of a year with ISO rules #48209 (issues: #41670, #42588, #43275, #43652)

Infra/Logging

• Slow log must use separate underlying logger for each index #47234 (issue: #42432)

Infra/Packaging

• Extend systemd timeout during startup #49784 (issue: #49593)

Infra/REST API

• Return 400 when handling invalid JSON #49552 (issue: #49428)

• Slash missed in indices.put_mapping url #49468

MULTIPLE AREA LABELS

• Fix meta version of task index mapping #50363 (issue: #48393)

• Prevent deadlock by using separate schedulers #48697 (issues: #41451, #47599)

• APM system_user #47668 (issues: #2708, #40876)

Machine Learning

• [ML] Validate classification dependent_variable cardinality is at lea… #51232

• Do not copy mapping from dependent variable to prediction field in regression analysis #51227

• Handle nested and aliased fields correctly when copying mapping. #50918 (issue: #50787)

• [ML] Fix off-by-one error in ml_classic tokenizer end offset #50655

• [ML] Improve uniqueness of result document IDs #50644 (issue: #50613)

• [7.x] Synchronize processInStream.close() call #50581 (issue: #49680)

• Fix accuracy metric #50310 (issue: #48759)

• Fix race condition when stopping a data frame analytics jobs immediately after starting it #50276 (issues: #49680, #50177)

• Use query in cardinality check #49939

• Make only a part of stop() method a critical section. #49756 (issue: #49680)

• Fix expired job results deletion audit message #49560 (issue: #49549)

• [ML] Apply source query on data frame analytics memory estimation #49517 (issue: #49454)

• Stop timing stats failure propagation #49495

• [ML] Fix r_squared eval when variance is 0 #49439

• Blacklist a number of prediction field names. #49371 (issue: #48808)

• Make AnalyticsProcessManager class more robust #49282 (issue: #49095)

• [ML] Fixes for stop datafeed edge cases #49191 (issues: #43670, #48931)

• [ML] Avoid NPE when node load is calculated on job assignment #49186 (issue: #49150)

• Do not throw exceptions resulting from persisting datafeed timing stats. #49044 (issue: #49032)

• [ML] Deduplicate multi-fields for data frame analytics #48799 (issues: #48756, #48770)

• [ML] Prevent fetching multi-field from source #48770 (issue: #48756)

Mapping

• Ensure that field collapsing works with field aliases. #50722 (issues: #32648, #50121)

• Improve DateFieldMapper ignore_malformed handling #50090 (issues: #46675, #50081)

• Annotated text type should extend TextFieldType #49555 (issue: #49289)

• Ensure parameters are updated when merging flattened mappings. #48971 (issue: #48907)

Network

• Fix TransportMasterNodeAction not Retrying NodeClosedException #51325

Percolator

• Correctly handle MSM for nested disjunctions #50669 (issue: #50305)

• Fix query analyzer logic for mixed conjunctions of terms and ranges #49803 (issue: #49684)

Recovery

• Check allocation id when failing shard on recovery #50656 (issue: #50508)

• Migrate peer recovery from translog to retention lease #49448 (issue: #45136)

• Ignore Lucene index in peer recovery if translog corrupted #49114

Reindex

• Reindex and friends fail on RED shards #45830 (issues: #42612, #45739)

SQL

• SQL: Optimisation fixes for conjunction merges #50703 (issue: #49637)

• SQL: Fix issue with CAST and NULL checking. #50371 (issue: #50191)

• SQL: fix NPE for JdbcResultSet.getDate(param, Calendar) calls #50184 (issue: #50174)

• SQL: COUNT DISTINCT returns 0 instead of NULL for no matching docs #50037 (issue: #50013)

• Fix LOCATE function optional parameter handling #49666 (issue: #49557)

• Fix NULL handling for FLOOR and CEIL functions #49644 (issue: #49556)

• Handle NULL arithmetic operations with INTERVALs #49633 (issue: #49297)

• Fix issue with GROUP BY YEAR() #49559 (issue: #49386)

• Fix issue with CASE/IIF pre-calculating results #49553 (issue: #49388)

• Fix issue with folding of CASE/IIF #49449 (issue: #49387)

• Fix issues with WEEK/ISO_WEEK/DATEDIFF #49405 (issues: #48209, #49376)

• SQL: Fix issue with mins & hours for DATEDIFF #49252

• SQL: Failing Group By queries due to different ExpressionIds #43072 (issues: #33361, #34543, #36074, #37044, #40001, #40240, #41159, #42041, #46316)

Search

• Fix upgrade of custom similarity #50851 (issue: #50763)

• Fix NPE bug inner_hits #50709 (issue: #50539)

• Collect results in a local list when notifying partial results #49828 (issue: #49778)

• Fixes a bug in interval filter serialization #49793 (issue: #49519)

• Correctly handle duplicates in unordered interval matching #49775

• Correct rewritting of script_score query #48425 (issue: #48081)

• Do not throw errors on unknown types in SearchAfterBuilder #48147 (issue: #48074)

Security

• Always consume the body in has privileges #50298 (issue: #50288)

Snapshot/Restore

• Fix Overly Aggressive Request DeDuplication #51270 (issue: #51253)

• Guard Repository#getRepositoryData for exception throw #50970

• Fix Index Deletion During Partial Snapshot Create #50234 (issues: #50200, #50202)

• Fix Index Deletion during Snapshot Finalization #50202 (issues: #45689, #50200)

• Fix RepoCleanup not Removed on Master-Failover #49217

• Make FsBlobContainer Listing Resilient to Concurrent Modifications #49142 (issue: #37581)

• Fix SnapshotShardStatus Reporting for Failed Shard #48556 (issue: #48526)

• Cleanup Concurrent RepositoryData Loading #48329 (issue: #48122)

Transform

• [Transform] fail to start/put on missing pipeline #50701 (issue: #50135)

• Fix possible audit logging disappearance after rolling upgrade #49731 (issue: #49730)

• [Transform] do not fail checkpoint creation due to global checkpoint mismatch #48423 (issue: #48379)

Upgrades

Engine

• Upgrade to Lucene 8.4.0. #50518

Infra/Packaging

• Upgrade the bundled JDK to JDK 13.0.2 #51511

NOT CLASSIFIED

Aggregations

• Revert "Begin moving date_histogram to offset rounding (backport of #50873) (#50978)" #51240

Authentication

• Change password API authenticating with a bearer token [ISSUE] #48752

[russcam]

Closing this - posting new issue to Github timed out. This is superseded by #4341