DRAFT [META] REST Compatible API V7 completeness

REST Compatible API v7 completeness

As described in https://github.com/elastic/elasticsearch/issues/51816 all breaking changes could be subject to applying some compatibility at the REST layer to assist with upgrades. This issue exists to ensure that there is a complete and exhaustive list of all breaking changes and make the determination REST API compatibility should be applied.

A complete list of all breaking changes categorized by

Requires REST API compatibility with v7
Does not require REST API compatibility with v7
Unsure - needs more investigation (temporary status while we work through the list)
Potentially leverage REST API compatibility, but out of immediate scope

The number of check boxes here should equal number of closed issues from this query: Eldest to newest breaking changes

Each line item should have a independent validation, and line items that require changes should be linked to the associated PR and issues. For specific compatibilities that require discussions, please open a new issue (as opposed to the comments of this issue).

Requires REST API compatibility with v7

Misc

//TODO: break up into better categories

[ ] Remove deprecated _time and _term sort orders https://github.com/elastic/elasticsearch/pull/39450 - //Przemek: this parses directly using XContentParser (not using ObjectParser and field declaration). Should be a fairly easy to do with surgical conditional (if parser.getRestVersion=V7)
[ ] Parse empty first line in msearch request body as action metadata https://github.com/elastic/elasticsearch/pull/41011 //Przemek: a request shape change. A fairly minor, but would be breaking users. The compatible change is basicallly bringing back the 7.x code (deprecation and parsing) and wrapping with if(isCompatible). The readMultiLineFormat would have to get that flag from restrequest in RestMultiSearchAction
[ ] Get snapshots support for multiple repositories https://github.com/elastic/elasticsearch/pull/42090 //Przemek: it is a fairly big change and might require help from es-distributed team. There is an issue for this work already https://github.com/elastic/elasticsearch/issues/69108
[ ] Remove CommonTermsQuery and cutoff_frequency param https://github.com/elastic/elasticsearch/pull/42654 //Przemek: it looks like a fairly simple change. For cutoff_frequency we could simply allow to parse these but do nothing. But for CommonTermsQuery it looks more complex and I am not sure how to implement this without bringing too much code back.
[ ] Remove deprecated sort options: nested_path and nested_filter https://github.com/elastic/elasticsearch/pull/42809
[x] Reindex remove outer level size https://github.com/elastic/elasticsearch/pull/43373 compatible reindex https://github.com/elastic/elasticsearch/pull/69037 compatible update/delete by query https://github.com/elastic/elasticsearch/pull/69606
[ ] [ML][Data Frame] removing format support in date_histogram group_by https://github.com/elastic/elasticsearch/pull/43659 (feature was beta when this change was made)
[ ] Force Merge should reject requests with only_expunge_deletes and max_num_segments set https://github.com/elastic/elasticsearch/pull/44761
[ ] Remove deprecated endpoints containing _xpack. https://github.com/elastic/elasticsearch/pull/48170
[ ] Remove the 'template' field in index templates https://github.com/elastic/elasticsearch/pull/49460
[ ] Remove the 'local' parameter of /_cat/nodes https://github.com/elastic/elasticsearch/pull/50594
[ ] Goodbye and thank you synced flush! https://github.com/elastic/elasticsearch/pull/50882
[ ] Remove support for settings in restore requests https://github.com/elastic/elasticsearch/pull/53284
[ ] Remove the object format for indices_boost https://github.com/elastic/elasticsearch/pull/55078
[ ] Remove local parameter for get field mapping request https://github.com/elastic/elasticsearch/pull/55100
[ ] Remove use_field_mapping format option for docvalue fields https://github.com/elastic/elasticsearch/pull/55622
[ ] Remove node filters for voting config exclusions https://github.com/elastic/elasticsearch/pull/55673
[ ] Scripting: Move script_cache into _nodes/stats https://github.com/elastic/elasticsearch/pull/59265
[ ] Consolidate script parsing from object https://github.com/elastic/elasticsearch/pull/59507
[ ] [ML] Remove deprecated _xpack endpoints #59870
[ ] Rename binary_soft_classification evaluation to outlier_detection https://github.com/elastic/elasticsearch/pull/59951 (feature was experimental when this change was made)
[ ] Remove deprecated _upgrade API https://github.com/elastic/elasticsearch/pull/64732
[ ] Remove the deprecated local parameter for _cat/shards https://github.com/elastic/elasticsearch/pull/64867
[ ] Remove the deprecated local parameter for _cat/indices https://github.com/elastic/elasticsearch/pull/64868
[ ] [Transform] use ISO dates in output instead of epoch millis https://github.com/elastic/elasticsearch/pull/65584
[ ] Returning nested data in fields API https://github.com/elastic/elasticsearch/pull/67432
[ ] Remove MovingAverage pipeline aggregation https://github.com/elastic/elasticsearch/pull/39328

Types

These are mostly managed by https://github.com/elastic/elasticsearch/issues/54160. The issues here match 1:1 to the issues with >breaking label to help ensure the counts of breaking changes and number of checkboxes are equal.

Removes typed URLs https://github.com/elastic/elasticsearch/pull/41636 not merged. splitted into multiple prs
[ ] Removes typed endpoint from search and related APIs https://github.com/elastic/elasticsearch/pull/41640
[ ] Removes tpyed endpoints for msearch and mtermvector APIs https://github.com/elastic/elasticsearch/pull/41674
[ ] Removes typed URLs from mapping APIs https://github.com/elastic/elasticsearch/pull/41676
[ ] Removes types from bulk API https://github.com/elastic/elasticsearch/pull/42194
[ ] Removes type from TermVectors APIs https://github.com/elastic/elasticsearch/pull/42198
[ ] Remove types from Get/MultiGet https://github.com/elastic/elasticsearch/pull/46587
[ ] Remove per-type indexing stats https://github.com/elastic/elasticsearch/pull/47203
[ ] Remove type query https://github.com/elastic/elasticsearch/pull/47207
[ ] Remove include_type_name parameter from REST layer https://github.com/elastic/elasticsearch/pull/48632
[ ] Remove support for _type in searches https://github.com/elastic/elasticsearch/pull/68564

Mappings

[ ] Remove support for chained multi-fields. https://github.com/elastic/elasticsearch/pull/42333
[ ] Remove support for string in unmapped_type https://github.com/elastic/elasticsearch/pull/45675
[ ] Remove support for sparse vectors https://github.com/elastic/elasticsearch/pull/48781
[ ] Remove xpack actions for the flattened field https://github.com/elastic/elasticsearch/pull/53076
[ ] Remove TypeFieldMapper https://github.com/elastic/elasticsearch/pull/53611
[ ] Fix updating include_in_parent/include_in_root of nested field https://github.com/elastic/elasticsearch/pull/54386
[ ] Disallow changing 'enabled' on the root mapper https://github.com/elastic/elasticsearch/pull/54463
[ ] Add declarative parameters to FieldMappers https://github.com/elastic/elasticsearch/pull/58663 (needs a better look)

Unsure - needs more investigation

[ ] [ML] add new flag exclude_generated that removes generated fields in GET config APIs https://github.com/elastic/elasticsearch/pull/63899
[ ] Don't expose TextFieldMapper subfields https://github.com/elastic/elasticsearch/pull/64597
[ ] QL: "fields" api implementation in QL https://github.com/elastic/elasticsearch/pull/68802

Potentially leverage REST API compatibility, but out of immediate scope

scripting

[ ] Update the signature of vector script functions https://github.com/elastic/elasticsearch/pull/48260
[ ] Remove deprecated vector functions https://github.com/elastic/elasticsearch/pull/48725

EQL

[ ] EQL: Forbid usage of ['] for string literals https://github.com/elastic/elasticsearch/pull/62458
[ ] EQL: Replace ?"..." with """...""" for unescaped strings https://github.com/elastic/elasticsearch/pull/62539
[ ] EQL: Replace [`] with ['] for identifiers https://github.com/elastic/elasticsearch/pull/62638
[ ] EQL: Remove support for = for comparisons https://github.com/elastic/elasticsearch/pull/62756

Does not require REST API compatibility with v7

Setttings

Settings are not subject to REST API compatibility. Often the changes behind the settings represent some behavior that is not carried forward.

[x] Remove obsolete security settings https://github.com/elastic/elasticsearch/pull/40496
[ ] Remove deprecated search.remote settings https://github.com/elastic/elasticsearch/pull/42381
[x] Remove node.max_local_storage_nodes https://github.com/elastic/elasticsearch/pull/42428
[x] Do not set a NameID format in Policy by default https://github.com/elastic/elasticsearch/pull/44090
[x] Remove processors setting https://github.com/elastic/elasticsearch/pull/45905
[ ] Remove the pidfile setting https://github.com/elastic/elasticsearch/pull/45940
[x] Forbid settings without a namespace https://github.com/elastic/elasticsearch/pull/45947
[ ] Disallow _field_names enabled setting https://github.com/elastic/elasticsearch/pull/46681
[ ] Remove include_relocations setting https://github.com/elastic/elasticsearch/pull/47717
[ ] [Transform] introduce new roles and deprecate old ones https://github.com/elastic/elasticsearch/pull/47780
[x] Make order setting mandatory for Realm config https://github.com/elastic/elasticsearch/pull/51195
[ ] Make soft-deletes mandatory in 8.0 https://github.com/elastic/elasticsearch/pull/51122
[ ] Remove translog retention settings https://github.com/elastic/elasticsearch/pull/51697
[x] Remove fixed_auto_queue_size threadpool type https://github.com/elastic/elasticsearch/pull/52280
[x] Remove the cluster.remote.connect setting https://github.com/elastic/elasticsearch/pull/54175
[x] Remove the node local storage setting https://github.com/elastic/elasticsearch/pull/54381
[ ] Remove support for delaying state recovery pending master https://github.com/elastic/elasticsearch/pull/53845
[ ] Remove deprecated AUTH_PASSWORD setting https://github.com/elastic/elasticsearch/pull/54892
[x] Remove deprecated basic license feature enablement settings from 8.0 https://github.com/elastic/elasticsearch/pull/56211
[x] Remove slowlog level https://github.com/elastic/elasticsearch/pull/57591
[ ] Default gateway.auto_import_dangling_indices to false https://github.com/elastic/elasticsearch/pull/58898
[ ] Scripting: Remove general cache settings https://github.com/elastic/elasticsearch/pull/59262
[ ] Set specific keepalive options by default on supported platforms https://github.com/elastic/elasticsearch/pull/59278
[ ] Remove the listener thread pool https://github.com/elastic/elasticsearch/pull/53314
[ ] Remove join timeout https://github.com/elastic/elasticsearch/pull/60873
[x] Remove content type required setting https://github.com/elastic/elasticsearch/pull/61043
[ ] Add a cluster setting to disallow loading fielddata on _id field https://github.com/elastic/elasticsearch/pull/49166
[x] Remove insecure settings https://github.com/elastic/elasticsearch/pull/46147
[x] Limit processors by available processors https://github.com/elastic/elasticsearch/pull/44894

Validation

Additional or stricter validation is a type of breaking change that can change the response of a request from a success to a specific failure. However, success -> error responses are not covered by REST API compatibility and is considered behavior.

[ ] Make Fuzziness reject illegal values earlier https://github.com/elastic/elasticsearch/pull/33511
[ ] Allow parsing Content-Type and Accept headers with version https://github.com/elastic/elasticsearch/pull/61427
[x] API key name should always be required for creation https://github.com/elastic/elasticsearch/pull/59836
[ ] Add validation for dynamic templates https://github.com/elastic/elasticsearch/pull/51233
[x] Reject misconfigured/ambiguous SSL server config https://github.com/elastic/elasticsearch/pull/45892
[ ] SQL: Test and fix the NULL handling of the String functions https://github.com/elastic/elasticsearch/pull/68379
[ ] Highlighters skip ignored keyword values https://github.com/elastic/elasticsearch/pull/53408
[ ] Move pipeline agg validation to coordinating node https://github.com/elastic/elasticsearch/pull/53669
[ ] Distinguish between simple matches with and without the terms index https://github.com/elastic/elasticsearch/pull/63945
[x] Fail node containing ancient closed index https://github.com/elastic/elasticsearch/pull/44264
[ ] Enforce Completion Context Limit https://github.com/elastic/elasticsearch/pull/38675
[ ] Add lower bound on poll_interval https://github.com/elastic/elasticsearch/pull/39593
[ ] Percentiles aggregation: disallow specifying same percentile values twice https://github.com/elastic/elasticsearch/pull/52257
[ ] Reject port ranges in `discovery.seed_hosts https://github.com/elastic/elasticsearch/pull/41404

Is actually breaking for 8.0 ?

Perhaps mislabeled or was beta/experimental and/or change was made in a minor

[ ] Stop returning cluster state size by default https://github.com/elastic/elasticsearch/pull/40016
[ ] Remove Migration Upgrade and Assistance APIs https://github.com/elastic/elasticsearch/pull/40075
[ ] Remove cluster state size https://github.com/elastic/elasticsearch/pull/40061
[ ] [ML-DataFrame] Combine task_state and indexer_state in _stats https://github.com/elastic/elasticsearch/pull/45276 (feature was beta when this change was made)
[ ] [ML] Improve response format of data frame stats endpoint https://github.com/elastic/elasticsearch/pull/44350 (feature was beta when this change was made)
[ ] [ML] Improve progress reportings for DF analytics https://github.com/elastic/elasticsearch/pull/45856 (feature was experimental when this change was made)
[ ] [ML][Transforms] remove force flag from _start https://github.com/elastic/elasticsearch/pull/46414 (feature was beta when this change was made)
[ ] Remove Repository Stats API https://github.com/elastic/elasticsearch/pull/62309

Misc

[ ] Drop support for the low-level REST client on JDK 7 https://github.com/elastic/elasticsearch/pull/38540 (drop support)
[ ] Blob Store compress default to true https://github.com/elastic/elasticsearch/pull/40033 (behavior)
[ ] Make remote cluster resolution stricter https://github.com/elastic/elasticsearch/pull/40419 (behavior)
[ ] Drop support for Ubuntu 14.04 https://github.com/elastic/elasticsearch/pull/40709 (not rest api)
[ ] Bump the minimum Java version to Java 11 https://github.com/elastic/elasticsearch/pull/40754 (not rest api)
[x] Fixed synchronizing inflight breaker with internal variable https://github.com/elastic/elasticsearch/pull/40878 (not rest api)
[ ] Reindex from Remote encoding https://github.com/elastic/elasticsearch/pull/41007 (behavior)
[ ] RollupStart endpoint should return OK if job already started https://github.com/elastic/elasticsearch/pull/41502 (behavior and response status code)
[ ] Cleanup versioned deprecations in analysis https://github.com/elastic/elasticsearch/pull/41560 (clean up no longer relvant)
[ ] Provide an Option to Use Path-Style-Access with S3 Repo https://github.com/elastic/elasticsearch/pull/41966 (behavior)
[ ] Remove the migrate tool https://github.com/elastic/elasticsearch/pull/42174 (clean up)
[x] Update the schema for the REST API specification https://github.com/elastic/elasticsearch/pull/42346 (not relevant)
[x] Remove "nodes/0" folder prefix from data path https://github.com/elastic/elasticsearch/pull/42489 (not relevant)
[ ] Remove client jar support from build https://github.com/elastic/elasticsearch/pull/42640 (cleanup)
[ ] Do not allow modify aliases on followers https://github.com/elastic/elasticsearch/pull/43017 (behavior)
[x] Remove the client transport profile filter https://github.com/elastic/elasticsearch/pull/43236 (cleanup)
[ ] Remove preconfigured `delimited_payload_filter https://github.com/elastic/elasticsearch/pull/43686 (behavior)
[ ] Forbid empty doc values on vector functions https://github.com/elastic/elasticsearch/pull/43944 (behavior)
[ ] [ML] Remove the ability to move datafeeds between jobs https://github.com/elastic/elasticsearch/issues/44616 (behavior ...but maybe this should be considered ?)
[ ] Remove client feature tracking https://github.com/elastic/elasticsearch/pull/44929 (cleanup)
[ ] Decouple shard allocation awareness from search and get requests https://github.com/elastic/elasticsearch/pull/45735 (behavior)
[ ] Use float instead of double for query vectors https://github.com/elastic/elasticsearch/pull/46004 (behavior)
[ ] Add support for aliases in queries on _index https://github.com/elastic/elasticsearch/pull/46640 (behavior, but needs another look)
[ ] [Transform]rename classes in transform plugin https://github.com/elastic/elasticsearch/pull/46784 (not rest api)
[ ] [Transform] Rename internal indexes for transform plugin https://github.com/elastic/elasticsearch/pull/47788 (behavior)
[ ] Link child requests to the root task that initiated the action https://github.com/elastic/elasticsearch/pull/48422 (closed...probably should remove breaking label)
[ ] Remove type metadata from ingest documents https://github.com/elastic/elasticsearch/pull/50131 (behvior)
[ ] Make range query rounding consistent https://github.com/elastic/elasticsearch/pull/50237
[x] Remove DEBUG-level default logging from actions https://github.com/elastic/elasticsearch/pull/51459 (behavior)
[ ] Drop support for older OS in the next major release https://github.com/elastic/elasticsearch/issues/51480 (dropping support)
[ ] Remove SysV init support https://github.com/elastic/elasticsearch/pull/51716 (dropping support)
[ ] Use a tiny base image for Docker builds https://github.com/elastic/elasticsearch/pull/52519 (build change)
[x] Remove support for fractional byte size values https://github.com/elastic/elasticsearch/pull/53927 (behavior)
[ ] value_count Aggregation optimization https://github.com/elastic/elasticsearch/pull/54854 (behavior)
[ ] Change prefer_v2_templates parameter to default to true https://github.com/elastic/elasticsearch/pull/55489 (behavior)
[ ] Add auto create action https://github.com/elastic/elasticsearch/pull/55858 (behavior)
[ ] Remove dangling index auto import functionality https://github.com/elastic/elasticsearch/pull/59698 (behavior)
[ ] EQL: Disallow chained comparisons https://github.com/elastic/elasticsearch/pull/62567 (behavior)
[ ] Fix range query on date fields for number inputs https://github.com/elastic/elasticsearch/pull/63692 (behavoir)
[ ] Move watcher history to data stream https://github.com/elastic/elasticsearch/pull/64252 (behvaior)
[x] Compress audit logs https://github.com/elastic/elasticsearch/pull/64472 (behavior)
[ ] Remove escape hatch permitting incompatible builds https://github.com/elastic/elasticsearch/pull/65753 (build)
[ ] Remove grace period from license expiration check https://github.com/elastic/elasticsearch/pull/67316 (behavior)
[ ] Remove support for JAVA_HOME https://github.com/elastic/elasticsearch/pull/69149 (build)

elastic / elasticsearch