[Meta] Index Patterns Roadmap

mattkime commented 4 years ago

Summary

The index patterns service is part of the data plugin and owned by @elastic/kibana-app-services. It provides a way to define sets of indices, load field lists, and format values. IndexPattern instances are also used as arguments for buildEsQuery and similar tooling.

Current goal - Index Patterns should be flexible and easy to use for both developers and Kibana users.

7.10 - Index Pattern API overhaul - clarity in functionality and persistence. 7.11 - Complete server side support, remove field list cache, create REST API

7.12

[x] Field editor plugin with runtime field editor support - https://github.com/elastic/kibana/issues/84786

7.13

[x] Plan to rename index patterns https://github.com/elastic/kibana/issues/94244
[ ] High level docs https://github.com/elastic/kibana/issues/92125 (in progress, nearly complete)

7.14

[ ] Index pattern creation dialog - https://github.com/elastic/kibana/issues/84725
[x] Index pattern field editor - add accessibility tests - https://github.com/elastic/kibana/issues/92536
[x] Use deprecation service for scripted fields - https://github.com/elastic/kibana/issues/97947
[x] REST API for selecting default pattern - https://github.com/elastic/kibana/issues/97337
[x] REST API for runtime field CRUD operations - https://github.com/elastic/kibana/issues/92126
[x] Switch to the new Elasticsearch client (tests) - https://github.com/elastic/kibana/issues/98608
[ ] Support fields that start with underscore - https://github.com/elastic/kibana/issues/2551
[ ] API for determining dependencies on a given index pattern - https://github.com/elastic/kibana/issues/97471
[ ] Index patterns management - tab that shows matched data sources - https://github.com/elastic/kibana/issues/72665
[x] (bugfix) url formatter stores base of url - https://github.com/elastic/kibana/issues/63924
[x] (bugfix) [App services] Permission errors logged when opening Kibana for the first time https://github.com/elastic/kibana/issues/95094
[x] Index patterns REST API docs https://github.com/elastic/kibana/issues/99372

7.x

[ ] Add Data field descriptions - https://github.com/elastic/kibana/issues/89726
[ ] REST API for list of all index patterns - https://github.com/elastic/kibana/issues/99928
[ ] Reduce index pattern field editor bundle size - https://github.com/elastic/kibana/issues/100366

8.0

[ ] Rename index patterns - https://github.com/elastic/kibana/issues/94244 - 8.1 - 8.3 (tbd)
[ ] Remove cached fields from old index patterns - https://github.com/elastic/kibana/issues/98383
[ ] Saved Object type conversion - https://github.com/elastic/kibana/issues/100489

Future

Bug Fixes

7.10 / 7.9.3
- [x] Allow creation of cross cluster index patterns with cluster version <7.9 https://github.com/elastic/kibana/issues/77825
[x] Excessive index patterns fetching on a dashboard - https://github.com/elastic/kibana/issues/82153
[ ] Improve text for Index Patterns UI - https://github.com/elastic/kibana/issues/80781
[ ] Kibana should display a message to user when there is no data in ES on index pattern management page - https://github.com/elastic/kibana/issues/81001
[ ] Space privilege "Index pattern management" read still shows delete button https://github.com/elastic/kibana/issues/53682
[x] Index pattern management field list UI needs to refresh after field refresh is performed - https://github.com/elastic/kibana/issues/60359
[ ] Index pattern creation - properly handle errors on failure to create saved object - https://github.com/elastic/kibana/issues/96043

Improve tests

[ ] https://github.com/elastic/kibana/issues/16194 - clean up snapshots

Refactoring

[x] Index Pattern API - clean up field index pattern and index patterns interfaces https://github.com/elastic/kibana/issues/68020
- [x] Move indexPattern.popularizeField into discover (In progress) https://github.com/elastic/kibana/pull/77668
- [x] Index pattern edit field formatter API (in progress) https://github.com/elastic/kibana/pull/78352
[ ] Simplify relationship between IndexPattern class, IIndexPattern, and IndexPatternSpec - https://github.com/elastic/kibana/issues/78378
[ ] Index Pattern API - type cleanup - https://github.com/elastic/kibana/issues/68012
[ ] Data stream code to xpack plugin - https://github.com/elastic/kibana/issues/71341
[ ] Replace StubIndexPattern with a more detailed jest mock - https://github.com/elastic/kibana/issues/78519
[ ] Index patterns scripted field type functionality removal (expressions script type no longer supported) - https://github.com/elastic/kibana/issues/74118
[ ] Index Patterns API - should pass errors on to callers instead of swallowing - https://github.com/elastic/kibana/issues/95993
[ ] Refactoring client / server to have more happen on the server, needs refinement - https://github.com/elastic/kibana/issues/90076

API improvements

[x] Data stream support - https://github.com/elastic/kibana/issues/62941
[x] Index Pattern server API - https://github.com/elastic/kibana/pull/69105
[x] Index Pattern toSpec / fromSpec - https://github.com/elastic/kibana/pull/68844
[x] Index Pattern API to server - https://github.com/elastic/kibana/issues/68003
[x] Remove field list cache - https://github.com/elastic/kibana/issues/71787
- [x] Support for loading field list in server API - https://github.com/elastic/kibana/issues/71588
[x] REST API - https://github.com/elastic/kibana/issues/72360 - in progress
[ ] Allow editing of index pattern title / pattern - https://github.com/elastic/kibana/issues/79412
[ ] Index patterns - allow title definition separate from pattern - https://github.com/elastic/kibana/issues/79413
[ ] Frozen index support on per index pattern basis - https://github.com/elastic/kibana/issues/49799
[ ] constant_keyword support - https://github.com/elastic/kibana/issues/52635
[ ] Index patterns do not support date_range ES type - https://github.com/elastic/kibana/issues/76980
[ ] Disambiguate IFieldType https://github.com/elastic/kibana/issues/100186

User Interface

[x] Empty index pattern page redesign - https://github.com/elastic/kibana/pull/68819
[ ] Index pattern creation dialog - https://github.com/elastic/kibana/issues/67711
[ ] Debounce missing index warnings - https://github.com/elastic/kibana/issues/62649

Index Pattern management

[ ] Index pattern matched source list, make sortable, show more items - https://github.com/elastic/kibana/issues/70512
[ ] Matched source summary header - https://github.com/elastic/kibana/issues/71505
[ ] Adding a field name length in index pattern display - https://github.com/elastic/kibana/issues/74220
- Would use EUI grid with optional column
[ ] Display more information about conflicts in index pattern fields - https://github.com/elastic/kibana/issues/74218
[ ] Index Patterns app doesn't unset its doc title- https://github.com/elastic/kibana/issues/72668
[ ] Tab that shows matched data sources - https://github.com/elastic/kibana/issues/72665
[ ] Index patterns management loads all field lists, lets optimize this - https://github.com/elastic/kibana/issues/100139

8.0

[ ] Remove expression-based scripted fields from Kibana. - https://github.com/elastic/kibana/issues/77669

Needs investigation

Removing getComputedFields - https://github.com/elastic/kibana/issues/85162
Better nested field support - https://github.com/elastic/kibana/issues/54957
Tighter field semantics for ECS indices - https://github.com/elastic/kibana/issues/69856
Field format editor via modal - investigate, design, and define tasks
Only fields with data - https://github.com/elastic/kibana/issues/81756
Provide a default field formatter based on the field metadata - https://github.com/elastic/kibana/issues/82348
Apply context filters when retrieving list of fields in application - https://github.com/elastic/kibana/issues/95558
- Related, but concerning CCS - https://github.com/elastic/kibana/issues/99100
Add telemetry to REST API - https://github.com/elastic/kibana/issues/100185

The long list of index pattern related problems - https://github.com/elastic/kibana/issues/35481

elasticmachine commented 4 years ago

Pinging @elastic/kibana-app-arch (Team:AppArch)

ppf2 commented 4 years ago

Would be nice to also catch potential misconfigurations: https://github.com/elastic/kibana/issues/80565

ppisljar commented 2 years ago

Thank you for contributing to this issue, however, we are closing this issue due to inactivity as part of a backlog grooming effort. If you believe this feature/bug should still be considered, please reopen with a comment.

elastic / kibana