[DISCUSS] Rethinking index patterns; improvements

[mattkime]

Note: There are a lot of overlapping issues. Lets aim to identify and address portions of the problem either in this issue or new issues.

Pain points

• User must have write permissions to modify field formatters - which are attached to index patterns.
• User must create index pattern before using most tools
• User must understand difference between indices and index patterns, understand how wildcards work
• User needs suffix / prefix for index names to use index pattern, might not have it
• Index mapping changes require explicit refresh - due to slow speed of refresh
  • We've had customers with such large response payloads that it's crashed the browser
• Can't create index pattern unless index and document exist
• Index patterns have wildcards, but indexes do not. If a new index is created that matches the wildcard, the index pattern might not match fully
• Security concerns
  • Users without index privileges are shown those indexes in Discover, Visualize, Canvas, etc
  • Users with field-level security are able to select restricted fields in visualizations
• Default index pattern concerns
  • More plugins like Beats and APM are generating index patterns programmatically, and might exist before any user interaction
  • Discover, visualize, and dashboards require a default index pattern preference
  • This interacts badly with the index privileges above, if not all users can see all indexes

Features of index patterns that have mixed support and don't map directly to ES indexes:

• Scripted fields
  • The "esaggs" function used by visualizations handles this translation automatically, but not the "esdocs" function. This is a problem in Canvas, and is handled differently in Maps.
• Field popularity, or "count"
  • Seems to only be used in discover to sort some fields to the top of the list
• Field formatters, including coloring
  • Formatters are applied client-side, so they are not consistently used. For example, Field formatters are not used by Canvas
  • Coloring might only work in data tables

Index-related issues due to the many kinds of indexes:

• Rollup indexes can only be queried by aggregations, and only certain aggregations are supported
• Data types in index patterns are simplified to Javascript types like "string" instead of "text" and "keyword"
  • Developers have to make a choice about which field to use in search vs visualization

Potential solutions

Smarter UI for using / selecting index patterns

• Allow using index as though its an index pattern. May depend upon selecting time field.
• Prompt for time field when needed
• Need UI that would help explain wildcards
• Make reusable

Some consequences - Need shared UI, code consuming index patterns shouldn't assume index patterns are complete. Is it possible to select a time field without having a field mapping?

Smarter relationship with field mapping

• Better way to bust field cache
• Generate field mapping when used, when index and documents exist

Some consequences - Need to make sure loading field mapping is async

Question - How expensive is generating the field mapping?

Index naming concerns

I'm not sure how to address this one.

Major related issues & enhancement requests

• Add semantic data for fields to either index patterns or mappings [#17087]
• Hide fields restricted by FLS (field-level security) [#8192]
• Allow index pattern creation without a matching index [#35898]
• Allow creating index patterns on indices with no data [#15666]
• Get normalized mappings from ES so we can remove index pattern mapping cache [#6498]
• Add extension points for field types so that non-OSS types such as constant_keyword can register themselves from x-pack. [#52635]

[elasticmachine]

Pinging @elastic/kibana-app-arch

[rayafratkina]

Thanks @mattkime, very helpful writeup! I think another pain point is lack of visibility when something changed in the indexes and index patterns need to be rebuilt.

Also, not sure how to factor in thinking about scripted fields and, in general, ways to customize data.

For reference, here are related issues

[nreese]

Another thing to consider is Field Level Security (FLS) https://github.com/elastic/kibana/issues/8192. Index Pattern saved object expose field names to all users.

https://github.com/elastic/kibana/issues/34334 was opened against the Maps application but is an Index Pattern issue.

[kobelb]

User needs suffix / prefix for index names to use index pattern, might not have it

Would you mind elaborating on this?

[mattkime]

Would you mind elaborating on this?

If there's no pattern to your index names then you'll have trouble constructing an index pattern.

[kobelb]

For what it's worth, it's not intuitive at all but you can create an index pattern for foo,bar and it'll let you query both foo and bar at the same time.

[ruflin]

Can't create index pattern unless index and document exist

You can and Beats and APM does this today. The problem that this brings is that Kibana shows an error if not data is there yet.

When we rethink index patterns one additional thing I would like to see being considered is that index patterns will likely to be more and more managed from the outside, meaning created by module definitions. For this it would be nice to have something like index pattern inheritance, meaning I can have 5 small index patterns which all apply to the same indices (similar to inheritance in index templates).

[mattkime]

For what it's worth, it's not intuitive at all but you can create an index pattern for foo,bar and it'll let you query both foo and bar at the same time.

I don't think it needs to be intuitive, but it should be explained in the user interface

[ruflin]

Adding two more things to the wishlist here:

• Giving index patterns a name: It's possible to use multiple patterns like filebeat-*,metricbeat-*,apm-* for one index pattern. It would be nice if then a name like o11y could be given to it instead of showing up like the above in the Discovery UI
• Support inheritance / composing of index pattern: It would be nice that if index patterns in Kibana could work the same as index templates in Elasticsearch where a priority can set. And if multiple match the one with higher priority is applied first. This would allow us on the Beats side to have on index pattern per module but still have for example a filebeat-* index pattern for all together.

[monfera]

While in agreeing with the above pain points, it's hard to incrementally address them one by one with the expectation that if all are solved, all will be good, or that fixing them one step at a time is optimal. The same way that it's hard to incrementally modify a slow helicopter into a fast airplane, though both are well understood things.

In this case, it doesn't even feel like a well-defined construct; more like an evolved hub that links certain things to other things, doing so informally. So it'd be great to tackle all these at one fell swoop, in form of conceptual design. See also my comment on renaming.

[mattkime]

https://github.com/elastic/kibana/issues/40071

[mattkime]

We had a video chat regarding index patterns. While the intention was to talk about naming, particularly the index pattern string (apm-*) vs the index pattern object but the conversation strayed through all index pattern concerns.

Todo - why do these problems exist? They're likely solving a problem. Does the problem being solved still exist?

@jasonrhodes pointed out that index patterns aren't referenced in es documentation. its just a reference to indices with a wildcard. interesting distinction.

@sqren says es meta data may be able to replace much of the field caching.

@jasonrhodes has some scars from working around the existing index patterns api. some development teams hack around it. we should talk to him and other people to understand how the api isn't working well for him.

[monfera]

I mentioned that

1. We're a search company, indexes themselves should be easy to search/select, we shouldn't have artificial obstacles for the users, ie. tools such as Lens, Graph etc. should be able to work from ES data and metadata directly (several folks said that it looks like most if not all field metadata are now in ES too)
2. By extension, index patterns could just go away as a requirement
3. Instead of the index patterns, we could have views (maybe named as views, as common in DBMS, maybe something else) for the purpose of controlling access (eg. group of indices and allowed document fields) for users or user roles
4. It would also be natural to link computed fields to said views (as computed fields may also be subject to access control), and eg. as SAP and other large entity management systems do, introduce some concept of data domains and data elements, which could include choices like default field formatting

So, in actionable terms,

1. Where currently an index pattern is needed, just let folks use an index, or an arbitrary set of index selections using regex matching (it might be a hidden object creation if need be, but then the problem is statefulness, resource allocation and possible accumulation, so ideally there's no object, or just something transient)
2. The index pattern (objects) could be renamed as Views or similar; where users can specify an index (or group of indices) directly, they can also specify a "view index"
3. These views would have proper, user-given names, rather than using the index group as the name, for numerous reasons
4. Users or user roles could be assigned with indices (all docs and all fields authorized) or Views (with fine-grained access control, more restricted than full indices)

[ruflin]

@monfera When you mention "views" in the above I always replace it with "alias" in my head.

[justinwalz]

Hi, I was directed here from the discuss forums https://discuss.elastic.co/t/update-index-pattern-to-only-add-fields/200489.

Feature request would be to add an option to the index pattern refresh operation to only add new fields, not remove older ones. Our use case is that we retire old logs somewhat aggressively, so if we refresh during a period where there are no logs with a given field, it will get removed from the pattern. This ends up breaking visualizations.

Thanks, Justin

[monfera]

@ruflin Good call, index aliases seem to be closely related.

tl;dr: What an index alias can do is covered by what I meant by the broader view (which would also cover field selections, scripted fields, format info etc. to preserve current index pattern functionality and be in sync with the general meaning of view, which is an accepted term for RDBMS views as well as noSQL eg. MongoDB views - great list of motives under both links!)

---

If I understand, index aliases can:

1. Be a stand in for one index, or multiple indices (in that, they seem to fully overlap with the Kibana regex pattern matching that we call "index patterns") - where one could use an index (or more), one can also use an alias. So this is already a major redundancy between ES and Kibana. Can someone tell why Kibana is not just piggybacking on aliases? I can think of reasons but don't want to second-guess.
2. Filter results (I assume it can be an arbitrary ES / Lucene query, but I suppose, no SQL, KQL, timelion etc.) - while index patterns don't do that, there's some redundancy as Kibana uses filters here and there, maybe it even has integration with aliases. So it's not relevant for index patterns right now but chances are, we would've added filtering functionality to index patterns eventually 😄 Because if index patterns can select "columns", why not "rows"?
3. Routing and possibly other neat stuff - irrelevant for index patterns now
4. Restricting document fields, or other field-specific config seems not to be part of index aliases.

In short, aliases seem like a good foundation for the index pattern part of index patterns.

Now it turns out there's another type of alias, the alias datatype, which is neat in that - if I understand - one can specify a top-level alias field to avoid spelling out a nested document field path each time. So it's somewhat related to index patterns. They're also related to field capabilities which can supply Kibana with info such as "is aggregation for this field in this index supported?". We're on the hunt for metadata, and tools like Lens may benefit from such info if not already using these (any comment @chrisdavies @wylieconlon @flash1293?)

The term alias is an ES-specific term, and implies substitution of an original item (or more) with a shorthand. I think the filtering capability already stretches the meaning of alias.

The term view is broader (DBMS - irrespective of relational, schema-free etc. - or even comp sci) and it could cover

• now: the enlisting of included fields (current functionality of index patterns)
• now: computing (deriving) fields - existing index pattern related functionality, called scripted fields
• now: renaming of fields: simplifying, user-friendlifying, or unnesting a deep document path (I think it'd be useful for Dashboard, Lens, Canvas, if it has a feature request issue, pls link). I tagged it now too because a scripted field can be used to just take the value from a physical, overly technically named or nested field.
• now: metadata eg. default field formatters, or some rudimentary type info (ordinal/cardinal, positive numbers, integers etc.) - these aren't part of eg. relational DBMS views, but field formatters already exist, and Lens, and charting in general, craves metadata, so they could be related to, or perhaps part of, views
• future: filtering - why would it not be a natural place? Also, one use of index patterns and the restricted field sets they allow is vertical access control; I suppose there is, or can be, horizontal access control. Also, ES index aliases already have this function, so if views piggyback on that, it'd be low hanging fruit
• future: aggregation. Data frames, and to some extent, rollups are that already
• future: joining. For example, picking up country names for country codes. Snowflake, star or arbitrary schemas

View is then a general capability to

• manipulate data for reporting (select index(es), fields, documents; compute things; collate, rename, sort...) atop of physically persisted ES data
• make Kibana reporting largely independent of the underlying schema; an index schema change would be easier to perform while preserving all reports
• exploit existing ES capabilities (alias, dataframe, rollup, ...) and abstract over their idiosyncracies

A nontrivial but essential part is figuring out where to draw the line between ES and Kibana, and how to still benefit from ES abstractions where Kibana also has overlapping abstractions.

[weltenwort]

Be a stand in for one index, or multiple indices [...] - where one could use an index (or more), one can also use an alias. So this is already a major redundancy between ES and Kibana. Can someone tell why Kibana is not just piggybacking on aliases?

There is a major limitation of Elasticsearch index aliases compared to the index glob expressions used on the front-end: Index aliases can't span clusters using CCS (elastic/elasticsearch#43312).

With CCS being a more recent concept, this was certainly not part of the initial consideration of Kibana index patterns, but it has become an important use-case.

[cjcenizal]

pointed out that index patterns aren't referenced in es documentation. its just a reference to indices with a wildcard. interesting distinction.

The term index_patterns is part of the index templates API. Possibly it originated there?

[mattkime]

Metric Explorer doing index pattern like custom work. Needs closer inspection to see how it relates to this conversation, just wanted to reference it before I forget. https://github.com/elastic/kibana/pull/43322

and Lens - https://github.com/elastic/kibana/tree/master/x-pack/legacy/plugins/lens/server/routes and https://github.com/elastic/kibana/blob/master/x-pack/legacy/plugins/lens/public/indexpattern_plugin/indexpattern.tsx#L36

[AndrewMcQuerry]

pointed out that index patterns aren't referenced in es documentation. its just a reference to indices with a wildcard. interesting distinction.

The term index_patterns is part of the index templates API. Possibly it originated there?

The usage of index_patterns in the index template API has only existed since 6.x. In 5.x and prior, the field was called template. Thus, the Kibana Index Pattern terminology has been around much longer than the Index Template API.

Each of those two uses of the term "index pattern" can exist without the other and they have no connection other than coincidental naming overlap.

[AndrewMcQuerry]

Historically, we have created various Kibana Index Patterns, with varying use of wildcards since that was the easiest thing to do. However, in practice, that has also caused an extremely confusing list of multiple Index Patterns each with a different purpose.

For example: logs (@timestamp) logs-elasticsearch (@timestamp) logs-kibana (@timestamp) logs- (ingest_timestamp)

In addition, creating these Index Patterns in Kibana will result in a random "Index Pattern ID" chosen during creation. This can be problematic if someone accidentally deletes the Index Pattern.. even during recreation, the Index Pattern ID would be a new ID and all existing saved objects would no longer reference a valid Index Pattern ID.

We are looking to address this by:

1. No longer create Index patterns using wildcard.
2. No longer allow Kibana to set a randomly generated Index Pattern ID.
3. Anything that is to be an "Index Pattern" must first be created as an Alias.
4. Only Aliases are then allowed to be created as Index Patterns in Kibana.
5. The "Index Pattern ID" is manually set during creation to be equal to the Index Pattern / Alias.
6. Curator or ILM is used to manage which indexes are associated with each Alias.

[craigboman]

I'm not sure if the use of the term "alias" makes a lot of sense from an end user perspective. Alias in my mind complicates the discussion even further. Despite this already being an ES label we'd like to borrow into Kibana, an alias doesn't mean anything to Kibana users, the end user like myself.

As we all may know an index "is a list of words or phrases and associated pointers to where useful material relating to that heading can be found in a document or collection of documents." (Wikipedia). As a librarian like myself, what ES devs are referring to as an alias is just an index. An index/alias in this case, is a list of pointers to where more information can be found. If I index all of my daily audit logs into ES/kibana under the index label/pattern "audit-", this would allow me to collect all of the pointers to data for those dates matching the same index pattern ("audit-2019-11-30", or "audit-2019-11-31", etc). From my perspective "audit-" is the audit index with pointers to "audit-2019-11-30". If all of my data in audit- has the exact same mapping, its just one index, not multiple indices; if I'm streaming audit data into the audit index using the audit index pattern, its still just the audit index. I would also go as far as to separate out the variable "" from the index pattern label. For instance instead of "audit-*" it should just be "audit" index. Yes this can be done through the custom index label name, but the custom label should just be default rather than custom.

This also makes sense from a front-end kibana user, in the sense that when I want to change my index pattern, I'm really just changing the index against which I'm trying to visualize, I'm not changing my "alias" when I visualize; that sounds weird. From the front-end, I would highly recommend just dropping the use of index pattern when referring to visualizations. So when I'm visualizing, I would only need to change my index; the "pattern" is extraneous.

Thanks in advance for this consideration. As always: love Kibana. Keep up the great work. :)

[petrklapka]

Thank you for contributing to this issue, however, we are closing this issue due to inactivity as part of a backlog grooming effort. If you believe this feature/bug should still be considered, please reopen with a comment.