Discover: Support accessing failure store documents

[flash1293]

The failure store is a new feature that will ship as tech preview in Elasticsearch 8.15. It allows to capture documents that couldn't be processed in a separate index per data stream to allow troubleshooting and re-indexing.

As part of the troubleshooting process, it's important to provide a way to look at the documents in the failure store. As Discover is the main app to look at individual documents, it should be possible to also view these docs.

This is complicated by the fact that data fetching work slightly different as an additional query parameter needs to be added to specify whether failure store docs are included into the search or not.

Implementation

The functionality should be added as a hidden flag in the discover app state that's passed along to data fetching endpoints:

• Field list
• Document search
• Histogram aggregation
• Source fetching
• Context fetching

The flag is synced to a URL parameter. There is no UI element for the user to actively change the value of the flag, but the discover locator has to support it as a parameter to be set when linking from other applications.

It will also be possible to control this flag within the logs explorer.

References:

• POC for adding this flag to the data view and passing it through to relevant Elasticsearch requests: https://github.com/elastic/kibana/pull/183742
• ESQL feature request https://github.com/elastic/elasticsearch/issues/108559

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

[weltenwort]

In order to work well with the Logs Explorer it would also be required for this flag to be represented in the app state. That would be probably be the case anyway, just wanted to mention it.

[davismcphee]

To @weltenwort's point and to make it clear on the Discover end, since failure stores are a general data stream feature and not something specific to Logs Explorer, we've agreed that it makes sense to add support to the core Discover codebase including any necessary additions to the app state.

[flash1293]

Thanks to both of you for highlighting that point - this was my understanding as well, I clarified the description to keep everything in one place for the implementation.

[timductive]

hey @flash1293 (and @ninoslavmiskovic) I'm trying to catch up on this topic. I understand that the current implementation uses query parameters on data-stream search requests to query the failure store and your POC is tying those query params to the data view. However, we may prefer that the ES API is changed to not use query params and support querying "out-of-the-box" like a normal index. Do you have more context on this?

[timductive]

Elasticsearch team is investigating changing this api to be similar to a regular index search.. GET failures-logs-*/_search At which point, the UI for searching failure store should be the same as any other index. Managing/fixing failure store data UI doesn't change, it still needs to modify ingest pipelines, etc..

We need to align on the API before we move forward on the UI work.

[flash1293]

@timductive I wasn't aware of plans to change the API - who is the point of contact about this?

[tylerperk]

@flash1293 - @dakrone and @jpountz and @javanna are looking into it - we'll discuss it at Friday's ES&Obs sync

[ninoslavmiskovic]

Agree here with @timductive to wait and get the API right before exploring the UI.

[gbamparop]

@flash1293 if this will be blocked for a while, we could explore whether it makes sense to surface additional information for the failure store in the dataset quality UI.

[mattkime]

I'm not sure I have full context and this seems worthy of a good pros and cons list but going with a regular index does seem like it would provide wider support from the start, including ESQL. ...but I can't imagine ESQL would be overlooked so I feel like I must be missing something.