[Index Management] Add 'promote' action to replicated data streams

yuliacech commented 4 years ago

Elasticsearch API changes

This PR will add a new 'replicated' field to getDataStream API and a new endpoint to promote a replicated data stream to a regular data stream.

Background

'Replicated' data stream is created by following a data stream in a remote cluster via Cross-Cluster replication. To follow a data stream, a matching auto-follow pattern needs to be created in the CCR app. This pattern can't have any prefix or suffix defined, so that 'replicated' data stream can create backing indices with the same name. Promoting data stream is done when a remote cluster goes down and the 'replicated' data stream needs to be converted to a regular data stream, so it functions instead of the one that is down.

UI changes

In Index Management UI/Data Streams tab, we need to add 'promote' action to the table for replicated data streams. We also need to label replicated data streams accordingly so that users understand why some data streams have this action and some don't.

Also, users need to be able to filter the table for replicated/regular data streams with a toggle to switch table view.

Further considerations

Currently, data streams are not showed in CCR app at all. An auto-follow pattern should be aware if the pattern matches a data stream or an individual index.
There is an error when indexing documents into a follower data stream
Backing indices of a follower data stream are not indicated as those in Index Management app.

elasticmachine commented 4 years ago

Pinging @elastic/es-ui (Team:Elasticsearch UI)

yuliacech commented 4 years ago

I also think we need to change toggles to a view dropdown similar to what we use in Index templates tab.

Currently, there are 2 toggles in data streams table: 'Include stats' and 'Include Fleet-managed data streams'. We are adding another toggle in 83709 for 'hidden' data streams and 'replicated' data streams need a separate toggle too.

martijnvg commented 4 years ago

Given that there is a toggle for hidden data streams, it makes sense to also have a toggle for data streams that are replicated by ccr.

yuliacech commented 4 years ago

There is also a related issue to add data streams support to CCR: #65335. Tagging @cjcenizal @jethr0null for visibility.

cjcenizal commented 4 years ago

The runtime fields project involves a concept called "promotion", in which a runtime field that's defined as part of an index pattern can be "promoted" to be defined on the underlying indices and data streams instead. I'm not sure if this term will make it into the UI, but I think it might be best to play it safe and avoid using this term in the context of this issue.

CCR already uses the terms "follower", "follow", and "unfollow" to describe the relationship between indices on the local and remote clusters. Can we apply the same terms to data streams? For example, you will have "follower data streams" on the local cluster, indicated by a "Follower" badge (already applied to indices), which follow "leader data streams" on the remote cluster, and you can sever this relationship by executing the "Unfollow leader data stream" action (currently possible in the CCR app with follower indices).

Taking this a step further, have we considered mirroring the entire follower-indices UX? The follower-indices UX consists of listing these indices inside of CCR and surfacing the option to unfollow them within that UI. This is complemented by listing the follower indices in the Index Management indices tab, identified with "Follower" badges. Would it make sense to do the same thing with data streams?

yuliacech commented 4 years ago

Thank you @cjcenizal ! That's a great point about possibly mixing concepts from other areas and using terms that our users already know. So do you think it would be better to have a Data Streams tab in CCR that would display follower data streams with an action to 'unfollow' a data stream? From this tab we can cross link to follower-indices table in the same app and to data streams tab in Index Management. That way IM would only need a new badge and a view filter for follower data streams.

sebelga commented 3 years ago

+1 to @cjcenizal idea to mirror what we have done with follower indices. Yes @yuliacech, from what I understand, this implies having a new tab in CCR for follower data streams. 👍

elasticmachine commented 2 months ago

Pinging @elastic/kibana-management (Team:Kibana Management)

elastic / kibana