search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.81k stars 8.2k forks source link

Allow users of ES Transform API to request deletion of the destination index when the transform is deleted. #80704

Open nnamdifrankie opened 4 years ago

nnamdifrankie commented 4 years ago

Describe the feature:

Given that I have a Transform running. And transforming data to a destination index. And the Transform has the destination index deletion flag enabled. When I delete the transform successfully Then the destination index should be deleted based on the index deletion flag condition.

Describe a specific use case for the feature:

When Transforms are defined in a Fleet package. It is the responsibility of Fleet to manage the installation, upgrade and removal of the Transform from ElasticSearch based on the future state of the package. It is possible that new version of the transform is not compatible with the documents produced by the older version and as such it is important to start afresh. In addition, when Transforms are removed we want to have the option to remove the destination index if it is not needed. We also have the undesirable case where a Transform has been modified or removed directly from ES using the Transform API, in this case we want the Transform API to follow the guidance of the index deletion flag. Making this change in the Transform API will push the logic to the edges. The user can define how the destination index should behave when the transform is deleted and the Transform API follows that action. The Fleet service is only responsible for installing and removing the Transform and will ignore the absence of the Transform when not present, deferring to the Transform API.

elasticmachine commented 4 years ago

Pinging @elastic/endpoint-management (Team:Endpoint Management)

elasticmachine commented 4 years ago

Pinging @elastic/ingest-management (Feature:Fleet)

nnamdifrankie commented 4 years ago

@elastic/ml-core

ph commented 3 years ago

@kevinlog Could you give us an hand on this? is this is still valid?

kevinlog commented 3 years ago

@ph I assumed it's still valid, but since we delete the indices manually in Fleet, I don't know that it's a high priority.

@pzl do you know if there have been any changes to the Transform API since October that will delete destination indices on it's own?