search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.71k stars 8.13k forks source link

Use Terraform to create/manage Anomaly Jobs #190692

Open ersin-erdal opened 3 weeks ago

ersin-erdal commented 3 weeks ago

Follow on: https://github.com/elastic/response-ops-team/issues/185

It is not possible to create and manage Anomaly Jobs by using Terraform with the providers we currently have (restapi and elasticstack).

restapi can create an anomaly job but fails (conflict error) on the following run of the terraform as there is already an anomaly job. We cannot use delete and create again approach as the anomaly jobs save the process data in an index.

I am creating this issue to discuss about the possible solutions, such as contributing to the elasticstack provider.

@jloleysens @mikecote @jgowdyelastic @elastic/machine-learning @heespi

elasticmachine commented 3 weeks ago

Pinging @elastic/response-ops (Team:ResponseOps)

elasticmachine commented 3 weeks ago

Pinging @elastic/ml-ui (:ml)

jgowdyelastic commented 3 weeks ago

We cannot use delete and create again approach as the anomaly jobs save the process data in an index

When you delete an anomaly detection job using the delete api, the results are also deleted. If you are using a dedicated index for the results, this index will be deleted.

ersin-erdal commented 1 week ago

When you delete an anomaly detection job using the delete api, the results are also deleted. If you are using a dedicated index for the results, this index will be deleted.

Actually this is part of the problem :)

We tried to use our terraform repo to create and manage anomaly jobs in serverless.

Currently there are 2 providers restapi and elasticstack. Unfortunately restapi doesn't support anomaly jobs yet, so we tried to use the restapi in order to benefit from the anomaly detection API

It can create an anomaly job but on the next terraform apply cycle the anomaly job blocks the process because the API returns an error as the anomaly job already exist.

We can't use delete-and-create-back way since the anomaly job is stateful (as you have mentioned)

I created this issue to discuss about the possible solutions like, changing the response of the create endpoint of the anomaly detection API, or contributing to the elasticstack provider to add anomaly job support (so it would not cancel the process on the error response), or adding a new Kibana API etc...