elastic / elasticsearch

Free and Open, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.53k stars 24.61k forks source link

[ML] Configure anomaly detection time shift using scheduled events #111634

Open valeriy42 opened 1 month ago

valeriy42 commented 1 month ago

Description

Daylight saving time events are causing a burst of false positives in anomaly detection jobs due to a sudden time shift. To remedy the situation, we allow users to configure an explicit time shift for detectors using customer_rule action force_time_shift implemented in https://github.com/elastic/elasticsearch/pull/110974.

However, since the DST events occur regularly and multiple jobs are usually affected, we need to allow users to control the explicit time shift using scheduled events.

To this end, we need to extend the POST calendars API to allow fine grade configuration:

POST _ml/calendars/dst-germany
{
    "events" : [
        {"description": "Summer 2025 ", "start_time": 1513814400000, "end_time": 1513818000000, skip_results: true, skip_model_update: false, force_time_shift:3600},
{"description": "Winter 2025", "start_time": 1514160000000, "end_time": 1514163600000, skip_results: true, skip_model_update: false, force_time_shift:-3600}
    ]
}

Internally, the calendar events should be converted to detection rules as it happens already, but with finer configuration.

Additionally, the detection rules corresponding to the past calendar events should be removed from the job configuration, in order to keep the configuration of long-running jobs slim.

elasticsearchmachine commented 1 month ago

Pinging @elastic/ml-core (Team:ML)