[BUG] Creating anomaly detector stucks at initializing state

[eslam-gomaa]

Describe the bug Hi I believe that this is not a bug, I'm just asking for guidance

When creating anomaly detector, it stuck at the initializing state for many days ! (some stuck for infinity) After a bit of investigation I found that this only happens when the index pattern used with the detector has very few events steam (due to the filter used with the detector)

Is there specific configuration I need to consider in my case ? 🤔

GET _opendistro/_anomaly_detection/detectors/vmwCWoEBNq3GYhZRwdMX/_profile?_all

{
  "state" : "INIT",
  "total_size_in_bytes" : 0,
  "init_progress" : {
    "percentage" : "0%",
    "estimated_minutes_left" : 256,
    "needed_shingles" : 128
  },
  "total_entities" : 911,
  "active_entities" : 0
}

To Reproduce Create an anomaly detectors on index pattern that has few events steam.

Expected behavior The detector to finish initialization successfully and becomes "running"

Plugins anomaly detection

GET _opendistro/_anomaly_detection/stats

{
  "anomaly_detectors_index_status" : "green",
  "anomaly_detection_state_status" : "green",
  "detector_count" : 19,
  "anomaly_detection_job_index_status" : "green",
  "models_checkpoint_index_status" : "green",
.....

Screenshots

sometimes to gets up to 2% and back to 0 again

Host/Environment (please complete the following information):

• AWS Opensearch service with Elasticsearch 7.10.2

---

If more information is needed pls let me know

Thanks in advance

[kaituo]

AD requires large sample sizes to create a comprehensive picture of the data patterns, making it suitable for dense time series that can be uniformly sampled. The detector gets stuck at initializing state due to sparse data and we cannot find continuous data patterns to train our models.

In the short term, you can try 1)using latest OpenSearch version (2.0.1 in OpenSearch or 1.2 in AWS service) 2)try increasing the interval to 1 hr

In the long term, we will look at speeding up model initialization of sparse data. For example, we can enable users to provide default value when a value is missing (say 0) and allowing specify cold start data range.

[zealsprince]

I would second this issue when working with sparse data sets even if it is for just debugging and rudimentary analysis purposes.

Being able to fill with empty events to pad out data, moving past the initialization step would be much appreciated.

[kaituo]

zealsprince@, yes, the improvement is in our plan