[BUG] Configuring Anomaly Detector

[jmurillo9]

What is the bug?

I am trying to configure an anomaly detector using the latest provider version ( v2.2.0). When I am defining a feature, I can only seem to get my code to work using aggregation_query which results in a custom expression to show up in the UI.

but... when I create a feature manually, I can have the feature show up like so:

I have been trying various variations of utilizing:

  "feature_attributes": [
      {
         "feature_name": "count_client_ip",
         "feature_enabled": true,
         "aggregation_query": {
            "count_client_ip": {
               "value_count": {
               "field": "client_ip.keyword"
               }
            }
         }
      }
  ],

 OR
   "feature_attributes": [
      {
         "feature_name": "count_client_ip",
         "feature_enabled": true,
         "aggregation_method": "value_count",
         "field": "client_ip"
      }
  ],

  etc...

How can one reproduce the bug?

reference Terraform docs regarding the resource in question, copy example snippet and try to deploy.

What is the expected behavior?

The features show up the same in the UI. There seems to be two distinct options to pick from when doing this manually.

What is your host/environment?

OpenSearch_2.9 terraform v1.2.2

Do you have any additional context?

• If the two variations are interchangeable, then can someone please provide a working query that I can use that would be the same as what works when doing this manually.

• If someone can provide an example of how to properly add categorical_fields as well, that would be greatly appreciated. I cannot seem to get that to work (again because the resource is expecting aggregation_query. I tried doing:

      "aggregation_query": {
          "category_field_1": {
             "terms": {
                "field": "category_field_1.keyword"
             }
          }
      }

[prudhvigodithi]

[Triage] Hey @jmurillo9 thanks for opening the issue, do you see this bug when using provider or also noticed when also using OpenSearch API ? You mentioned manually works from dashboard, can you please share your tf file or query etc, for us to re produce ? Adding @rblcoder @bbarani Thanks

[jmurillo9]

Hello @prudhvigodithi - I personally haven't tried using the OpenSearch API. I just noticed a difference in what shows up in the UI when you point and click versus when you use the Terraform provider ( v2.2.0). The visual comparison was just throwing me off at first.

terraform {
   required_providers {
      opensearch = {
        source = "opensearch-project/opensearch"
        version = "2.2.0"
      }
   }
}

Complete Terraform code from the snippet I posted above:

resource "opensearch_anomaly_detection" "this" {
  body = <<EOF
{
  "name": "my-awesome-detector",
  "description": "An anomaly detector for ingress logs created via Terraform.",
  "time_field": "@timestamp",
  "result_index" : "opensearch-ad-plugin-result-my-awesome-detector",
  "indices": [
    "*ingress*"
  ],
  "feature_attributes": [
      {
         "feature_name": "count_client_ip",
         "feature_enabled": true,
         "aggregation_query": {
            "count_client_ip": {
               "value_count": {
               "field": "client_ip.keyword"
               }
            }
         }
      }
  ],
  "filter_query": {
      "bool": {
      "filter": [
         {
            "range": {
               "value": {
                  "gt": 1
               }
            }
         }
      ],
      "adjust_pure_negative": true,
      "boost": 1
      }
  },
   "detection_interval": {
      "period": {
         "interval": 10,
         "unit": "Minutes"
      }
   },
   "window_delay": {
      "period": {
         "interval": 1,
         "unit": "Minutes"
      }
   }
}
EOF
}

[rblcoder]

@jmurillo9 Creating an anomaly detector using

terraform {
  required_providers {
    opensearch = {
      source = "opensearch-project/opensearch"
      version = "2.2.1"
    }
  }
}

provider "opensearch" {
  url = "url"
  healthcheck        = "false"
  aws_region          = "region"
  version_ping_timeout = "10"
  sign_aws_requests = "false"
  username          = "username"
  password          = "password"
}

resource "opensearch_anomaly_detection" "test-detector12" {
body       = <<EOF
{

    "name": "sample-http-responses-detector2",
    "description": "A sample detector to detect anomalies with HTTP response code logs.",
    "time_field": "timestamp",
    "indices": [
      "sample-http-responses"
    ],
    "filter_query": {
      "match_all": {
        "boost": 1
      }
    },
    "detection_interval": {
      "period": {
        "interval": 10,
        "unit": "Minutes"
      }
    },
    "window_delay": {
      "period": {
        "interval": 1,
        "unit": "Minutes"
      }
    },
    "shingle_size": 8,
    "schema_version": 0,
    "feature_attributes": [
      {
        "feature_id": "8Z6-oo4BhbT1HUOvhfMe",
        "feature_name": "sum_http_4xx",
        "feature_enabled": true,
        "aggregation_query": {
          "sum_http_4xx": {
            "sum": {
              "field": "http_4xx"
            }
          }
        }
      },
      {
        "feature_id": "8p6-oo4BhbT1HUOvhfMl",
        "feature_name": "sum_http_5xx",
        "feature_enabled": true,
        "aggregation_query": {
          "sum_http_5xx": {
            "sum": {
              "field": "http_5xx"
            }
          }
        }
      }
    ],
    "ui_metadata": {
      "features": {
        "sum_http_5xx": {
          "aggregationBy": "sum",
          "aggregationOf": "http_5xx",
          "featureType": "simple_aggs"
        },
        "sum_http_4xx": {
          "aggregationBy": "sum",
          "aggregationOf": "http_4xx",
          "featureType": "simple_aggs"
        }
      },
      "filters": []
    },
    "last_update_time": 1712127380464,
    "user": {
      "name": "ce80y7khowl5",
      "backend_roles": [],
      "roles": [
        "security_manager",
        "all_access"
      ],
      "custom_attribute_names": [],
      "user_requested_tenant": null
    },
    "detector_type": "SINGLE_ENTITY"

}
EOF
}

when I check the configuration in OpenSearch Dashboard

[jmurillo9]

I'll have to give this a try. Thanks @rblcoder !