search

opensearch-project / index-management

🗃 Automate periodic data operations, such as deleting indices at a certain age or performing a rollover at a certain size
https://opensearch.org/docs/latest/im-plugin/index/
Apache License 2.0
53 stars 112 forks source link

[BUG] ISM Error Notification - Not sending request for custom webhook #523

Open xavier-isecurity opened 2 years ago

xavier-isecurity commented 2 years ago

Describe the bug I have an ISM policy working for all my indices on my opensearch environment and I have configured a custom webhook for error notification.

When I force the index to fail, changing the rollover alias to a wrong one, the ISM notification works fine, it sends the notification for my webhook.

But when it fails because of any other issue in Opensearch, it does not send the notification after the state is changed to failed.

There isn't also error logs about the error notification.

To Reproduce Steps to reproduce the behavior:

  1. Create an ISM policy with notification error for custom webhook.
  2. Put heavy load on Opensearch in a point a indice fails when trying to change policy state or execute an action.
  3. After the index is on failed state check if an error notification was send to the webhook.

Expected behavior After the state of the index is changed to failed, in any situation the error notification should send the notification for the custom webhook configured.

Plugins opensearch-alerting opensearch-anomaly-detection opensearch-asynchronous-search opensearch-cross-cluster-replication opensearch-index-management opensearch-job-scheduler opensearch-knn opensearch-ml opensearch-observability opensearch-performance-analyzer opensearch-reports-scheduler opensearch-security opensearch-sql

Host/Environment (please complete the following information):

Angie-Zhang commented 1 year ago

@xavier-isecurity We have tried to reproduce the bug. We discover that when there is some error (i.e. unhealthy index) blocking the policy execution, it can't send the notification. When running docker image, it lacks error messages for sending notification failure. We are going to fix it.