Open tantra35 opened 3 years ago
Little update, we fully recreate all policies in production cluster, but result was discouraging: policies that were previously applied without problems have ceased to be applied, and vice versa. Result is fully random:
For example policy
{
"policy_id": "ci",
"description": "Политика для индексов ci-*",
"last_updated_time": 1627034284633,
"schema_version": 1,
"error_notification": null,
"default_state": "search",
"states": [
{
"name": "search",
"actions": [],
"transitions": [
{
"state_name": "delete",
"conditions": {
"min_index_age": "30d"
}
}
]
},
{
"name": "delete",
"actions": [
{
"delete": {}
}
],
"transitions": []
}
],
"ism_template": {
"index_patterns": [
"ci-*"
],
"priority": 100,
"last_updated_time": 1627034284633
}
}
before recreation works as expected and was applied to indexes like ci-
, after recreation all new created indexes doesn't hold any policy, this strange behavior happens in different combinations
Seems we found the problem:
index management request templates from .opendistro-ism-config
index and templates returned by elasticsearch request similar to this:
curl -s --insecure --cert ../usercert-admin.crt --key ../usercert-admin-key.pem 'https://securedelklogsv2-input.service.consul:9200/.opendistro-ism-config/_search?pretty' -H'Content-Type: application/json' -d' {
"query": {
"exists": {
"field": "policy.ism_template"
}
}
}'; echo
but this request returns only 10 results by default https://www.elastic.co/guide/en/elasticsearch/reference/7.x/paginate-search-results.html
and in version 1.13.2
opendistro have buggy request in
https://github.com/opendistro-for-elasticsearch/index-management/blob/v1.13.2.0/src/main/kotlin/com/amazon/opendistroforelasticsearch/indexmanagement/indexstatemanagement/ManagedIndexCoordinator.kt#L306
in mainstream of deveplopment of this plugin problem was fixed
How you fixed this problem?
we just apply our patch аnd rebuild securityplugin
patch
From 22650245885aa9842e6d2a776420039c4a65af5f Mon Sep 17 00:00:00 2001
From: tantra35 <ruslan.usifov@gmail.com>
Date: Wed, 4 Aug 2021 22:38:39 +0300
Subject: [PATCH] fix wrong ism managment
---
.../indexstatemanagement/ManagedIndexCoordinator.kt | 2 +-
.../action/indexpolicy/TransportIndexPolicyAction.kt | 5 ++++-
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/src/main/kotlin/com/amazon/opendistroforelasticsearch/indexmanagement/indexstatemanagement/ManagedIndexCoordinator.kt b/src/main/kotlin/com/amazon/opendistroforelasticsearch/indexmanagement/indexstatemanagement/ManagedIndexCoordinator.kt
index a94b82a..64a586d 100644
--- a/src/main/kotlin/com/amazon/opendistroforelasticsearch/indexmanagement/indexstatemanagement/ManagedIndexCoordinator.kt
+++ b/src/main/kotlin/com/amazon/opendistroforelasticsearch/indexmanagement/indexstatemanagement/ManagedIndexCoordinator.kt
@@ -308,7 +308,7 @@ class ManagedIndexCoordinator(
.source(
SearchSourceBuilder().query(
QueryBuilders.existsQuery(ISM_TEMPLATE_FIELD)
- )
+ ).size(MAX_HITS)
)
.indices(INDEX_MANAGEMENT_INDEX)
diff --git a/src/main/kotlin/com/amazon/opendistroforelasticsearch/indexmanagement/indexstatemanagement/transport/action/indexpolicy/TransportIndexPolicyAction.kt b/src/main/kotlin/com/amazon/opendistroforelasticsearch/indexmanagement/indexstatemanagement/transport/action/indexpolicy/TransportIndexPolicyAction.kt
index 6e48cb6..4bb1280 100644
--- patch
a/src/main/kotlin/com/amazon/opendistroforelasticsearch/indexmanagement/indexstatemanagement/transport/action/indexpolicy/TransportIndexPolicyAction.kt
+++ b/src/main/kotlin/com/amazon/opendistroforelasticsearch/indexmanagement/indexstatemanagement/transport/action/indexpolicy/TransportIndexPolicyAction.kt
@@ -17,6 +17,7 @@ package com.amazon.opendistroforelasticsearch.indexmanagement.indexstatemanageme
import com.amazon.opendistroforelasticsearch.indexmanagement.IndexManagementIndices
import com.amazon.opendistroforelasticsearch.indexmanagement.IndexManagementPlugin
+import com.amazon.opendistroforelasticsearch.indexmanagement.indexstatemanagement.ManagedIndexCoordinator.Companion.MAX_HITS
import com.amazon.opendistroforelasticsearch.indexmanagement.indexstatemanagement.elasticapi.filterNotNullValues
import com.amazon.opendistroforelasticsearch.indexmanagement.indexstatemanagement.elasticapi.getPolicyToTemplateMap
import com.amazon.opendistroforelasticsearch.indexmanagement.indexstatemanagement.findConflictingPolicyTemplates
@@ -107,7 +108,9 @@ class TransportIndexPolicyAction @Inject constructor(
val searchRequest = SearchRequest()
.source(
SearchSourceBuilder().query(
- QueryBuilders.existsQuery(ISM_TEMPLATE_FIELD)))
+ QueryBuilders.existsQuery(ISM_TEMPLATE_FIELD)
+ ).size(MAX_HITS)
+ )
.indices(IndexManagementPlugin.INDEX_MANAGEMENT_INDEX)
client.search(searchRequest, object : ActionListener<SearchResponse> {
--
2.28.0.windows.1
but later aws fix this, but not rebuild plugin due they migrate to opensearch
In out production cluster based on opendistro [1.13.2] we have multiple Index Management Index Policies with ism_template defined field, where we declare index_patterns. For example
this policy must be applied to any newly created index, that doesn’t match index patterns in other index policies, but nothing happens, and to that indexes doens’t have any policy applied
On test stand where configured much less index policies - all working, but in production does not, but in production cluster we have only about 20 policies
How this can be, and how we can solve this problem?