It is questionable whether this should be a dynamic setting as this documentation states that:
You can no longer configure Slack accounts using elasticsearch.yml settings. Please use Elasticsearch’s secure [keystore](https://www.elastic.co/guide/en/elasticsearch/reference/current/secure-settings.html) method instead.
Notice as well that in this below example, the default account value contains spaces. We could not find any workaround to recover from this situation (AFAIK it is impossible to configure the account in the elasticsearch.yml file or to define the secure URL in the keystore while the account name has a space in it)
Steps to Reproduce
Create a deployment in ESS with 2AZ for the hot data and content tier
Run the below API and notice that the API is successfully acknowledged:
PUT _cluster/settings
{
"persistent": {
"xpack.notification.slack.default_account": "Slack Alerts"
}
}
Run GET _cluster/settings and observe that xpack.notification.slack.default_account is not in the result
Run the below API to reset the setting:
PUT _cluster/settings
{
"persistent": {
"xpack.notification.slack.default_account": ""
}
}
[tiebreaker-0000000002] failed to apply settings org.elasticsearch.common.settings.SettingsException: could not find default account [Slack Alerts] at org.elasticsearch.xpack.watcher.notification.NotificationService.findDefaultAccountOrNull(NotificationService.java:178) ~[?:?] at org.elasticsearch.xpack.watcher.notification.NotificationService.buildAccounts(NotificationService.java:107) ~[?:?] at org.elasticsearch.xpack.watcher.notification.NotificationService.clusterSettingsConsumer(NotificationService.java:77) ~[?:?] at org.elasticsearch.common.settings.Setting$2.apply(Setting.java:850) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.common.settings.Setting$2.apply(Setting.java:822) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.common.settings.AbstractScopedSettings$SettingUpdater.lambda$updater$0(AbstractScopedSettings.java:654) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.common.settings.AbstractScopedSettings.applySettings(AbstractScopedSettings.java:174) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:498) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:432) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:156) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:917) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:217) ~[elasticsearch-8.15.3.jar:?] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:183) ~[elasticsearch-8.15.3.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.lang.Thread.run(Thread.java:1570) ~[?:?]
Elasticsearch Version
8.15.3
Installed Plugins
No response
Java Version
bundled
OS Version
N/A
Problem Description
This has been observed in the field and the problem is reproducible.
Updating
xpack.notification.slack.default_account
(c.f Slack Notification Settings) to an account that does not exist via the cluster update settings API leads to a dead cluster (c.f steps to reproduce below)It is questionable whether this should be a dynamic setting as this documentation states that:
Notice as well that in this below example, the default account value contains spaces. We could not find any workaround to recover from this situation (AFAIK it is impossible to configure the account in the elasticsearch.yml file or to define the secure URL in the keystore while the account name has a space in it)
Steps to Reproduce
Create a deployment in ESS with 2AZ for the hot data and content tier
Run the below API and notice that the API is successfully acknowledged:
Run
GET _cluster/settings
and observe thatxpack.notification.slack.default_account
is not in the resultRun the below API to reset the setting:
and observe the below error:
a) Master node keeps changing (c.f
master node changed
event logs)b) All nodes are reporting similar log message
Logs (if relevant)
No response