Open brsolomon-deloitte opened 1 year ago
Contrary to https://github.com/elastic/kibana/issues/139969 I have verified the alerts are not in cold/frozen/snapshot phase.
{
"indices": {
".internal.alerts-security.alerts-default-000001": {
"index": ".internal.alerts-security.alerts-default-000001",
"managed": true,
"policy": ".alerts-ilm-policy",
"index_creation_date_millis": 1672866669277,
"time_since_index_creation": "20.52h",
"lifecycle_date_millis": 1672866669277,
"age": "20.52h",
"phase": "hot",
"phase_time_millis": 1672866669344,
"action": "rollover",
"action_time_millis": 1672866669344,
"step": "check-rollover-ready",
"step_time_millis": 1672866669344,
"phase_execution": {
"policy": ".alerts-ilm-policy",
"phase_definition": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_primary_shard_size": "50gb",
"max_age": "30d"
}
}
},
"version": 3,
"modified_date_in_millis": 1672843617932
}
}
}
}
Pinging @elastic/kibana-security (Team:Security)
Pinging @elastic/security-solution (Team: SecuritySolution)
Pinging @elastic/security-detections-response (Team:Detections and Resp)
@brsolomon-deloitte the 3 posted log lines are likely a red herring, and the relevant log entry likely higher up in the file. Generally I've seen this when resources fail to install for some reason https://github.com/elastic/kibana/blob/main/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.ts#L96 perhaps try to look for the error message seen on this line https://github.com/elastic/kibana/blob/b7ff3549ec30f7babd60ca876911aceaf667d099/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.ts#L120 otherwise set the logging level to debug and restart kibana, looking for some of the log messages in that file. Error has nothing to do with the rule definition itself, and is happening before it's ever defined.
deleting preview index and restarting kibana fixed this. will it be long term not sure, but everything finally works again.
using version 8.5.3
happens in 8.6.2 too. different cluster too. It's all to do with the preview index & the way Kibana gets restarted either during upgrades/outages/etc or just any sort of flakey ILM issue. Here is another related error for anyone else who ends up here.
Bulk Indexing of signals failed: RuleDataWriterInitializationError: There has been a catastrophic error trying to install index level resources for the following registration context: security. This may have been due to a non-additive change to the mappings, removal and type changes are not permitted. Full error: Error: Failure installing common resources shared between all indices. process_cluster_event_timeout_exception: [process_cluster_event_timeout_exception] Reason: failed to process cluster event (put-lifecycle-.alerts-ilm-policy) within 30s
Same fix, restart Kibana. Trying to force an ILM step did not work either.
Kibana version: 8.5.3
Elasticsearch version: 8.5.3
Browser version: Chrome 108.0.5359.124
Original install method (e.g. download page, yum, from source, etc.): ECK
Describe the bug: Custom rule fails to run with
Bulk Indexing of signals failed: RuleDataWriteDisabledError: Rule registry writing is disabled due to an error during Rule Data Client initialization.
Steps to reproduce:
suricata
moduleFailed
runs with zero useful debugging informationExpected behavior: Rule runs successfully.
Screenshots (if relevant):
Provide logs and/or server output (if relevant):
Kibana logs:
The rule in question
Fetched with
GET kbn:/api/alerting/rules/_find?search_fields=name&search=Suricata*
.