cisagov / LME

Logging Made Easy (LME) is a no cost, open source platform that centralizes log collection, enhances threat detection, and enables real-time alerting, helping small to medium-sized organizations secure their infrastructure.
https://www.cisa.gov/resources-tools/services/logging-made-easy
Other
889 stars 72 forks source link

[BUG] ERROR Failed to create the lists index in step 4.2 #437

Closed khamilton479 closed 2 months ago

khamilton479 commented 2 months ago

BEFORE CREATING THE ISSUE, CHECK THE FOLLOWING GUIDES:

If the above did not answer your question, proceed with creating an issue below:

Describe the bug

On a fresh install of LME 1.4 Logged in with the built in superuser "elastic" Receive the following error when trying to go to the Alerts page under Security as described in step 4.2 Enable Alerts.

Failed to create the lists index illegal_argument_exception: matching index template [number_of_replicas] for data stream [.lists-default] has no data stream template (400) { "name": "Error", "body": { "message": "illegal_argument_exception: matching index template [number_of_replicas] for data stream [.lists-default] has no data stream template", "status_code": 400 }, "message": "Bad Request", "stack": "Error: Bad Request\n at fetch_Fetch.fetchResponse (https://fpl-elk/68203/bundles/core/core.entry.js:1:276857)\n at async https://fpl-elk/68203/bundles/core/core.entry.js:1:274790\n at async https://fpl-elk/68203/bundles/core/core.entry.js:1:274747" }

Expected behavior

For the Alerts page to load with the Manage rules button visible.

To Reproduce

Install LME 1.4 Log into Elastic using the built in superuser "elastic" Go to Alerts page under Security.

Please complete the following information

Setup

Desktop: (Client Machines)

Domain Controller:

ElasticSearch/Kibana Server:

OPTIONAL:

df -h

Filesystem Size Used Avail Use% Mounted on tmpfs 3.2G 1.4M 3.2G 1% /run /dev/mapper/ubuntu--vg-ubuntu--lv 456G 15G 418G 4% / tmpfs 16G 0 16G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock /dev/vda2 2.0G 131M 1.7G 8% /boot tmpfs 3.2G 4.0K 3.2G 1% /run/user/1000

uname -a

Linux fpl-elk 5.15.0-121-generic #131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a

Distributor ID: Ubuntu Description: Ubuntu 22.04.5 LTS Release: 22.04 Codename: jammy

- Relevant container logs: 

for name in $(sudo docker ps -a --format '{{.Names}}'); do echo -e "\n\n\n-----------$name----------"; sudo docker logs $name | tail -n 20; done

-----------lme_kibana.1.031s779qkt6am51vfdtvx45jp---------- [2024-09-13T14:39:41.284+00:00][INFO ][plugins.observability] Installing SLO summary transform [slo-summary-timeslices-90d-rolling] [2024-09-13T14:39:41.792+00:00][INFO ][plugins.observability] Starting SLO summary transform [slo-summary-timeslices-90d-rolling] [2024-09-13T14:39:41.906+00:00][INFO ][plugins.observability] Installing SLO summary transform [slo-summary-timeslices-weekly-aligned] [2024-09-13T14:39:42.006+00:00][INFO ][plugins.observability] Starting SLO summary transform [slo-summary-timeslices-weekly-aligned] [2024-09-13T14:39:42.166+00:00][INFO ][plugins.observability] Installing SLO summary transform [slo-summary-timeslices-monthly-aligned] [2024-09-13T14:39:42.260+00:00][INFO ][plugins.observability] Starting SLO summary transform [slo-summary-timeslices-monthly-aligned] [2024-09-13T14:39:42.413+00:00][INFO ][plugins.observability] SLO summary transforms installed and started [2024-09-13T14:39:42.631+00:00][INFO ][plugins.synthetics] Installed synthetics index templates [2024-09-13T14:44:21.513+00:00][INFO ][plugins.securitySolution] Fetch risk engine metrics [2024-09-13T14:52:27.526+00:00][INFO ][plugins.security.routes] Logging in with provider "basic" (basic) [2024-09-13T14:52:53.137+00:00][INFO ][plugins.fleet] Beginning fleet setup [2024-09-13T14:52:53.839+00:00][INFO ][plugins.fleet] Fleet setup completed [2024-09-13T14:53:21.702+00:00][INFO ][plugins.fleet] Beginning fleet setup [2024-09-13T14:53:22.263+00:00][INFO ][plugins.fleet] Fleet setup completed [2024-09-13T14:54:28.456+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"num_host_urls":0}} [2024-09-13T15:09:28.526+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"num_host_urls":0}} [2024-09-13T15:24:31.605+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"num_host_urls":0}} [2024-09-13T15:39:28.592+00:00][INFO ][plugins.fleet] Running Fleet Usage telemetry send task [2024-09-13T15:39:31.604+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"num_host_urls":0}} [2024-09-13T15:54:31.662+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"num_host_urls":0}}

-----------lme_elasticsearch.1.x4ul4b21j2ouvx6cqqdyz7rgl---------- Sep 13, 2024 2:38:35 PM sun.util.locale.provider.LocaleProviderAdapter WARNING: COMPAT locale provider will be removed in a future release {"@timestamp":"2024-09-13T14:58:50.489Z", "log.level": "INFO", "message":"Finished execution of ESQL query.\nQuery string: [from .alerts-security.alerts-default,apm--transaction,auditbeat-,endgame-,filebeat-,logs-,packetbeat-,traces-apm,winlogbeat-,-elastic-cloud-logs- | limit 10]\nExecution time: [377]ms", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][esql_worker][T#1]","log.logger":"org.elasticsearch.xpack.esql.action.EsqlResponseListener","trace.id":"da0d4e784b9b6f4158c84b9d7e91ce9c","elasticsearch.cluster.uuid":"K0__JHLnRwq8fzCcvC6fwg","elasticsearch.node.id":"e8KC0YiIQMyxw9TyTW5dRA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"} {"@timestamp":"2024-09-13T14:59:13.231Z", "log.level": "INFO", "message":"Beginning execution of ESQL query.\nQuery string: [from .alerts-security.alerts-default,apm--transaction,auditbeat-,endgame-,filebeat-,logs-,packetbeat-,traces-apm,winlogbeat-,-elastic-cloud-logs- | limit 10]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][transport_worker][T#1]","log.logger":"org.elasticsearch.xpack.esql.action.RestEsqlQueryAction","trace.id":"15af53b91fb5bc93eb60e87f6f84aa76","elasticsearch.cluster.uuid":"K0JHLnRwq8fzCcvC6fwg","elasticsearch.node.id":"e8KC0YiIQMyxw9TyTW5dRA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"} {"@timestamp":"2024-09-13T14:59:13.279Z", "log.level": "INFO", "message":"Beginning execution of ESQL query.\nQuery string: [from .alerts-security.alerts-default,apm--transaction,auditbeat-,endgame-,filebeat-,logs-,packetbeat-,traces-apm,winlogbeat-,-elastic-cloud-logs-* | limit 10]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][transport_worker][T#2]","log.logger":"org.elasticsearch.xpack.esql.action.RestEsqlQueryAction","trace.id":"e5cc3464a2f462c32f61fa78ed917664","elasticsearch.cluster.uuid":"K0JHLnRwq8fzCcvC6fwg","elasticsearch.node.id":"e8KC0YiIQMyxw9TyTW5dRA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"} {"@timestamp":"2024-09-13T14:59:13.320Z", "log.level": "INFO", "message":"Beginning execution of ESQL query.\nQuery string: [from .alerts-security.alerts-default,apm--transaction,auditbeat-,endgame-,filebeat-,logs-,packetbeat-,traces-apm,winlogbeat-,-elastic-cloud-logs- | limit 10]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][transport_worker][T#1]","log.logger":"org.elasticsearch.xpack.esql.action.RestEsqlQueryAction","trace.id":"bdfbb94a8d82bdec36ec9be63b0b21ce","elasticsearch.cluster.uuid":"K0__JHLnRwq8fzCcvC6fwg","elasticsearch.node.id":"e8KC0YiIQMyxw9TyTW5dRA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"} {"@timestamp":"2024-09-13T14:59:14.009Z", "log.level": "INFO", "message":"Finished execution of ESQL query.\nQuery string: [from .alerts-security.alerts-default,apm--transaction,auditbeat-,endgame-,filebeat-,logs-,packetbeat-,traces-apm,winlogbeat-,-elastic-cloud-logs- | limit 10]\nExecution time: [777]ms", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][esql][T#1]","log.logger":"org.elasticsearch.xpack.esql.action.EsqlResponseListener","trace.id":"15af53b91fb5bc93eb60e87f6f84aa76","elasticsearch.cluster.uuid":"K0JHLnRwq8fzCcvC6fwg","elasticsearch.node.id":"e8KC0YiIQMyxw9TyTW5dRA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"} {"@timestamp":"2024-09-13T14:59:14.036Z", "log.level": "INFO", "message":"Finished execution of ESQL query.\nQuery string: [from .alerts-security.alerts-default,apm--transaction,auditbeat-,endgame-,filebeat-,logs-,packetbeat-,traces-apm,winlogbeat-,-elastic-cloud-logs- | limit 10]\nExecution time: [756]ms", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][esql][T#1]","log.logger":"org.elasticsearch.xpack.esql.action.EsqlResponseListener","trace.id":"e5cc3464a2f462c32f61fa78ed917664","elasticsearch.cluster.uuid":"K0__JHLnRwq8fzCcvC6fwg","elasticsearch.node.id":"e8KC0YiIQMyxw9TyTW5dRA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"} {"@timestamp":"2024-09-13T14:59:14.142Z", "log.level": "INFO", "message":"Finished execution of ESQL query.\nQuery string: [from .alerts-security.alerts-default,apm--transaction,auditbeat-,endgame-,filebeat-,logs-,packetbeat-,traces-apm,winlogbeat-,-elastic-cloud-logs- | limit 10]\nExecution time: [821]ms", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][esql][T#2]","log.logger":"org.elasticsearch.xpack.esql.action.EsqlResponseListener","trace.id":"bdfbb94a8d82bdec36ec9be63b0b21ce","elasticsearch.cluster.uuid":"K0JHLnRwq8fzCcvC6fwg","elasticsearch.node.id":"e8KC0YiIQMyxw9TyTW5dRA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"} {"@timestamp":"2024-09-13T14:59:14.341Z", "log.level": "INFO", "message":"Beginning execution of ESQL query.\nQuery string: [from .alerts-security.alerts-default,apm--transaction,auditbeat-,endgame-,filebeat-,logs-,packetbeat-,traces-apm,winlogbeat-,-elastic-cloud-logs- | limit 10]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][transport_worker][T#1]","log.logger":"org.elasticsearch.xpack.esql.action.RestEsqlQueryAction","trace.id":"7c8633670e2d388bbc94fe1433d7705a","elasticsearch.cluster.uuid":"K0__JHLnRwq8fzCcvC6fwg","elasticsearch.node.id":"e8KC0YiIQMyxw9TyTW5dRA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"} {"@timestamp":"2024-09-13T14:59:14.682Z", "log.level": "INFO", "message":"Finished execution of ESQL query.\nQuery string: [from .alerts-security.alerts-default,apm--transaction,auditbeat-,endgame-,filebeat-,logs-,packetbeat-,traces-apm,winlogbeat-,-elastic-cloud-logs- | limit 10]\nExecution time: [337]ms", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][esql][T#2]","log.logger":"org.elasticsearch.xpack.esql.action.EsqlResponseListener","trace.id":"7c8633670e2d388bbc94fe1433d7705a","elasticsearch.cluster.uuid":"K0JHLnRwq8fzCcvC6fwg","elasticsearch.node.id":"e8KC0YiIQMyxw9TyTW5dRA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"} {"@timestamp":"2024-09-13T14:59:52.100Z", "log.level": "INFO", "message":"Beginning execution of ESQL query.\nQuery string: [from .alerts-security.alerts-default,apm--transaction,auditbeat-,endgame-,filebeat-,logs-,packetbeat-,traces-apm,winlogbeat-,-elastic-cloud-logs-* | limit 10]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][transport_worker][T#2]","log.logger":"org.elasticsearch.xpack.esql.action.RestEsqlQueryAction","trace.id":"367dbddfdfd17909b3f3abb564ac6550","elasticsearch.cluster.uuid":"K0JHLnRwq8fzCcvC6fwg","elasticsearch.node.id":"e8KC0YiIQMyxw9TyTW5dRA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"} {"@timestamp":"2024-09-13T14:59:52.161Z", "log.level": "INFO", "message":"Beginning execution of ESQL query.\nQuery string: [from .alerts-security.alerts-default,apm--transaction,auditbeat-,endgame-,filebeat-,logs-,packetbeat-,traces-apm,winlogbeat-,-elastic-cloud-logs- | limit 10]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][transport_worker][T#2]","log.logger":"org.elasticsearch.xpack.esql.action.RestEsqlQueryAction","trace.id":"192ff0c16e7fed8ec18f1cdea9b0b556","elasticsearch.cluster.uuid":"K0__JHLnRwq8fzCcvC6fwg","elasticsearch.node.id":"e8KC0YiIQMyxw9TyTW5dRA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"} {"@timestamp":"2024-09-13T14:59:52.185Z", "log.level": "INFO", "message":"Beginning execution of ESQL query.\nQuery string: [from .alerts-security.alerts-default,apm--transaction,auditbeat-,endgame-,filebeat-,logs-,packetbeat-,traces-apm,winlogbeat-,-elastic-cloud-logs- | limit 10]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][transport_worker][T#2]","log.logger":"org.elasticsearch.xpack.esql.action.RestEsqlQueryAction","trace.id":"56ddeec61b7241118bbb3520dd0d705c","elasticsearch.cluster.uuid":"K0JHLnRwq8fzCcvC6fwg","elasticsearch.node.id":"e8KC0YiIQMyxw9TyTW5dRA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"} {"@timestamp":"2024-09-13T14:59:52.616Z", "log.level": "INFO", "message":"Finished execution of ESQL query.\nQuery string: [from .alerts-security.alerts-default,apm--transaction,auditbeat-,endgame-,filebeat-,logs-,packetbeat-,traces-apm,winlogbeat-,-elastic-cloud-logs- | limit 10]\nExecution time: [452]ms", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][esql][T#1]","log.logger":"org.elasticsearch.xpack.esql.action.EsqlResponseListener","trace.id":"192ff0c16e7fed8ec18f1cdea9b0b556","elasticsearch.cluster.uuid":"K0__JHLnRwq8fzCcvC6fwg","elasticsearch.node.id":"e8KC0YiIQMyxw9TyTW5dRA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"} {"@timestamp":"2024-09-13T14:59:52.870Z", "log.level": "INFO", "message":"Finished execution of ESQL query.\nQuery string: [from .alerts-security.alerts-default,apm--transaction,auditbeat-,endgame-,filebeat-,logs-,packetbeat-,traces-apm,winlogbeat-,-elastic-cloud-logs- | limit 10]\nExecution time: [685]ms", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][esql][T#1]","log.logger":"org.elasticsearch.xpack.esql.action.EsqlResponseListener","trace.id":"56ddeec61b7241118bbb3520dd0d705c","elasticsearch.cluster.uuid":"K0JHLnRwq8fzCcvC6fwg","elasticsearch.node.id":"e8KC0YiIQMyxw9TyTW5dRA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"} {"@timestamp":"2024-09-13T14:59:53.049Z", "log.level": "INFO", "message":"Finished execution of ESQL query.\nQuery string: [from .alerts-security.alerts-default,apm--transaction,auditbeat-,endgame-,filebeat-,logs-,packetbeat-,traces-apm,winlogbeat-,-elastic-cloud-logs- | limit 10]\nExecution time: [945]ms", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][esql][T#2]","log.logger":"org.elasticsearch.xpack.esql.action.EsqlResponseListener","trace.id":"367dbddfdfd17909b3f3abb564ac6550","elasticsearch.cluster.uuid":"K0__JHLnRwq8fzCcvC6fwg","elasticsearch.node.id":"e8KC0YiIQMyxw9TyTW5dRA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"} {"@timestamp":"2024-09-13T14:59:53.080Z", "log.level": "INFO", "message":"Beginning execution of ESQL query.\nQuery string: [from .alerts-security.alerts-default,apm--transaction,auditbeat-,endgame-,filebeat-,logs-,packetbeat-,traces-apm,winlogbeat-,-elastic-cloud-logs- | limit 10]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][transport_worker][T#2]","log.logger":"org.elasticsearch.xpack.esql.action.RestEsqlQueryAction","trace.id":"1da43253d8ab66f96cff4954c7ace881","elasticsearch.cluster.uuid":"K0JHLnRwq8fzCcvC6fwg","elasticsearch.node.id":"e8KC0YiIQMyxw9TyTW5dRA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"} {"@timestamp":"2024-09-13T14:59:53.434Z", "log.level": "INFO", "message":"Finished execution of ESQL query.\nQuery string: [from .alerts-security.alerts-default,apm--transaction,auditbeat-,endgame-,filebeat-,logs-,packetbeat-,traces-apm,winlogbeat-,-elastic-cloud-logs-* | limit 10]\nExecution time: [353]ms", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][esql][T#1]","log.logger":"org.elasticsearch.xpack.esql.action.EsqlResponseListener","trace.id":"1da43253d8ab66f96cff4954c7ace881","elasticsearch.cluster.uuid":"K0__JHLnRwq8fzCcvC6fwg","elasticsearch.node.id":"e8KC0YiIQMyxw9TyTW5dRA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"} {"@timestamp":"2024-09-13T15:32:31.150Z", "log.level": "INFO", "message":"[winlogbeat-000001/x06WLiZrTrGP0we6M50Ohg] update_mapping [_doc]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][masterService#updateTask][T#12]","log.logger":"org.elasticsearch.cluster.metadata.MetadataMappingService","elasticsearch.cluster.uuid":"K0JHLnRwq8fzCcvC6fwg","elasticsearch.node.id":"e8KC0YiIQMyxw9TyTW5dRA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"} {"@timestamp":"2024-09-13T15:32:31.232Z", "log.level": "INFO", "message":"[winlogbeat-000001/x06WLiZrTrGP0we6M50Ohg] update_mapping [_doc]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][masterService#updateTask][T#12]","log.logger":"org.elasticsearch.cluster.metadata.MetadataMappingService","elasticsearch.cluster.uuid":"K0__JHLnRwq8fzCcvC6fwg","elasticsearch.node.id":"e8KC0YiIQMyxw9TyTW5dRA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"} {"@timestamp":"2024-09-13T15:54:38.531Z", "log.level": "INFO", "message":"[gc][4535] overhead, spent [250ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"K0__JHLnRwq8fzCcvC6fwg","elasticsearch.node.id":"e8KC0YiIQMyxw9TyTW5dRA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"}

-----------lme_logstash.1.n0afy8a1zw50sgjzyicw00aqy---------- 2024/09/13 14:37:42 Setting 'queue.type' from environment. 2024/09/13 14:37:42 Setting 'xpack.monitoring.enabled' from environment. 2024/09/13 14:37:42 Setting 'pipeline.ecs_compatibility' from environment. /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:536: warning: already initialized constant Manticore::Client::StringEntity /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:536: warning: already initialized constant Manticore::Client::StringEntity [2024-09-13T14:39:14,491][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://logstash_writer:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'https://elasticsearch:9200/'"} [2024-09-13T14:39:14,490][WARN ][logstash.outputs.elasticsearch][main] Health check failed {:code=>401, :url=>https://elasticsearch:9200/, :message=>"Got response code '401' contacting Elasticsearch at URL 'https://elasticsearch:9200/'"} [2024-09-13T14:39:14,493][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://logstash_writer:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'https://elasticsearch:9200/'"} [2024-09-13T14:39:14,498][WARN ][logstash.outputs.elasticsearch][main] Health check failed {:code=>401, :url=>https://elasticsearch:9200/, :message=>"Got response code '401' contacting Elasticsearch at URL 'https://elasticsearch:9200/'"} [2024-09-13T14:39:14,499][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://logstash_writer:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'https://elasticsearch:9200/'"} [2024-09-13T14:39:19,747][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://logstash_writer:xxxxxx@elasticsearch:9200/"} [2024-09-13T14:39:19,770][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.11.1) {:es_version=>8} [2024-09-13T14:39:19,771][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>8} [2024-09-13T14:39:19,805][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://logstash_writer:xxxxxx@elasticsearch:9200/"} [2024-09-13T14:39:19,806][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.11.1) {:es_version=>8} [2024-09-13T14:39:19,806][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>8} [2024-09-13T14:39:19,856][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://logstash_writer:xxxxxx@elasticsearch:9200/"} [2024-09-13T14:39:19,860][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.11.1) {:es_version=>8} [2024-09-13T14:39:19,860][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>8} [2024-09-13T14:39:41,145][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8} [2024-09-13T14:39:41,145][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8} [2024-09-13T14:39:41,216][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8} [2024-09-13T14:39:41,267][INFO ][logstash.outputs.elasticsearch][main] Installing Elasticsearch template {:name=>"ecs-logstash"} [2024-09-13T14:39:41,268][INFO ][logstash.outputs.elasticsearch][main] Installing Elasticsearch template {:name=>"ecs-logstash"} [2024-09-13T14:39:41,307][INFO ][logstash.outputs.elasticsearch][main] Installing Elasticsearch template {:name=>"ecs-logstash"}


Increase the number of lines if your issue is not present, or include a relevant log of the erroring container
- Output of the relevant /var/log/cron_logs/ file

## Expected behavior
For the Alerts page to load with the Manage rules button visible.

## Screenshots
If applicable, add screenshots to help explain your problem.

![Alerts page](https://github.com/user-attachments/assets/3316a4e3-b6d0-4e18-8e75-1fde4761d0ca)
![Failed to create the lists index](https://github.com/user-attachments/assets/842d184d-bb42-4fa3-a210-887d7dc27e1f)

## Additional context
Logs are being generated and shown on the various Dashboards under Analytics-> Dashboards, such as User Security.  However when trying to bring up pages under Security such as the Alerts page results in this error.  The built-in superuser "elastic" should have the right permissions to view this page, I have tried making a new users and giving the explicit permissions needed but still get the same error.
aarz-snl commented 2 months ago

Please run the following command with sudo privs

curl --cacert /opt/lme/Chapter\ 3\ Files/certs/root-ca.crt --user elastic:NbKfxYICdnLTCsBuTScHwNmtT6rEaz5c -X PUT "https://127.0.0.1:9200/_index_template/number_of_replicas" -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["*"],
  "template": {
    "settings": {
      "number_of_replicas": 0
    }
  },
  "data_stream": {},
  "priority": 1
}'
khamilton479 commented 2 months ago

Thank you for the reply. Just to clarify, do you run this command in the Console Dev Tools in Elastic, or on the Ubuntu machine? Do I need to replace the elastic user password with the one it generated for me, or run the command exactly as shown?

aarz-snl commented 2 months ago

Sorry, run this in the ubuntu machine.

Also, where you see this: elastic:NbKfxYICdnLTCsBuTScHwNmtT6rEaz5c

Use YOUR elastic password. Thats just a generic one I was using temporarily for testing. Change to root before running it as well

khamilton479 commented 2 months ago

Thank you for the clarification and the solution! The Alerts and Rules pages are now working.