Closed wesliix closed 6 months ago
llwaterhouse
commented
7 months ago Thank you for reaching out. It seems the problem might stem from slight variations in following the installation guidelines. Could you please review the troubleshooting guide here?
If you still encounter issues deploying LME, submit a bug report here, ensuring you provide all details requested in the Bug template.
We're here to provide further assistance with your LME installation and usage should you need it!
wesliix
commented
6 months ago I've done all the troubleshooting, but nothing is working.
This is the error in the event viewer from winlogbeat i found it does this every time when i start the winlogbeat service, do you need to see any log file from me?
I've uninstalled and installed the script a few times aswell, but no result.
I am running the whole LME project on a ESX server with virtual machines, I am using: 1 Windows Server with a Domain controller, 1 Windows Event Collector, 4 Windows Clients, and 1 Ubuntu 22.04 Jammy LTS server, everything is linked except the Ubuntu server. Each device can ping each other.
llwaterhouse
commented
6 months ago Please submit the output of these commands: free -h df -h uname -a lsb_release -a
for name in $(sudo docker ps -a --format '{{.Names}}'); do echo -e "\n\n\n-----------$name----------"; sudo docker logs $name | tail -n 20; done
wesliix
commented
6 months ago free -h df -h uname -a lsb_release -a output:
can you tell me how i can send you the for name in logs? do i have to put it in a file and upload it here? thank you for all your help by the way.
aarz-snl
commented
6 months ago This is most likely an improperly configured winlogbeat.yml file.
Please review 3.3.2 carefully and ensure you performed the unzip / moving of files correctly
Especially this step:
Then, move the 'winlogbeat.yml' file located at C:\Program Files\lme\winlogbeat.yml into the winlogbeat folder C:\Program Files\lme\winlogbeat-8.[x].[y]-windows-x86_64, overwriting the existing file when prompted to do so.
Should look something like this:
https://github.com/cisagov/LME/blob/main/Chapter%203%20Files/winlogbeat.yml
aarz-snl
commented
6 months ago you can also run:
.\winlogbeat.exe test config -c .\winlogbeat.yml to test the configuration file
if the winlogbeat.exe exists in there. I dont have my lab in front of my right now -- but could be an option
wesliix
commented
6 months ago Thank you @aarz-snl ,
I already moved the files correctly and overwritten the existing file.
https://github.com/cisagov/LME/blob/main/Chapter%203%20Files/winlogbeat.yml I took this config, and put it into my winlogbeat.yml.
When i run .\winlogbeat.exe test config -c .\winlogbeat.yml
I get the reply:
Config OK.
tried to run the service again, still instantly stops.
aarz-snl
commented
6 months ago Any logs in your windows logs under 'application' ?
You can add this to bottom of the winlogbeat.yml:
logging.level: debug logging.to_files: false logging.to_stderr: true
then run
.\winlogbeat.exe -e -c .\winlogbeat.yml
llwaterhouse
commented
6 months ago @wesliix, if it's easier, you can send a screenshot of the "for name" command. If it's too long, feel free to upload a file.
wesliix
commented
6 months ago @aarz-snl nope, no logs in Windows Logs > Application when i run the service
Here is the log file when running the cmd .\winlogbeat.exe -e -c .\winlogbeat.yml :
I have changed the path to /lme/winlogbeat/Winlogbeat to try if that works, but no result.
wesliix
commented
6 months ago @llwaterhouse here is the "for name" log, might be a little long:
`Mar 29, 2024 9:28:32 AM sun.util.locale.provider.LocaleProviderAdapter
-----------lme_elasticsearch.1.w413kv8j5dnjyd6nci0se5c3b----------
Mar 29, 2024 9:28:32 AM sun.util.locale.provider.LocaleProviderAdapter allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#7]","log.logger":"org.elasticsearch.xpack.transform.transforms.TransformFailureHandler","elasticsearch.cluster.uuid":"Pux7zTuwQ8i1NkXF7AAGEA","elasticsearch.node.id":"KGN1tdLjRAaLX_m-AgcRXg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"}
{"@timestamp":"2024-03-29T09:30:04.138Z", "log.level": "INFO", "message":"[slo-summary-timeslices-90d-rolling] successfully completed and scheduled task in node operation", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.transform.transforms.TransformPersistentTasksExecutor","elasticsearch.cluster.uuid":"Pux7zTuwQ8i1NkXF7AAGEA","elasticsearch.node.id":"KGN1tdLjRAaLX_m-AgcRXg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"}
{"@timestamp":"2024-03-29T09:30:04.149Z", "log.level": "INFO", "message":"[slo-summary-timeslices-90d-rolling] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#8]","log.logger":"org.elasticsearch.xpack.transform.transforms.TransformFailureHandler","elasticsearch.cluster.uuid":"Pux7zTuwQ8i1NkXF7AAGEA","elasticsearch.node.id":"KGN1tdLjRAaLX_m-AgcRXg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"}
{"@timestamp":"2024-03-29T09:30:04.380Z", "log.level": "INFO", "message":"[slo-summary-occurrences-7d-rolling] successfully completed and scheduled task in node operation", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.transform.transforms.TransformPersistentTasksExecutor","elasticsearch.cluster.uuid":"Pux7zTuwQ8i1NkXF7AAGEA","elasticsearch.node.id":"KGN1tdLjRAaLX_m-AgcRXg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"}
{"@timestamp":"2024-03-29T09:30:04.391Z", "log.level": "INFO", "message":"[slo-summary-occurrences-7d-rolling] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#6]","log.logger":"org.elasticsearch.xpack.transform.transforms.TransformFailureHandler","elasticsearch.cluster.uuid":"Pux7zTuwQ8i1NkXF7AAGEA","elasticsearch.node.id":"KGN1tdLjRAaLX_m-AgcRXg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"}
{"@timestamp":"2024-03-29T09:30:04.626Z", "log.level": "INFO", "message":"[slo-summary-timeslices-monthly-aligned] successfully completed and scheduled task in node operation", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.transform.transforms.TransformPersistentTasksExecutor","elasticsearch.cluster.uuid":"Pux7zTuwQ8i1NkXF7AAGEA","elasticsearch.node.id":"KGN1tdLjRAaLX_m-AgcRXg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"}
{"@timestamp":"2024-03-29T09:30:04.629Z", "log.level": "INFO", "message":"[slo-summary-timeslices-monthly-aligned] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#4]","log.logger":"org.elasticsearch.xpack.transform.transforms.TransformFailureHandler","elasticsearch.cluster.uuid":"Pux7zTuwQ8i1NkXF7AAGEA","elasticsearch.node.id":"KGN1tdLjRAaLX_m-AgcRXg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"}
{"@timestamp":"2024-03-29T09:30:04.893Z", "log.level": "INFO", "message":"[slo-summary-timeslices-weekly-aligned] successfully completed and scheduled task in node operation", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.transform.transforms.TransformPersistentTasksExecutor","elasticsearch.cluster.uuid":"Pux7zTuwQ8i1NkXF7AAGEA","elasticsearch.node.id":"KGN1tdLjRAaLX_m-AgcRXg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"}
{"@timestamp":"2024-03-29T09:30:04.895Z", "log.level": "INFO", "message":"[slo-summary-timeslices-weekly-aligned] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#3]","log.logger":"org.elasticsearch.xpack.transform.transforms.TransformFailureHandler","elasticsearch.cluster.uuid":"Pux7zTuwQ8i1NkXF7AAGEA","elasticsearch.node.id":"KGN1tdLjRAaLX_m-AgcRXg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"}
{"@timestamp":"2024-03-29T09:30:05.098Z", "log.level": "INFO", "message":"reloading search analyzers", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#4]","log.logger":"org.elasticsearch.index.mapper.MapperService","elasticsearch.cluster.uuid":"Pux7zTuwQ8i1NkXF7AAGEA","elasticsearch.node.id":"KGN1tdLjRAaLX_m-AgcRXg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es","tags":[" [.apm-source-map]"]}
{"@timestamp":"2024-03-29T09:30:05.169Z", "log.level": "INFO", "message":"reloading search analyzers", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#5]","log.logger":"org.elasticsearch.index.mapper.MapperService","elasticsearch.cluster.uuid":"Pux7zTuwQ8i1NkXF7AAGEA","elasticsearch.node.id":"KGN1tdLjRAaLX_m-AgcRXg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es","tags":[" [.ds-.kibana-event-log-ds-2024.03.26-000001]"]}
{"@timestamp":"2024-03-29T09:30:05.212Z", "log.level": "INFO", "message":"reloading search analyzers", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#9]","log.logger":"org.elasticsearch.index.mapper.MapperService","elasticsearch.cluster.uuid":"Pux7zTuwQ8i1NkXF7AAGEA","elasticsearch.node.id":"KGN1tdLjRAaLX_m-AgcRXg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es","tags":[" [.slo-observability.sli-v2]"]}
{"@timestamp":"2024-03-29T09:30:05.296Z", "log.level": "INFO", "message":"reloading search analyzers", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#7]","log.logger":"org.elasticsearch.index.mapper.MapperService","elasticsearch.cluster.uuid":"Pux7zTuwQ8i1NkXF7AAGEA","elasticsearch.node.id":"KGN1tdLjRAaLX_m-AgcRXg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es","tags":[" [.kibana-observability-ai-assistant-conversations-000001]"]}
{"@timestamp":"2024-03-29T09:30:05.620Z", "log.level": "INFO", "message":"reloading search analyzers", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#9]","log.logger":"org.elasticsearch.index.mapper.MapperService","elasticsearch.cluster.uuid":"Pux7zTuwQ8i1NkXF7AAGEA","elasticsearch.node.id":"KGN1tdLjRAaLX_m-AgcRXg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es","tags":[" [winlogbeat-000001]"]}
{"@timestamp":"2024-03-29T09:30:05.641Z", "log.level": "INFO", "message":"reloading search analyzers", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#7]","log.logger":"org.elasticsearch.index.mapper.MapperService","elasticsearch.cluster.uuid":"Pux7zTuwQ8i1NkXF7AAGEA","elasticsearch.node.id":"KGN1tdLjRAaLX_m-AgcRXg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es","tags":[" [.ds-ilm-history-5-2024.03.26-000001]"]}
{"@timestamp":"2024-03-29T09:30:05.733Z", "log.level": "INFO", "message":"reloading search analyzers", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#10]","log.logger":"org.elasticsearch.index.mapper.MapperService","elasticsearch.cluster.uuid":"Pux7zTuwQ8i1NkXF7AAGEA","elasticsearch.node.id":"KGN1tdLjRAaLX_m-AgcRXg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es","tags":[" [.ds-.logs-deprecation.elasticsearch-default-2024.03.26-000001]"]}
{"@timestamp":"2024-03-29T09:30:05.988Z", "log.level": "INFO", "current.health":"GREEN","message":"Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.ds-.logs-deprecation.elasticsearch-default-2024.03.26-000001][0]]]).","previous.health":"RED","reason":"shards started [[.ds-.logs-deprecation.elasticsearch-default-2024.03.26-000001][0]]" , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"Pux7zTuwQ8i1NkXF7AAGEA","elasticsearch.node.id":"KGN1tdLjRAaLX_m-AgcRXg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"}
{"@timestamp":"2024-03-29T09:30:08.711Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-City.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#2]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"Pux7zTuwQ8i1NkXF7AAGEA","elasticsearch.node.id":"KGN1tdLjRAaLX_m-AgcRXg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"}
{"@timestamp":"2024-03-29T09:04:10.857Z", "log.level": "WARN", "message":"absolute clock went backwards by [1h/3607732ms] while timer thread was sleeping", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][[timer]]","log.logger":"org.elasticsearch.threadpool.ThreadPool","elasticsearch.cluster.uuid":"Pux7zTuwQ8i1NkXF7AAGEA","elasticsearch.node.id":"KGN1tdLjRAaLX_m-AgcRXg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"loggingmadeeasy-es"}
-----------lme_kibana.1.tdmpfls71e19ha1jqiyqq4gb7---------- [2024-03-29T09:30:17.580+00:00][INFO ][plugins.alerting] Creating concrete write index - .internal.alerts-observability.slo.alerts-default-000001 [2024-03-29T09:30:17.587+00:00][INFO ][plugins.alerting] Creating concrete write index - .internal.alerts-observability.logs.alerts-default-000001 [2024-03-29T09:30:17.600+00:00][INFO ][plugins.alerting] Creating concrete write index - .internal.alerts-observability.uptime.alerts-default-000001 [2024-03-29T09:30:17.875+00:00][INFO ][plugins.alerting] Creating concrete write index - .internal.alerts-ml.anomaly-detection.alerts-default-000001 [2024-03-29T09:30:17.885+00:00][INFO ][plugins.alerting] Creating concrete write index - .internal.alerts-observability.threshold.alerts-default-000001 [2024-03-29T09:30:20.046+00:00][INFO ][plugins.observability] Installing SLO ingest pipeline [.slo-observability.sli.pipeline] [2024-03-29T09:30:20.345+00:00][INFO ][plugins.observability] Installing SLO ingest pipeline [.slo-observability.summary.pipeline] [2024-03-29T09:30:20.459+00:00][INFO ][status] Kibana is now available (was degraded) [2024-03-29T09:30:20.595+00:00][INFO ][plugins.observability] SLO summary transforms already installed - skipping [2024-03-29T09:30:20.791+00:00][INFO ][plugins.observabilityAIAssistant.service] Creating concrete write index - .kibana-observability-ai-assistant-kb-000001 [2024-03-29T09:30:21.735+00:00][INFO ][plugins.observabilityAIAssistant.service] Successfully set up index assets [2024-03-29T09:31:36.914+00:00][ERROR][plugins.fleet] Failed to fetch latest version of synthetics from registry: Error connecting to package registry: request to https://epr.elastic.co/search?package=synthetics&prerelease=true&kibana.version=8.11.1 failed, reason: getaddrinfo EAI_AGAIN epr.elastic.co [2024-03-29T09:31:37.149+00:00][INFO ][plugins.fleet] Fleet setup completed [2024-03-29T09:31:37.155+00:00][INFO ][plugins.securitySolution] Dependent plugin setup complete - Starting ManifestTask [2024-03-29T09:31:37.159+00:00][INFO ][plugins.securitySolution.endpoint.policyProtections] App feature [endpoint_policy_protections] is enabled. Nothing to do! [2024-03-29T09:32:10.283+00:00][ERROR][plugins.fleet] Failed to fetch latest version of synthetics from registry: Error connecting to package registry: request to https://epr.elastic.co/search?package=synthetics&prerelease=true&kibana.version=8.11.1 failed, reason: getaddrinfo EAI_AGAIN epr.elastic.co [2024-03-29T09:32:10.307+00:00][INFO ][plugins.synthetics] Installed synthetics index templates [2024-03-29T09:35:25.008+00:00][ERROR][plugins.telemetry.fetcher] Cannot reach the remote telemetry endpoint https://telemetry.elastic.co/v3/send/kibana-snapshot [2024-03-29T09:45:16.681+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"num_host_urls":0}} [2024-03-29T10:00:16.804+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"num_host_urls":0}}
-----------lme_logstash.1.prm1n5nai119o8tmr48efstf3----------
2024/03/29 09:27:34 Setting 'queue.type' from environment.
2024/03/29 09:27:34 Setting 'xpack.monitoring.enabled' from environment.
2024/03/29 09:27:34 Setting 'pipeline.ecs_compatibility' from environment.
[2024-03-29T09:30:06,144][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://logstash_writer:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
[2024-03-29T09:30:06,148][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://logstash_writer:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
[2024-03-29T09:30:11,192][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"elasticsearch", :exception=>Manticore::ResolutionFailure, :cause=>#type event field won't be used to determine the document _type {:es_version=>8}
[2024-03-29T09:30:18,841][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.11.1) {:es_version=>8}
[2024-03-29T09:30:18,841][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>8}
[2024-03-29T09:30:19,478][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://logstash_writer:xxxxxx@elasticsearch:9200/"}
[2024-03-29T09:30:19,479][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.11.1) {:es_version=>8}
[2024-03-29T09:30:19,480][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>8}
[2024-03-29T09:30:41,012][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2024-03-29T09:30:41,192][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2024-03-29T09:30:41,217][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
-----------lme_kibana.1.w0x4ly3hgn2hb811n0g5fqxnr---------- [2024-03-28T14:03:55.013+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"num_host_urls":0}} [2024-03-28T14:18:55.156+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"num_host_urls":0}} [2024-03-28T14:33:55.211+00:00][INFO ][plugins.fleet] Running Fleet Usage telemetry send task [2024-03-28T14:33:55.371+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"num_host_urls":0}} [2024-03-28T14:48:55.293+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"num_host_urls":0}} [2024-03-28T15:03:58.391+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"num_host_urls":0}} [2024-03-28T15:19:01.550+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"num_host_urls":0}} [2024-03-28T15:33:57.161+00:00][INFO ][plugins.fleet] Running Fleet Usage telemetry send task [2024-03-28T15:33:59.647+00:00][INFO ][plugins.securitySolution.endpoint:metadata-check-transforms-task:0.0.1] no endpoint installation found [2024-03-28T15:34:02.551+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"num_host_urls":0}} [2024-03-28T15:49:02.715+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"num_host_urls":0}} [2024-03-28T15:56:56.933+00:00][INFO ][plugins.security.routes] Logging in with provider "basic" (basic) [2024-03-28T15:57:42.392+00:00][INFO ][plugins.security.routes] Logging in with provider "basic" (basic) [2024-03-28T15:58:57.811+00:00][INFO ][plugins.securitySolution] Fetch risk engine metrics [2024-03-28T16:04:02.986+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"num_host_urls":0}} [2024-03-28T16:10:01.484+00:00][INFO ][root] SIGTERM received - initiating shutdown [2024-03-28T16:10:01.485+00:00][INFO ][root] Kibana is shutting down [2024-03-28T16:10:02.753+00:00][INFO ][root] SIGTERM received - initiating shutdown [2024-03-28T16:10:06.584+00:00][INFO ][plugins-system.standard] Stopping all plugins. [2024-03-28T16:10:06.603+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Monitoring stats collection is stopped `
cbaxley
commented
6 months ago For starters, it looks like the ubuntu machine may have problems getting to the internet.
I would try connecting to some hosts on the internet first.
curl https://www.google.com
Once you can get some connectivity, I would uninstall lme on the linux machine and do another install.
cd /opt/lme/Chapter\ 3\ Files
./deploy.sh uninstall
# Clean up the volumes as the last output indicates
./deploy.sh installAfter that is finished I would check the connectivity from windows in a browser by going to
https://ip.of.linux.machine
And seeing if I get a response. If you do not receive a response, check the firewalls on both machines to make sure traffic can flow.
wesliix
commented
6 months ago @cbaxley Thank you for your reply,
I've done thecurl https://www.google.com, and i get some long output.
After that:
I have installed a new Linux machine and installed the script again, I can reach https://ip.of.linux.machine , it is opening the Elastic login page on my Windows Event Collector,
When I run sudo docker stack ps lme I get this output:
When i try to reach Kibana, with https://ip.of.linux.machine:5044 I get an error: This site can't provide a secure connection, ERR_BAD_SSL_CLIENT_AUTH_CERT.
And when i start my winlogbeat, it is again stopping the service instantly.
wesliix
Edit:after rebooting i get this output now with sudo docker stack ps lme
I tried:
cd /opt/lme/Chapter\ 3\ Files/, then run sudo ./deploy.sh uninstall
Run rm -r /opt/lme
Reclone the LME repository into /opt/lme/: git clone git@github.com:cisagov/LME.git /opt/lme/
Navigate back to Chapter 3 Files: cd /opt/lme/Chapter\ 3\ Files/, then sudo ./deploy.sh install
Save credentials, then continue with Chapter 3 installationbut its still the same.
llwaterhouse
commented
6 months ago Hello, I can't find where you have listed all the versions of the software you've installed. Could you please repaste it here as it was laid out in the Bug Template, especially the version of winlogbeat that you're running?
wesliix
commented
6 months ago @llwaterhouse thanks, yep i will do as much as i can:
Windows Server (DC and Event Collector) version: Microsoft Windows Server 21H2 (OS Build 20348.1787) Winlogbeat.exe version: 8.5.0 (amd64), libbeat 8.5.0 Docker version: 26.0.0 build 2ae903e Linux Ubuntu version: Ubuntu 22.04.4 LTS Jammy Jellyfish
All machines above have the required infrastructure.
If you need any more informaton, let me know please.
aarz-snl
commented
6 months ago note that 5044 is logstash not kibana. Kibana is set to port 443 right now so when you go to https://ipoflinuxmachine that is kibana. You wont be able to browse to logstash
When running docker stack ps lme its normal to see some of these in a shutdown state. that just means a task was completed on bootup of the stack.
The real key is when you run docker ps are the containers up and healthy and based on the time provided have been up and healthy since bootup or restart of the stack.
I believe you have internet connection if you successfully cloned the repo... but you can do
curl -vs https://www.google.com -o /dev/null 2>&1 | grep -i "handshake"
if you see tls handshake finished in the output the tls worked.
I'm having trouble understand why winlogbeat is failing for you -- and what I really need is some type of information that points to why. a log, or something. Without that I really cant resolve it. It could be a 100 different things on your server / network keeping it from starting. The install is really quite basic and cut and try so I wouldn't say winlogbeat is causing the problem. I will have to jump into some docs and see if i can find away to debug this
Can you please show the output of docker ps (after the server has been up for a bit)
aarz-snl
commented
6 months ago could you also copy the contents of your winlogbeat.yml -- redact any sensitive information that may appear
wesliix
commented
6 months ago @aarz-snl thank you for your reply once again.
Yeah, when i go to it is going to Elastic login page.
When i'm using curl -vs https://www.google.com/ -o /dev/null 2>&1 | grep -i "handshake"
I get this output, so the TLS is working:
I've had the Ubuntu server running for about 10 minutes and this is the docker ps output:
I found something aswell in the C:\ProgramData\winlogbeat\logs and I think that might probably be the problem:
{"log.level":"info","@timestamp":"2024-04-03T11:45:15.528Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Starting metrics logging every 30s","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-04-03T11:45:15.533Z","log.logger":"winlogbeat","log.origin":{"file.name":"beater/winlogbeat.go","file.line":149},"message":"Winlogbeat is unable to load the ingest pipelines because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines, you can ignore this warning.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-04-03T11:45:15.591Z","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":476},"message":"failed to load publisher metadata for Microsoft-Windows-EventForwarder (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-03T11:45:17.676Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":139},"message":"Connecting to backoff(async(tcp://logstash_dns_name:5044))","service.name":"winlogbeat","ecs.version":"1.6.0"}Here is my winlogbeat.yml file:
winlogbeat.event_logs:
- name: Application
ignore_older: 72h
- name: System
- name: Security
- name: Microsoft-Windows-Sysmon/Operational
- name: Windows PowerShell
event_id: 400, 403, 600, 800
- name: Microsoft-Windows-PowerShell/Operational
event_id: 4103, 4104, 4105, 4106
- name: ForwardedEvents
tags: [forwarded]
output.logstash:
# The Logstash hosts
hosts: ["logstash_dns_name:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
ssl.certificate_authorities: ["C:\\Program Files\\lme\\root-ca.crt"]
# Certificate for SSL client authentication
ssl.certificate: "C:\\Program Files\\lme\\wlbclient.crt"
# Client Certificate Key
ssl.key: "C:\\Program Files\\lme\\wlbclient.key"
Thinking of reinstalling everything if there is no solution, but there should be one i hope.
aarz-snl
commented
6 months ago Do you have any errors in the forwarded events section of your events?
Shot in the dark looking at this:
https://github.com/elastic/beats/issues/34705
If there appears to be some tie in here -- we may want to look into upgrading your winlogbeat to at least 8.7.2
Or maybe 8.8.0
I will try in my lab but if things start lining up where these are similiar you can always give it a shot yourself
uninstall using .\winlogbeat.exe uninstall
And then perform the SAME steps from our instructions with an updated version of winlogbeat:
https://www.elastic.co/downloads/past-releases#winlogbeat
Be careful where everything goes and ensure you have crts and everything all in the right spot after making the change
aarz-snl
commented
6 months ago i may have found errors in your log that look somewhat like the ones from the github issue:
Exception 0xc0000005 0x1 0x0 0x7ffc6af2e254 PC=0x7ffc6af2e254
runtime.cgocall(0xfb1e00, 0xc0000776c0) /usr/local/go/src/runtime/cgocall.go:157 +0x4a fp=0xc0005e4478 sp=0xc0005e4440 pc=0xf44aea syscall.SyscallN(0x7ffc65e5cbb0, {0xc0005e4520?, 0x9, 0x0?}) /usr/local/go/src/runtime/syscall_windows.go:556 +0x12b fp=0xc0005e4500 sp=0xc0005e4478 pc=0xface2b syscall.Syscall9(0xc0018d1980?, 0xc0004c5380?, 0x4c59866?, 0x8?, 0xc0018ca4b0?, 0x29?, 0xc0005e4650?, 0x32f75eb?, 0x4c98d65?, 0x0, ...) /usr/local/go/src/runtime/syscall_windows.go:506 +0x78 fp=0xc0005e4578 sp=0xc0005e4500 pc=0xfacb18 github.com/elastic/beats/v7/winlogbeat/sys/wineventlog._EvtFormatMessage(0x100002c, 0x12, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0xc0005e4664) /go/src/github.com/elastic/beats/winlogbeat/sys/wineventlog/zsyscall_windows.go:132 +0x105 fp=0xc0005e4628 sp=0xc0005e4578 pc=0x1e206a5 github.com/elastic/beats/v7/winlogbeat/sys/wineventlog.evtFormatMessage(0x100002c, 0x12, 0x0, {0x0, 0x0, 0x0}, 0x1) /go/src/github.com/elastic/beats/winlogbeat/sys/wineventlog/format_message.go:82 +0x105 fp=0xc0005e4850 sp=0xc0005e4628 pc=0x1e038a5 github.com/elastic/beats/v7/winlogbeat/sys/wineventlog.getMessageString(0xc0005e4a50, 0x12, 0x0, {0x0, 0x0, 0x0}) /go/src/github.com/elastic/beats/winlogbeat/sys/wineventlog/format_message.go:58 +0x105 fp=0xc0005e4908 sp=0xc0005e4850 pc=0x1e03545 github.com/elastic/beats/v7/winlogbeat/sys/wineventlog.getMessageStringFromHandle(0xc0005e4a50, 0x12, {0x0, 0x0, 0x0}) /go/src/github.com/elastic/beats/winlogbeat/sys/wineventlog/format_message.go:34 +0x95 fp=0xc0005e49a8 sp=0xc0005e4908 pc=0x1e03215 github.com/elastic/beats/v7/winlogbeat/sys/wineventlog.Message(0x12, {0xc0005b4000, 0x4000, 0x4000}, 0xc0005e4b60) /go/src/github.com/elastic/beats/winlogbeat/sys/wineventlog/wineventlog_windows.go:274 +0x26c fp=0xc0005e4ae8 sp=0xc0005e49a8 pc=0x1e1dd2c github.com/elastic/beats/v7/winlogbeat/eventlog.newWinEventLog.func5(0x12) /go/src/github.com/elastic/beats/winlogbeat/eventlog/wineventlog.go:568 +0xcc fp=0xc0005e4bb8 sp=0xc0005e4ae8 pc=0x33007ec github.com/elastic/beats/v7/winlogbeat/eventlog.(winEventLog).Read(0xc00054f1e0) /go/src/github.com/elastic/beats/winlogbeat/eventlog/wineventlog.go:333 +0xcd8 fp=0xc0005e5520 sp=0xc0005e4bb8 pc=0x32fcf58 github.com/elastic/beats/v7/winlogbeat/beater.(eventLogger).run(0xc00001c840, 0xc000103020, {0x4f7c548, 0xc000004540}, {{0x0, 0x0}, 0x0, {0x0, 0x0, 0x0}, ...}, ...) /go/src/github.com/elastic/beats/winlogbeat/beater/eventlogger.go:154 +0x94f fp=0xc0005e5e50 sp=0xc0005e5520 pc=0x330756f github.com/elastic/beats/v7/winlogbeat/beater.(Winlogbeat).processEventLog(0xc000018090, 0xc000293c20, 0xc00001c840, {{0x0, 0x0}, 0x0, {0x0, 0x0, 0x0}, {0x0, ...}}, ...) /go/src/github.com/elastic/beats/winlogbeat/beater/winlogbeat.go:196 +0x132 fp=0xc0005e5f48 sp=0xc0005e5e50 pc=0x330a652 github.com/elastic/beats/v7/winlogbeat/beater.(Winlogbeat).Run.func2() /go/src/github.com/elastic/beats/winlogbeat/beater/winlogbeat.go:164 +0xa5 fp=0xc0005e5fe0 sp=0xc0005e5f48 pc=0x330a2a5 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0005e5fe8 sp=0xc0005e5fe0 pc=0xfb0501 created by github.com/elastic/beats/v7/winlogbeat/beater.(*Winlogbeat).Run /go/src/github.com/elastic/beats/winlogbeat/beater/winlogbeat.go:164 +0x6a5
wesliix
commented
6 months ago @aarz-snl Thank you again for the reply, going to test some stuff now you mentioned, will keep you updated.
Update:
Finally fixed it, the winlogbeat service is running now and not stopping anymore.
Thanks to @aarz-snl, Uninstalling winlogbeat 8.5.0 and installing winlogbeat 8.8.0 was the solution for me. Very simple, but I simply didn't think of this.
Stupid from me, but for some reason is winlogbeat version 8.5.0 not working for me.
I uninstalled my winlogbeat 8.5.0, then I installed winlogbeat version 8.8.0 with the same instructions from Chapter 3.3, and it is working perfectly, so now i can finally continue.
I would also like to thank the rest for their input.
wesliix
llwaterhouse
commented
6 months ago @wesliix thank you for letting us know what solved your issue. We appreciate your feedback.
Hi its me again, i've almost finished Chapter 3 but i'm having a little problem.
I completed all steps, ran the script ./install-service-winlogbeat.ps1, everything completed.
Then, in powershell i start the winlogbeatservice with Start-Service winlogbeat.
When I open services.msc the winlogbeat service is running for 1 second, and then it instantly stops. I can keep trying whatever i want but it keeps stopping after running it.
Anyone can help me out? Thanks in advance.
wesliix