search

Security-Onion-Solutions / securityonion

Security Onion is a free and open platform for threat hunting, enterprise security monitoring, and log management. It includes our own interfaces for alerting, dashboards, hunting, PCAP, detections, and case management. It also includes other tools such as osquery, CyberChef, Elasticsearch, Logstash, Kibana, Suricata, and Zeek.
https://securityonion.net
3.28k stars 507 forks source link

FIX: outdated import-evtx-logs pipeline versions #11889

Closed chateaulav closed 11 months ago

chateaulav commented 11 months ago

information about your system and how you installed Security Onion

Oracle Linux Server release 9.3 Linux version 5.15.0-101.103.2.1.el9uek.x86_64 (mockbuild@host-100-100-224-7) (gcc (GCC) 11.3.1 20220421 (Red Hat 11.3.1-2.1.0.2), GNU ld version 2.35.2-24.0.1.el9) #2 SMP Tue May 2 01:10:45 PDT 2023

Security Onion Version: 2.4.30

relevant log files

the import is trying to use logs-system.security-1.34.0 based on the elastic-agent policy when Elastic and Logstash are configured with logs-system.security-1.43.0

This applies to the following pipelines, based on the preset agent policy that was not updated with the 2.4 release.

logs-system.system-1.34.0 -> logs-system.system-1.43.0 logs-system.security-1.34.0 -> logs-system.security-1.43.0 logs-system.application-1.34.0 -> logs-system.application-1.43.0 logs-windows.sysmon_operational-1.24.0 -> logs-windows.sysmon_operational-1.38.0 logs-windows.powershell_operational-1.24.0 -> logs-windows.powershell_operational-1.38.0

Affected file:
salt/elasticfleet/files/integrations/grid-nodes_general/import-evtx-logs.json

Logstash ingest pipeline versions:
image

Logstash error when attempting to import evtx data: 

=========================================================================
 Checking log file /opt/so/log/logstash/logstash.log
=========================================================================
[2023-11-29T12:44:31,487][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["create", {:_id=>nil, :_index=>"logs-import-so", :routing=>nil, :pipeline=>"logs-windows.sysmon_operational-1.24.0"}, {"log"=>{"offset"=>242913, "file"=>{"path"=>"/nsm/import/fff4261b8064add1ddbb546f4c59dece/evtx/data.json", "name"=>"/tmp/data.evtx"}}, "event"=>{"dataset"=>"windows.sysmon_operational", "created"=>"2023-11-29T07:48:43.020873Z", "imported"=>true, "module"=>"windows", "code"=>1}, "message"=>"{\"event_record_id\":13741,\"timestamp\":\"2023-11-29T07:48:43.020873Z\",\"winlog\":{\"channel\":\"Microsoft-Windows-Sysmon/Operational\",\"computer_name\":\"Windows10-WS1.acmeonions.com\",\"event_id\":1,\"opcode\":0,\"provider_guid\":\"5770385F-C22A-43E0-BF4C-06F5698FFBD9\",\"provider_name\":\"Microsoft-Windows-Sysmon\",\"record_id\":13741,\"task\":1,\"version\":5,\"process\":{\"pid\":2880,\"thread_id\":4400},\"event_data\":{\"CommandLine\":\"\\\"C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\platform\\\\4.18.2011.6-0\\\\MpCmdRun.exe\\\" -IdleTask -TaskName WdCacheMaintenance\",\"Company\":\"Microsoft Corporation\",\"CurrentDirectory\":\"C:\\\\Windows\\\\system32\\\\\",\"Description\":\"Microsoft Malware Protection Command Line Utility\",\"FileVersion\":\"4.18.2011.6 (WinBuild.160101.0800)\",\"Hashes\":\"MD5=D1DC475DC8A08618A40809F5F2CBC5E4,SHA256=FD00C4BA457AB1B207EDE405AB6CFBA2EE76D82C5936A10CA77D82CD5F0E7588,IMPHASH=F214FB46830FF51943E760685C3F8DA7\",\"Image\":\"C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\Platform\\\\4.18.2011.6-0\\\\MpCmdRun.exe\",\"IntegrityLevel\":\"System\",\"LogonGuid\":\"36C8AC58-C64E-5FD3-E703-000000000000\",\"LogonId\":\"0x3e7\",\"OriginalFileName\":\"MpCmdRun.exe\",\"ParentCommandLine\":\"C:\\\\Windows\\\\system32\\\\svchost.exe -k netsvcs -p -s Schedule\",\"ParentImage\":\"C:\\\\Windows\\\\System32\\\\svchost.exe\",\"ParentProcessGuid\":\"36C8AC58-C64F-5FD3-1C00-000000001700\",\"ParentProcessId\":1200,\"ProcessGuid\":\"36C8AC58-3039-5FD4-B604-000000001700\",\"ProcessId\":7816,\"Product\":\"Microsoft® Windows® Operating System\",\"RuleName\":\"-\",\"TerminalSessionId\":0,\"User\":\"NT AUTHORITY\\\\SYSTEM\",\"UtcTime\":\"2023-11-29 07:48:43.020\",\"Status\":null}},\"log\":{\"file\":{\"name\":\"/tmp/data.evtx\"}},\"event\":{\"code\":1,\"created\":\"2023-11-29T07:48:43.020873Z\"},\"@timestamp\":\"2023-11-29T07:48:43.020873Z\"}", "@timestamp"=>2023-11-29T12:44:29.814Z, "type"=>"redis-input", "ecs"=>{"version"=>"8.0.0"}, "winlog"=>{"event_id"=>1, "opcode"=>0, "event_data"=>{"ProcessGuid"=>"36C8AC58-3039-5FD4-B604-000000001700", "TerminalSessionId"=>0, "Image"=>"C:\\ProgramData\\Microsoft\\Windows Defender\\Platform\\4.18.2011.6-0\\MpCmdRun.exe", "LogonGuid"=>"36C8AC58-C64E-5FD3-E703-000000000000", "ParentProcessGuid"=>"36C8AC58-C64F-5FD3-1C00-000000001700", "CurrentDirectory"=>"C:\\Windows\\system32\\", "Hashes"=>"MD5=D1DC475DC8A08618A40809F5F2CBC5E4,SHA256=FD00C4BA457AB1B207EDE405AB6CFBA2EE76D82C5936A10CA77D82CD5F0E7588,IMPHASH=F214FB46830FF51943E760685C3F8DA7", "FileVersion"=>"4.18.2011.6 (WinBuild.160101.0800)", "ParentProcessId"=>1200, "UtcTime"=>"2023-11-29 07:48:43.020", "RuleName"=>"-", "ParentCommandLine"=>"C:\\Windows\\system32\\svchost.exe -k netsvcs -p -s Schedule", "Status"=>nil, "User"=>"NT AUTHORITY\\SYSTEM", "ProcessId"=>7816, "CommandLine"=>"\"C:\\ProgramData\\Microsoft\\Windows Defender\\platform\\4.18.2011.6-0\\MpCmdRun.exe\" -IdleTask -TaskName WdCacheMaintenance", "IntegrityLevel"=>"System", "ParentImage"=>"C:\\Windows\\System32\\svchost.exe", "Description"=>"Microsoft Malware Protection Command Line Utility", "OriginalFileName"=>"MpCmdRun.exe", "Product"=>"Microsoft® Windows® Operating System", "LogonId"=>"0x3e7", "Company"=>"Microsoft Corporation"}, "task"=>1, "process"=>{"thread_id"=>4400, "pid"=>2880}, "provider_name"=>"Microsoft-Windows-Sysmon", "channel"=>"Microsoft-Windows-Sysmon/Operational", "computer_name"=>"Windows10-WS1.acmeonions.com", "version"=>5, "provider_guid"=>"5770385F-C22A-43E0-BF4C-06F5698FFBD9", "record_id"=>13741}, "event_record_id"=>13741, "@version"=>"1", "timestamp"=>"2023-11-29T07:48:43.020873Z", "tags"=>["import", "elastic-agent", "input-so-manager", "beats_input_codec_plain_applied"], "input"=>{"type"=>"log"}, "import"=>{"file"=>"data.json", "id"=>"fff4261b8064add1ddbb546f4c59dece"}, "metadata"=>{"input"=>{"beats"=>{"host"=>{"ip"=>"172.17.1.1"}}}, "raw_index"=>"logs-import-so", "beat"=>"filebeat", "version"=>"8.10.4", "type"=>"_doc", "input_id"=>"logfile-logs-ac008d73-25c0-4ed5-a104-4876aa45d27f", "stream_id"=>"logfile-log.logs-ac008d73-25c0-4ed5-a104-4876aa45d27f", "pipeline"=>"logs-windows.sysmon_operational-1.24.0"}, "elastic_agent"=>{"snapshot"=>false, "id"=>"d9f14abd-247c-4981-894a-ff87ee155934", "version"=>"8.10.4"}, "data_stream"=>{"dataset"=>"import", "type"=>"logs", "namespace"=>"so"}, "cloud"=>{"region"=>"", "instance"=>{"id"=>"38268ba4-d95c-47f7-ac34-b0cbc716fd86"}, "provider"=>"huawei", "availability_zone"=>"nova", "service"=>{"name"=>"ECS"}}, "agent"=>{"version"=>"8.10.4", "id"=>"d9f14abd-247c-4981-894a-ff87ee155934", "type"=>"filebeat", "name"=>"so-manager", "ephemeral_id"=>"ac7bbe98-958d-4d60-a6f4-acdbcac892f6"}, "host"=>{"mac"=>["02-34-61-95-16-5F", "02-42-0E-FB-CC-A7", "02-42-9B-A0-58-3C", "0A-56-9F-DC-0B-6C", "0A-6F-8C-FE-D7-19", "26-A1-C3-A3-0C-39", "26-D9-04-56-9F-A1", "2A-4C-9D-01-05-17", "2A-E5-09-E4-3E-7C", "36-80-EA-2B-8E-3F", "52-CB-BD-49-BB-08", "6A-30-68-7C-1F-42", "6E-37-0C-12-A7-16", "72-55-C5-4B-83-73", "82-BF-CB-6A-2B-73", "86-8E-76-2C-A1-82", "8A-6E-95-70-8D-80", "96-1E-D7-C6-88-E5", "A2-51-7F-1F-D5-EC", "AE-5F-B8-32-AA-57", "AE-C9-21-B9-D6-9E", "B6-D1-7F-F1-CD-EC", "BE-B7-0C-17-16-1A", "E2-BA-27-EF-9C-00", "EA-50-B6-3F-DF-BA", "FA-16-3E-33-24-71", "FA-16-3E-94-9B-30"], "hostname"=>"so-manager", "id"=>"38268ba4d95c47f7ac34b0cbc716fd86", "name"=>"so-manager", "containerized"=>false, "architecture"=>"x86_64", "os"=>{"kernel"=>"5.15.0-101.103.2.1.el9uek.x86_64", "version"=>"9.3", "type"=>"linux", "name"=>"Oracle Linux Server", "family"=>"redhat", "platform"=>"ol"}, "ip"=>["192.168.75.15", "192.168.76.30", "172.17.0.1", "172.17.1.1"]}, "container"=>{"id"=>"data.json"}}], :response=>{"create"=>{"_index"=>"logs-import-so", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"pipeline with id [logs-windows.sysmon_operational-1.24.0] does not exist"}}}}

include reproduction steps

weslambert commented 11 months ago

Confirmed working with latest changes.