taylorwalton
commented
1 month ago Hey @ioscanner can you share the json payload of the alert you are trying to use the exclusion feature on?
Open ioscanner opened 1 month ago
taylorwalton
commented
1 month ago Hey @ioscanner can you share the json payload of the alert you are trying to use the exclusion feature on?
ioscanner
commented
1 month ago Any rule I try to exclude gives me the same error: Missing 'rule_group1' in prompt. It seems the few versions back started creating all the pipeline entries. So seems a lot of the docs are outdated on how to install the new version and get it working. Everything seems to work except when provisioning a customer it has an issue with: DFIR-IRIS that will not allow me to provision a customer and I can't exclude any rules for Wazuh.
{
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
"data-raw": {
"integration": "wazuh-rule-exclusion",
"prompt": {
"data_win_system_eventRecordID": "2158046",
"data_win_eventdata_user": "user",
"agent_id": "200",
"agent_name": "agentname",
"gl2_remote_ip": "rip",
"data_win_system_eventID": "11",
"gl2_remote_port": 51704,
"agent_labels_customer": "soc",
"source": "dip",
"gl2_source_input": "665bcb09a23199664ba941XX",
"rule_level": 15,
"data_win_system_task": "11",
"timestamp_utc": "2024-07-10T23:51:14.588Z",
"data_win_system_threadID": "5288",
"rule_description": "Executable file dropped in folder commonly used by malware",
"gl2_source_node": "4475d18d-4677-49e1-902c-5079db9b963c",
"id": "1720655475.2181045517",
"rule_mitre_tactic": "Command and Control",
"gl2_accounted_message_size": 3794,
"data_win_eventdata_utcTime": "2024-07-10 23:51:14.576",
"streams": [
"665bd219a23199664ba9503a",
"6664df95cd1d3133564d59c3"
],
"rule_mitre_id": "T1105",
"gl2_message_id": "01J2FGY1JZGWN80P1Y5GAR1TXX",
"data_win_system_computer": "agent",
"data_win_eventdata_ruleName": "DLL",
"agent_ip": "192.168.x.x",
"data_win_eventdata_image": "C:\\WINDOWS\\system32\\cleanmgr.exe",
"true": 1720655475.958588,
"rule_groups": "sysmon, sysmon_eid11_detections, windows",
"data_win_system_keywords": "0x8000000000000000",
"data_win_system_level": "4",
"data_win_system_severityValue": "INFORMATION",
"data_win_eventdata_processGuid": "{5240f0a8-1e71-668f-29ea-010000002500}",
"rule_mitre_technique": "Ingress Tool Transfer",
"rule_firedtimes": 52,
"data_win_system_systemTime": "2024-07-10T23:51:14.5881438Z",
"rule_mail": true,
"data_win_eventdata_creationUtcTime": "2024-07-10 23:51:14.576",
"log_type": "wazuh",
"decoder_name": "windows_eventchannel",
"data_win_system_processID": "4996",
"data_win_system_channel": "Microsoft-Windows-Sysmon/Operational",
"syslog_level": "ALERT",
"data_win_system_providerName": "Microsoft-Windows-Sysmon",
"data_win_eventdata_processId": "6308",
"data_win_system_version": "2",
"data_win_system_providerGuid": "{5770385f-c22a-43e0-bf4c-06f5698ffbd9}",
"timestamp": "2024-07-10 23:51:20.415",
"data_win_system_opcode": "0",
"data_win_eventdata_targetFilename": "C:\\Users\\USER\\AppData\\Local\\Temp\\D1E98911-C1BD-4784-929F-14F476C3EC42\\WimProvider.dll",
"gl2_processing_error": "Replaced invalid timestamp value in message <4e690d80-3f17-11ef-adfb-ee8f761b6cad> with current time - Value <2024-07-10T18:51:15.260-0500> caused exception: Invalid format: \"2024-07-10T18:51:15.260-0500\" is malformed at \"T18:51:15.260-0500\".",
"message": {
"true": 1720655475.958588,
"timestamp": "2024-07-10T18:51:15.260-0500",
"rule": {
"level": 15,
"description": "Executable file dropped in folder commonly used by malware",
"id": "92213",
"mitre": {
"id": [
"T1105"
],
"tactic": [
"Command and Control"
],
"technique": [
"Ingress Tool Transfer"
]
},
"firedtimes": 52,
"mail": true,
"groups": [
"sysmon",
"sysmon_eid11_detections",
"windows"
]
},
"agent": {
"id": "200",
"name": "agentname",
"ip": "192.168.x.x",
"labels": {
"customer": "socfp"
}
},
"manager": {
"name": "mon"
},
"id": "1720655475.2181045517",
"decoder": {
"name": "windows_eventchannel"
},
"data": {
"win": {
"system": {
"providerName": "Microsoft-Windows-Sysmon",
"providerGuid": "{5770385f-c22a-43e0-bf4c-06f5698ffbd9}",
"eventID": "11",
"version": "2",
"level": "4",
"task": "11",
"opcode": "0",
"keywords": "0x8000000000000000",
"systemTime": "2024-07-10T23:51:14.5881438Z",
"eventRecordID": "2158046",
"processID": "4996",
"threadID": "5288",
"channel": "Microsoft-Windows-Sysmon/Operational",
"computer": "agentname",
"severityValue": "INFORMATION",
"message": "File created:\r\nRuleName: DLL\r\nUtcTime: 2024-07-10 23:51:14.576\r\nProcessGuid: {5240f0a8-1e71-668f-29ea-010000002500}\r\nProcessId: 6308\r\nImage: C:\\WINDOWS\\system32\\cleanmgr.exe\r\nTargetFilename: C:\\Users\\user\\AppData\\Local\\Temp\\D1E98911-C1BD-4784-929F-14F476C3EC42\\WimProvider.dll\r\nCreationUtcTime: 2024-07-10 23:51:14.576\r\nUser: agentname\\user"
},
"eventdata": {
"ruleName": "DLL",
"utcTime": "2024-07-10 23:51:14.576",
"processGuid": "{5240f0a8-1e71-668f-29ea-010000002500}",
"processId": "6308",
"image": "C:\\WINDOWS\\system32\\cleanmgr.exe",
"targetFilename": "C:\\Users\\user\\AppData\\Local\\Temp\\D1E98911-C1BD-4784-929F-14F476C3EC42\\WimProvider.dll",
"creationUtcTime": "2024-07-10 23:51:14.576",
"user": "agent\\user"
}
}
},
"location": "EventChannel"
},
"rule_id": "92213",
"manager_name": "mon",
"location": "EventChannel",
"data_win_system_message": "File created:\r\nRuleName: DLL\r\nUtcTime: 2024-07-10 23:51:14.576\r\nProcessGuid: {5240f0a8-1e71-668f-29ea-010000002500}\r\nProcessId: 6308\r\nImage: C:\\WINDOWS\\system32\\cleanmgr.exe\r\nTargetFilename: C:\\Users\\user\\AppData\\Local\\Temp\\D1E98911-C1BD-4784-929F-14F476C3EC42\\WimProvider.dll\r\nCreationUtcTime: 2024-07-10 23:51:14.576\r\nUser: agentname\\user"
}
},
"insecure": true
}Sorry I mean raw payload of the alert not of the json payload being sent to the copilot-ai-module. Are you using the Graylog content back as shown here: https://youtu.be/euFrHP0VkD8?si=TL_rrtShOD5Yq3IC
ioscanner
commented
1 month ago Ah I am at graylog 5.0.13, maybe I need to upgrade. I do have the content pack. Will just upgrading graylog be enough or will I have to reinstall the content pack?
ioscanner
commented
1 month ago Here is an alert. I upgraded to 6.0.3 of Graylog and now I just get when trying to exclude a rule: [Errno -2] Name or service not known
{ "data_win_system_eventRecordID": "1891977", "agent_id": "304", "agent_name": "TABLET", "gl2_remote_ip": "mon", "data_win_system_eventID": "4624", "gl2_remote_port": 40274, "rule_tsc": "CC6.8, CC7.2, CC7.3", "data_win_eventdata_impersonationLevel": "%%1833", "source": "mon", "gl2_source_input": "665bd795a23199664ba95bc6", "rule_level": 3, "data_win_eventdata_keyLength": "0", "gl2_processing_timestamp": "2024-07-13 22:05:39.197", "data_win_system_task": "12544", "timestamp_utc": "2024-07-13T21:10:09.866Z", "syslog_type": "wazuh", "data_win_system_threadID": "3736", "rule_description": "Windows logon success.", "gl2_source_node": "4475d18d-4677-49e1-902c-5079db9b963c", "id": "1720905010.1503758953", "gl2_processing_duration_ms": 3323792, "rule_mitre_tactic": "Defense Evasion, Persistence, Privilege Escalation, Initial Access", "gl2_accounted_message_size": 8653, "streams": [ "665bd795a23199664ba95be8" ], "rule_mitre_id": "T1078", "data_win_eventdata_virtualAccount": "%%1843", "gl2_message_id": "01J2Q22NZX0000003JF6EYPW6P", "data_win_eventdata_targetUserName": "SYSTEM", "data_win_system_computer": "TABLET", "data_win_eventdata_targetLogonId": "0x3e7", "data_win_eventdata_targetLinkedLogonId": "0x0", "data_win_eventdata_subjectUserName": "TABLET$", "agent_ip": "10.x.x.x", "data_win_eventdata_targetDomainName": "NT AUTHORITY", "true": 1720905010.753119, "data_win_eventdata_logonProcessName": "Advapi", "rule_hipaa": "164.312.b", "data_win_eventdata_logonType": "5", "_id": "09f8cc90-4164-11ef-8a61-ee8f761b6cad", "rule_groups": "windows, windows_security, authentication_success", "data_win_system_keywords": "0x8020000000000000", "data_win_system_level": "0", "data_win_eventdata_processName": "C:\\Windows\\System32\\services.exe", "gl2_receive_timestamp": "2024-07-13 21:10:15.405", "data_win_system_severityValue": "AUDIT_SUCCESS", "data_win_eventdata_elevatedToken": "%%1842", "rule_gdpr": "IV_32.2", "data_win_eventdata_subjectUserSid": "S-1-5-18", "rule_mitre_technique": "Valid Accounts", "rule_firedtimes": 113, "data_win_system_systemTime": "2024-07-13T21:10:09.8662435Z", "rule_mail": false, "rule_pci_dss": "10.2.5", "data_win_eventdata_subjectDomainName": "WORKGROUP", "rule_nist_800_53": "AC.7, AU.14", "decoder_name": "windows_eventchannel", "data_win_eventdata_subjectLogonId": "0x3e7", "data_win_system_processID": "1228", "data_win_system_channel": "Security", "syslog_level": "INFO", "data_win_system_providerName": "Microsoft-Windows-Security-Auditing", "data_win_eventdata_processId": "0x4a8", "data_win_system_version": "3", "data_win_system_providerGuid": "{54849625-5478-4994-a5ba-3e3b0328c30d}", "timestamp": "2024-07-13T22:05:39.197Z", "data_win_system_opcode": "0", "gl2_processing_error": "Replaced invalid timestamp value in message <09f8cc90-4164-11ef-8a61-ee8f761b6cad> with current time - Value <2024-07-13T16:10:10.433-0500> caused exception: Invalid format: \"2024-07-13T16:10:10.433-0500\" is malformed at \"T16:10:10.433-0500\".", "data_win_eventdata_authenticationPackageName": "Negotiate", "message": "{\"true\":1720905010.753119,\"timestamp\":\"2024-07-13T16:10:10.433-0500\",\"rule\":{\"level\":3,\"description\":\"Windows logon success.\",\"id\":\"60106\",\"mitre\":{\"id\":[\"T1078\"],\"tactic\":[\"Defense Evasion\",\"Persistence\",\"Privilege Escalation\",\"Initial Access\"],\"technique\":[\"Valid Accounts\"]},\"firedtimes\":113,\"mail\":false,\"groups\":[\"windows\",\"windows_security\",\"authentication_success\"],\"gdpr\":[\"IV_32.2\"],\"gpg13\":[\"7.1\",\"7.2\"],\"hipaa\":[\"164.312.b\"],\"nist_800_53\":[\"AC.7\",\"AU.14\"],\"pci_dss\":[\"10.2.5\"],\"tsc\":[\"CC6.8\",\"CC7.2\",\"CC7.3\"]},\"agent\":{\"id\":\"304\",\"name\":\"TABLET\",\"ip\":\"10.x.x.x\"},\"manager\":{\"name\":\"mon\"},\"id\":\"1720905010.1503758953\",\"decoder\":{\"name\":\"windows_eventchannel\"},\"data\":{\"win\":{\"system\":{\"providerName\":\"Microsoft-Windows-Security-Auditing\",\"providerGuid\":\"{54849625-5478-4994-a5ba-3e3b0328c30d}\",\"eventID\":\"4624\",\"version\":\"3\",\"level\":\"0\",\"task\":\"12544\",\"opcode\":\"0\",\"keywords\":\"0x8020000000000000\",\"systemTime\":\"2024-07-13T21:10:09.8662435Z\",\"eventRecordID\":\"1891977\",\"processID\":\"1228\",\"threadID\":\"3736\",\"channel\":\"Security\",\"computer\":\"TABLET\",\"severityValue\":\"AUDIT_SUCCESS\",\"message\":\"\\"An account was successfully logged on.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-18\r\n\tAccount Name:\t\tTABLET$\r\n\tAccount Domain:\t\tWORKGROUP\r\n\tLogon ID:\t\t0x3E7\r\n\r\nLogon Information:\r\n\tLogon Type:\t\t5\r\n\tRestricted Admin Mode:\t-\r\n\tRemote Credential Guard:\t-\r\n\tVirtual Account:\t\tNo\r\n\tElevated Token:\t\tYes\r\n\r\nImpersonation Level:\t\tImpersonation\r\n\r\nNew Logon:\r\n\tSecurity ID:\t\tS-1-5-18\r\n\tAccount Name:\t\tSYSTEM\r\n\tAccount Domain:\t\tNT AUTHORITY\r\n\tLogon ID:\t\t0x3E7\r\n\tLinked Logon ID:\t\t0x0\r\n\tNetwork Account Name:\t-\r\n\tNetwork Account Domain:\t-\r\n\tLogon GUID:\t\t{00000000-0000-0000-0000-000000000000}\r\n\r\nProcess Information:\r\n\tProcess ID:\t\t0x4a8\r\n\tProcess Name:\t\tC:\\Windows\\System32\\services.exe\r\n\r\nNetwork Information:\r\n\tWorkstation Name:\t-\r\n\tSource Network Address:\t-\r\n\tSource Port:\t\t-\r\n\r\nDetailed Authentication Information:\r\n\tLogon Process:\t\tAdvapi \r\n\tAuthentication Package:\tNegotiate\r\n\tTransited Services:\t-\r\n\tPackage Name (NTLM only):\t-\r\n\tKey Length:\t\t0\r\n\r\nThis event is generated when a logon session is created. It is generated on the computer that was accessed.\r\n\r\nThe subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.\r\n\r\nThe logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).\r\n\r\nThe New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.\r\n\r\nThe network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.\r\n\r\nThe impersonation level field indicates the extent to which a process in the logon session can impersonate.\r\n\r\nThe authentication information fields provide detailed information about this specific logon request.\r\n\t- Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.\r\n\t- Transited services indicate which intermediate services have participated in this logon request.\r\n\t- Package name indicates which sub-protocol was used among the NTLM protocols.\r\n\t- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.\\"\"},\"eventdata\":{\"subjectUserSid\":\"S-1-5-18\",\"subjectUserName\":\"TABLET$\",\"subjectDomainName\":\"WORKGROUP\",\"subjectLogonId\":\"0x3e7\",\"targetUserSid\":\"S-1-5-18\",\"targetUserName\":\"SYSTEM\",\"targetDomainName\":\"NT AUTHORITY\",\"targetLogonId\":\"0x3e7\",\"logonType\":\"5\",\"logonProcessName\":\"Advapi\",\"authenticationPackageName\":\"Negotiate\",\"logonGuid\":\"{00000000-0000-0000-0000-000000000000}\",\"keyLength\":\"0\",\"processId\":\"0x4a8\",\"processName\":\"C:\\\\Windows\\\\System32\\\\services.exe\",\"impersonationLevel\":\"%%1833\",\"virtualAccount\":\"%%1843\",\"targetLinkedLogonId\":\"0x0\",\"elevatedToken\":\"%%1842\"}}},\"location\":\"EventChannel\"}", "rule_id": "60106", "manager_name": "mon", "data_win_eventdata_logonGuid": "{00000000-0000-0000-0000-000000000000}", "rule_gpg13": "7.1, 7.2", "data_win_eventdata_targetUserSid": "S-1-5-18", "location": "EventChannel", "rule_group3": "authentication_success", "data_win_system_message": "\"An account was successfully logged on.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-18\r\n\tAccount Name:\t\tTABLET$\r\n\tAccount Domain:\t\tWORKGROUP\r\n\tLogon ID:\t\t0x3E7\r\n\r\nLogon Information:\r\n\tLogon Type:\t\t5\r\n\tRestricted Admin Mode:\t-\r\n\tRemote Credential Guard:\t-\r\n\tVirtual Account:\t\tNo\r\n\tElevated Token:\t\tYes\r\n\r\nImpersonation Level:\t\tImpersonation\r\n\r\nNew Logon:\r\n\tSecurity ID:\t\tS-1-5-18\r\n\tAccount Name:\t\tSYSTEM\r\n\tAccount Domain:\t\tNT AUTHORITY\r\n\tLogon ID:\t\t0x3E7\r\n\tLinked Logon ID:\t\t0x0\r\n\tNetwork Account Name:\t-\r\n\tNetwork Account Domain:\t-\r\n\tLogon GUID:\t\t{00000000-0000-0000-0000-000000000000}\r\n\r\nProcess Information:\r\n\tProcess ID:\t\t0x4a8\r\n\tProcess Name:\t\tC:\Windows\System32\services.exe\r\n\r\nNetwork Information:\r\n\tWorkstation Name:\t-\r\n\tSource Network Address:\t-\r\n\tSource Port:\t\t-\r\n\r\nDetailed Authentication Information:\r\n\tLogon Process:\t\tAdvapi \r\n\tAuthentication Package:\tNegotiate\r\n\tTransited Services:\t-\r\n\tPackage Name (NTLM only):\t-\r\n\tKey Length:\t\t0\r\n\r\nThis event is generated when a logon session is created. It is generated on the computer that was accessed.\r\n\r\nThe subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.\r\n\r\nThe logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).\r\n\r\nThe New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.\r\n\r\nThe network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.\r\n\r\nThe impersonation level field indicates the extent to which a process in the logon session can impersonate.\r\n\r\nThe authentication information fields provide detailed information about this specific logon request.\r\n\t- Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.\r\n\t- Transited services indicate which intermediate services have participated in this logon request.\r\n\t- Package name indicates which sub-protocol was used among the NTLM protocols.\r\n\t- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.\"", "rule_group2": "windows_security", "rule_group1": "windows" }
taylorwalton
commented
1 month ago Name or service not known sounds like it cannot resolve a hostname. How is your docker-compose.yml defined? Within the copilot-ai-module container can it access the internet? The container will need to access OpenAIs API service
ioscanner
commented
1 month ago @taylorwalton Seems the docker-compose.yml didn't have dns settings. I added DNS settings to the docker-compose.yml file. The error: Name or service not known is fixed, but still can't exclude the rule still gives me the error: Missing 'rule_group1' in prompt.
Any update on how to resolve this? I still can't seem to resolve I either get the Name or service not known or Missing rule_group1.
ioscanner
commented
3 weeks ago I wonder if this has to do with the deployment pack? Because I installed all of this before you made those major changes and the deployment pack was already installed. I wonder if it is missing the changes to the deployment pack. Is there a way to remove the deployment pack to re-deploy?
taylorwalton
commented
3 weeks ago Yes, you will need to ensure the content pack has been applied. Currently there is no way to redeploy a content pack. Can you share the extractors for your Wazuh Input?
I'd expect to see the below 3:
ioscanner
commented
2 weeks ago Any update on this?
@taylorwalton Thanks for the reply. I am still getting the Name or service not known or Missing rule_group1 I do have DNS as I can use the threat intel. I just wonder if something broke, I did the install before you automated a lot of this stuff. It does seem there is a place in graylog to delete the contact pack, I didn't try yet.
The issues I am still facing:
I can't provision a customer as it seems to have an issue at the very end with iris, but it does create the customer. Failed to add user 1 to customers {customer_name‘: ‘IrislnitialClient’, customer_id‘: 1, I deleted the customer IrisInitialClient and it just shows the next customer it gets. But when I check it does create the customer. I am running [IRIS v2.3.7]
Can't exclude wazuh alerts (not sure if this is related to customer provision issue) Error: [Errno -2] Name or service not known. The logs in Copilot has this: /api/wazuh_manager/rule/exclude Invalid 'rule_group1' or 'rule_group3'. At least one must be 'windows'.
Can't create a soc alert: Customer code customer1 is not valid. Has the customer been provisioned? The customer is created, but not provisioned as it errors (flashes): Failed to add user 1 to customers {customer_name‘: ‘IrislnitialClient’, customer_id‘: 1, Even with the error the new customer is created in iris. This is also in the log in copilot log in the gui: /api/soc/general_alert/create Failed to collect alert details: 1 validation error for GenericSourceModel timestamp_utc field required (type=value_error.missing)
-Seems you removed nuclei from the docker config. I added it back and the scan hosts now works.
Everything seems to be around provisioning finishing.
Below are the extractors:
When I go to Alerts and select to exclude alert in Wazuh it give error: Missing 'rule_group1' in prompt.
The graylog pipeline has the rules. So not sure what more it needs. I did just update of the docker images, I notice new features cool, but still have yet to get this working correctly.