search

brimdata / zui

Zui is a powerful desktop application for exploring and working with data. The official front-end to the Zed lake.
https://www.brimdata.io/download/
Other
1.8k stars 132 forks source link

Allow usage of local Suricata rules #3047

Closed colder1989 closed 6 months ago

colder1989 commented 6 months ago

When i try to add suricata rules or other yaml setting i have always this error, and then i need to reset to default setting because broke the setting window. precisely I would like to import the rules from https://github.com/AssoEchap/stalkerware-indicators

Screenshots immagine_2024-04-11_152144692

Desktop Info

philrz commented 6 months ago

@colder1989: Sorry you're having difficulty. Indeed, you've hit some problems in an area that we know could use some improvement. I'll try to explain what's going on and see if I can offer you an interim fix while we improve.

Based on how your question is worded I suspect there might be some confusion regarding the Brimcap YAML Config File setting that you appear to be using. It sounded like you were trying to use it to "add Suricata rules", but that's not what it's intended to do (at least not directly). The YAML file pointed at by that setting is intended to be a file for configuring Brimcap, which is the tool bundled with Zui that invokes Zeek and Suricata, which are also bundled with Zui. The "docs" link in that entry of the Settings menu points at this Custom Brimcap Config article which goes into detail on how the YAML file is structured and what can be accomplished with it. Ultimately, what you're trying to achieve can be done with this approach, but there's more steps to it than just adding a Suricata rules file. Instead it assumes you're maintaining your own customized Suricata installation that's separate from the one that's bundled with Zui, and that Suricata installation could be configured with the additional rules or any other customizations you require.

However, there are a couple known caveats with that approach. First, it has been a while since that Custom Brimcap Config article has been updated and it needs to be brought current. I just happened to finish an updated draft today that's currently being reviewed, so you can refer to that draft here. Also, there's been a known issue with the Brimcap YAML Config File setting in Zui which has been tracked in #2949. I suspect that's the problem you were hitting when you spoke of having to "reset to default setting because broke the setting window". That issue has been fixed in the code base, but a new GA release of Zui that would include the fix is not expected for a couple weeks. In the interim, you could start taking advantage of the fix by installing Zui Insiders which is a daily build that can be installed alongside regular Zui.

All that having been said... I totally understand wanting to just add some additional Suricata rules and have Zui leverage them without having to setup a wholly separate Suricata installation and a custom Brimcap config to go with it. You're not the first person to have this request, and I think I see a way we could add it as a Zui enhancement pretty easily, so I intend to work on that shortly. In the meantime, if you're interested in a quick hack you can use with your regular Zui install, I can offer you an option.

I've attached an executable suricataupdater.txt that can replace the one that shipped with Zui (I renamed it to the .txt extension because most firewalls would prevent the transfer of a .exe file.) Backup or rename the existing updater which will be located at %USERPROFILE%\AppData\Local\Programs\Zui\resources\app.asar.unpacked\zdeps\suricata\suricataupdater.exe and download the suricataupdater.txt into that folder and rename its extension to .exe. This modified updater has been modified to look for a file of additional rules in the path \temp\suricata.rules on whichever drive Zui is installed, e.g., if Zui is installed on the C: drive then it would look in C:\temp\suricata.rules. You can then download your desired rules from https://github.com/AssoEchap/stalkerware-indicators/raw/master/generated/suricata.rules to that \temp\suricata.rules location and when you close/relaunch Zui this new suricataupdater.exe will run and it should find those additional rules and include them along with the default Emerging Threats Open set. You can confirm by looking for the "stalkerware" entries in the final rule set that will be assembled post-update at %APPDATA%\Zui\plugins\brimcap\storage\suricata\rules\suricata.rules. However, keep in mind this is a temporary hack. In addition to relying on this hard-coded path, if you were to reinstall/upgrade Zui, the original suricataupdater.exe would overwrite this one.

My ultimate goal will be to add a new option in Zui's Settings menu that would allow you specify a pathname for such additional rules. If I'm successful in putting an enhancement together for that soon, perhaps you could help test it out. In the meantime, let me know if you have any problems trying the hack or if you decide to go the Custom Brimcap Config route and need any help with that.

colder1989 commented 6 months ago

thank you very much for the clear explanation and for the hack... I'll try as soon as I can... also I don't know if it could be a good idea but it wouldn't be bad to have an export in pdf... I'll explain my goal: I'm basically using a program called Spyguard ( https://github.com/SpyGuard/spyguard) which allows the analysis of a device connected to a hotspot network that creates... the analysis is carried out via zeek and suricata... the very good thing is the report it carries out at the end of the analysis. .the limit is that the device must be connected via the hotspot..I instead would like to carry out a remote scan so I created a VPN with wireguard and then via tshark I also carry out the traffic analysis remotely.... but from here on, creating a report to give to someone who doesn't understand is very difficult... for now I've created a small program in python that reads the suricata scan and prints the report in PDF looking at it from a rules database if there are any alerts (stalkeware). I don't know if you understood much but this is more or less what I needed from Zui.. thanks again

philrz commented 6 months ago

@colder1989: Regarding "export in PDF", have you been able to create the contents of your report in Zui using a Zed query run against the logs generated by importing a pcap, or have you been blocked on even attempting that because of the reasons you opened this issue? If the feature request is truly just wanting to have the ability to export one of Zui's query outputs as PDF, that certainly makes sense as an enhancement and I see there's libraries we could maybe leverage to accomplish that quickly, so please confirm if that's what you have in mind and if so I'll be happy to open a request and size the effort with our lead UX developer. However, it wasn't clear from your question if you were hoping to also/instead recreate some additional specific functionality/reporting from SpyGuard in Zui. If that were the case, Zui being open source, obviously anyone would be free to attempt enhancing it however they wish, but I'd set expectations that anything beyond a simple "export query results as PDF" is unlikely to be taken up with priority by the core Dev team here at Brim Data that looks after the project. While Zui/Zed have roots in the network security use cases, for the past few years the team has been largely focused on making the projects broadly applicable to all kinds of data and scaling it up and that remains a focus, so we're unlikely to use our limited resources on heavier development efforts that impact just a narrower use case like security. Hope that makes sense. Let me know what you think.

colder1989 commented 6 months ago

I opened the post due to the problem of the suricata rules, I saw a video in a previous issue where you tried to load rules and so I thought it was possible. but I understood that it was just an example to show that the setting was broken. then in addition I gave feedback on the possibility of a pdf report. obviously I'm happy if this function is implemented but other aspects certainly have priority.

philrz commented 6 months ago

@colder1989: I've made a first attempt at adding support for pointing at additional local Suricata rules in Zui. This should be an easier approach than using the Suricata updater hack I attached previously. You can download a test Windows installer of Zui with that enhancement from https://storage.googleapis.com/brim-public-test-data/Zui_Setup_1.7.0.exe. The text and video at #3049 show how to make use of it with the stalkerware rules you linked to previously. Please do let me know if you're successful with it. Thanks!

philrz commented 6 months ago

Verified in Zui commit f1aef59.

The attached video shows the feature in use with the the attached sample data example.pcap.gz (after uncompressing) which contains traffic of doing a ping to one of the addresses that would be flagged by the rules at https://github.com/AssoEchap/stalkerware-indicators/raw/master/generated/suricata.rules noted above. Loading the test pcap as a baseline into Zui with default settings, no alert event is generated because the default Emerging Threats Open rule set does not flag this traffic. I then enter Zui's Setitngs menu and in the Brimcap Settings section click the button for the Local Suricata Rules Folder and select a folder to which I've downloaded the Stalkerware rules. Once the selection is made Zui immediately triggers a re-update of its assembled Suricata rules to include whatever rules are found in that folder, and after a few seconds these Stalkerware rules are added to the assembled final set alongside the Emerging Threats Open ones. Re-importing the test pcap, now we see the expected alert event.

https://github.com/brimdata/zui/assets/5934157/dae4b776-624e-4f8c-895a-5ccaa3601d68

@colder1989: I've also confirmed that this enhancement is working the same in the most recent Zui Insiders release 1.7.1-6. Therefore, if you want to make use of this enhancement in the short term you should use that rather than any of the early hacks I pointed to previously in this issue. The enhancement will also be in the next GA Zui release which I expect will come out in the next couple weeks.