Suggestion to add Praeco as elastalert visualizer

priamai commented 3 years ago

Hi there, is anybody working on adding Praeco to manage and visualize the Alert rules? There is a full docker configuration here: https://github.com/johnsusek/praeco My understanding is that you can simply add the web app into the docker composer file and point it to the ElasticSearch server.

priamai commented 3 years ago

With the help of the author I managed to replicate the docker container and add a few rules. Now the final challenge I believe is for each rule to match the template to avoid that error. I need to dig a bit more into the code.

priamai commented 3 years ago

I also found an interesting blog where they talk about the same issue:

https://medium.com/@ibrahim.ayadhi/hello-and-welcome-to-our-new-article-which-will-be-covering-the-alerting-part-in-our-socaas-136cf6258c49

"Unfortunately, this rule can’t be directly used by ( Praeco/Elastalert-Server ) because of several missing fields. So, you can pick the important information from this rule (the query string) and create your own rule in Praeco interface with this information. This tool is very important because it helps you to gather information about a lot of rules and their query strings. Note: Sometimes you have to check your logs in (Kibana → Discover Interface) and their fields to make sure that name fields in your logs match with name fields in sigma rules. If your fields show a yellow error, go to index pattern, choose the index to match and click refresh fields."

My aim now is to first generate the rules via sigma converter and then create templates via a script. The templates which are easier to generate can then be converted into rules.

priamai commented 3 years ago

Hi @neu5ron, I have an early xmas gift for you.

I will post my docker configuration on github and you can integrate into HELK. I can also try to integrate into HELK via merge request.

OHHHH OHHHHH OHHHHHHH

neu5ron commented 3 years ago

@priamai awesome! Do you have the configs to share?

priamai commented 3 years ago

@neu5ron fork here and working on this branch: https://github.com/priamai/HELK/tree/praeco

I will let you one you can test it.

priamai commented 3 years ago

I just found out during the integration test that Praeco webapp comes packaged with Nginx which is interfering with the HELK Nginx configuration. I have to find a way maybe to disable it and then apply the same config into HELK. Argh!

priamai commented 3 years ago

@neu5ron if you want to try this commit: I have added an option 3 for the new praeco setup. The praeco up fails to start:

Starting Nginx
nginx: [emerg] host not found in upstream "helk-elastalert-praeco" in /etc/nginx/conf.d/default.conf:11

Line 11: proxy_pass http://helk-elastalert-praeco:3030/;

I am not sure why it is unable to find the host which I have defined in the corresponding docker compoese file with the proper dependencies.

Problem solved: I missed the network config in the docker composer file and now is all working. Last step is to add the rules conversion and we should be ready.

priamai commented 3 years ago

By the way this is a POC to show that the configuration (outside HELK) with praeco is working: https://gitlab.com/priampraeco/praecoes7/-/tree/master Good for testing or educational purposes.

erezhazan1 commented 3 years ago

By the way this is a POC to show that the configuration (outside HELK) with praeco is working: https://gitlab.com/priampraeco/praecoes7/-/tree/master Good for testing or educational purposes.

I got curious and wanted to see what it is, but unfortunately, it needs some modification since it doesn't work out of the box

robomotic commented 3 years ago

By the way this is a POC to show that the configuration (outside HELK) with praeco is working: https://gitlab.com/priampraeco/praecoes7/-/tree/master Good for testing or educational purposes.

I got curious and wanted to see what it is, but unfortunately, it needs some modification since it doesn't work out of the box

Can you describe the issue and attach here the relevant logs ?

priamai commented 3 years ago

Hi @neu5ron, I have completed the branch and you can now test it: https://github.com/priamai/HELK/tree/praeco it was also very simple to rename the HELK rules. There are a few things to notice:

praeco web ui is available at port 8080
I did modify the NGINX to proxy the /praeco url to it but it has some issues (I am not experienced enough in nginx)
I have made a few changes to the install and uninstall script but you can safely ignore them
I have discovered a bug and reported it to praeco devs, it will be fixed in the next release
I don't fully understand the sigma conversion script e.g. it is still working and the rules are generated (also copied locally to speed up the diff process) but I have not added the converted rules into praeco (just the HELK ones)
I have not tested the alert rules with live data (I will do that in the following weeks)

Feel free to reach me out here on in private if you have any issues merging/testing. Cheers.

neu5ron commented 3 years ago

elastalert has really died - @priamai have you found a fork/branch of elastalert?

defensivedepth commented 3 years ago

https://github.com/jertel/elastalert2

Cyb3rWard0g / HELK

Suggestion to add Praeco as elastalert visualizer #525