Cyb3rWard0g / HELK

The Hunting ELK
GNU General Public License v3.0
3.76k stars 682 forks source link

Rule type blacklist not loading despite successful rule test #437

Open mutedmouse opened 4 years ago

mutedmouse commented 4 years ago

I have added a custom rule to helk-elastalert that tests blacklist hashes. This rule has been tested against sample data using elastalert-test-rule and returns the sample hit as seen below in the final output from the elastalert-test-rule command:

Would have written the following documents to writeback index (default is elastalert_status):

elastalert_status - {'endtime': datetime.datetime(2020, 3, 5, 15, 1, 8, 269247, tzinfo=tzutc()), '@timestamp': datetime.datetime(2020, 3, 5, 15, 1, 20, 928247, tzinfo=tzutc()), 'matches': 25, 'hits': 25, 'rule_name': 'Virusshare-blacklist-alert', 'starttime': datetime.datetime(2020, 3, 5, 0, 0, tzinfo=<UTC>), 'time_taken': 12.567991018295288}

When reviewing elastalert_status_status I can see that the rule is loading and matching. However no alert match_information is being shown in elastalert_status. All sigma and original rules are appearing as predicted.

My custom rule, heavily redacted is shown below and placed in the /etc/elastalert/rules directory in the helk-elastalert container:

alert:
- debug
description: Match MD5 of processes against virusshare hashes
filter:
- query:
    query_string:
      query: (hash_md5_lower:* AND event_id:1 AND 
 hash_md5_lower:9271fa29b49fac08e6a2a5824afb820f OR hash_md5_lower:9a95eb3084512939c7d6c9e09598f4ea))
index: logs-endpoint-winevent-sysmon-*
name: Virusshare-blacklist-alert_0
priority: 3
realert:
  minutes: 0
type: any

What can I do to get the match data to populate in elastalert status with the same level of information as the sigma rules do. For example, when sigma rules populate the elastalert_status index, I receive the entire match_body fields broken out from the original event.

Get operating system and version

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Get disk space, memory, processor cores, and docker storage

Docker Space:
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   92G   20G   72G  22% /

Memory:
              total        used        free      shared  buff/cache   available
Mem:             15          11           0           0           3           3
Swap:             7           0           7

Cores:
6
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                                                                              NAMES
3eb45b88f53e        confluentinc/cp-ksql-cli:5.1.3                        "/bin/sh"                28 hours ago        Up 12 minutes                                                                                          helk-ksql-cli
288884b93c2d        confluentinc/cp-ksql-server:5.1.3                     "/etc/confluent/dock…"   28 hours ago        Up 12 minutes       0.0.0.0:8088->8088/tcp                                                             helk-ksql-server
24b835000e90        otrf/helk-kafka-broker:2.3.0                          "./kafka-entrypoint.…"   28 hours ago        Up 12 minutes       0.0.0.0:9092->9092/tcp                                                             helk-kafka-broker
d2c86260d4b8        otrf/helk-spark-worker:2.4.4                          "./spark-worker-entr…"   28 hours ago        Up 12 minutes                                                                                          helk-spark-worker
3b1ac5645326        otrf/helk-elastalert:0.2.6                            "./elastalert-entryp…"   28 hours ago        Up 12 minutes                                                                                          helk-elastalert
4f52166ceb40        docker_helk-jupyter                                   "/opt/jupyter/script…"   28 hours ago        Up 12 minutes       8000/tcp, 8888/tcp                                                                 helk-jupyter
b4a0df7dc33e        otrf/helk-spark-master:2.4.4                          "./spark-master-entr…"   28 hours ago        Up 12 minutes       7077/tcp, 0.0.0.0:8080->8080/tcp                                                   helk-spark-master
f4f938666ecc        otrf/helk-zookeeper:2.3.0                             "./zookeeper-entrypo…"   28 hours ago        Up 12 minutes       2181/tcp, 2888/tcp, 3888/tcp                                                       helk-zookeeper
096976ad0d9e        otrf/helk-nginx:0.0.8                                 "/opt/helk/scripts/n…"   28 hours ago        Up 12 minutes       0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                                           helk-nginx
dd8cd0093af0        otrf/helk-logstash:7.5.2.2                            "/usr/share/logstash…"   28 hours ago        Up 12 minutes       0.0.0.0:3515->3515/tcp, 0.0.0.0:5044->5044/tcp, 0.0.0.0:8531->8531/tcp, 9600/tcp   helk-logstash
44128b03807a        docker.elastic.co/kibana/kibana:7.5.2                 "/usr/share/kibana/s…"   28 hours ago        Up 12 minutes       5601/tcp                                                                           helk-kibana
193a7167576b        docker.elastic.co/elasticsearch/elasticsearch:7.5.2   "/usr/share/elastics…"   28 hours ago        Up 12 minutes       9200/tcp, 9300/tcp                                                                 helk-elasticsearch
e819bc892a1fe3830f62ec4d33eeaa7f43eb2c51
Cyb3rWard0g commented 4 years ago

Have you checked the other indices? Apparently you get the match body in the elastlalert index https://elastalert.readthedocs.io/en/latest/elastalert_status.html

mutedmouse commented 4 years ago

Sry about the last one comment, I had not seen the update. I'll check the elastalert_status and send the results of the rule in this ticket tomorrow.

mutedmouse commented 4 years ago

Blacklist rule now works with match_body extraction on latest helk build. I will post the rule I used here and for users to reference as an example. There are a surprisingly few number of working examples so perhaps it can help others.

mutedmouse commented 4 years ago

Steps taken to build rule and resolve question are below:

Entering the docker container sudo docker exec -it -u0 helk-elastalert /bin/bash

Change user to elastalertuser: su elastalert

Creating and testing rule before applying vim /etc/elastalert/blacklist_hash.yml

add the following lines to the blacklist_hash.yml

alert:
- debug
description: Detects an blacklisted hash
filter:
- query:
    query_string:
      query: (event.code:1)

ignore_null: true
compare_key: hash_md5

blacklist:
  - "!file /etc/elastalert/blacklist.csv"

index: logs-*
name: Hash-Blacklist-Match_0
priority: 2 
realert:
  minutes: 0
type: blacklist

save the blacklist_hash.yml

Create your blacklist.csv lookup file vim /etc/elastalert/blacklist.csv add your hash or blacklist values separated by newlines here check you case as elastic sees all uppercase values for hashes save the blacklist.csv

Test you rule against recent data /usr/local/bin/elastalert-test-rule --config /etc/elastalert/config.yaml --days 1 /etc/elastalert/blacklist_hash.yml verify you have hits in the final json output

Copy your rule to production folder cp /etc/elastalert/blacklist_hash.yml /etc/elastalert/rules/

Using kibana Discovery page search index elastalert_status_status for "Hash-Blacklist-Match_0" Once it loads and elastalert runs this rule you will see it appear in the results.

Your results should now appear in the elastalert_status index with match_body information relating to the md5 matching value from your blacklist.csv file.

Hope this helps any who find this. The query is as basic as it comes and can be significantly improved upon but provides a syntactic example of a functional blacklist elastalert rule.

Thanks for the hint on where to look @Cyb3rWard0g and the awesome platform.

Cyb3rWard0g commented 4 years ago

Thank you very much @mutedmouse ! I will tag this to be added to the Wiki. Thank you!