I'm trying to learn this for my studies. I installed helk using option 1.
Downloaded Mordor dataset & uploaded to kibana using kcat.
It populates mitre dashboards, but NOT global, sysmon or process
Appreciate your help, if I'm missing something
Provide the output of the following commands
Get operating system and version
for linux (except Mac) use: cat /etc/os-release
Get disk space, memory, processor cores, and docker storage echo -e "\nDocker Space:" && df -h /var/lib/docker; echo -e "\nMemory:" && free -g; echo -e "\nCores:" && getconf _NPROCESSORS_ONLN
Docker Space:
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 49G 17G 30G 36% /
Memory:
total used free shared buff/cache available
Mem: 7 7 0 0 0 0
Swap: 1 1 0
Cores:
4
Get output of the HELK docker containers: docker ps --filter "name=helk"
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
2e92c8ced16d confluentinc/ksqldb-server:latest "/usr/bin/docker/run" 2 days ago Up 2 hours 0.0.0.0:8088->8088/tcp, :::8
51e0405f3d47 otrf/helk-kafka-broker:2.4.0 "./kafka-entrypoint.…" 2 days ago Up 2 hours 0.0.0.0:9092->9092/tcp, :::9
6e21e926114f otrf/helk-zookeeper:2.4.0 "./zookeeper-entrypo…" 2 days ago Up 2 hours 2181/tcp, 2888/tcp, 3888/tcp
2328e19ce870 otrf/helk-logstash:7.6.2.1 "/usr/share/logstash…" 2 days ago Up 2 hours 0.0.0.0:3515->3515/tcp, :::3
cf6dadb00ac3 otrf/helk-nginx:0.3.0 "/opt/helk/scripts/n…" 2 days ago Up 2 hours 0.0.0.0:80->80/tcp, :::80->8
52f9430eac38 docker.elastic.co/kibana/kibana:7.6.2 "/usr/share/kibana/s…" 2 days ago Up 2 hours 5601/tcp
f8a0194c1256 docker.elastic.co/elasticsearch/elasticsearch:7.6.2 "/usr/share/elastics…" 2 days ago Up 2 hours 9200/tcp, 9300/tcp
Provide the HELK installation logs located at /var/log/helk-install.log if you are having install errors
Hit:1 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Get:2 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 DEP-11 Metadata [51.4 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2,212 kB]
Get:7 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 DEP-11 Metadata [57.9 kB]
Get:8 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 DEP-11 Metadata [2,464 B]
Get:9 http://us.archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,342 kB]
Get:10 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 DEP-11 Metadata [293 kB]
Get:11 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,749 kB]
Get:12 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,575 kB]
Get:13 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 DEP-11 Metadata [295 kB]
Get:14 http://us.archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 DEP-11 Metadata [2,468 B]
Get:15 http://us.archive.ubuntu.com/ubuntu bionic-backports/universe amd64 DEP-11 Metadata [9,272 B]
Fetched 7,841 kB in 5s (1,647 kB/s)
Reading package lists...
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
python3-click python3-colorama
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
libapr1 libaprutil1
The following NEW packages will be installed:
apache2-utils libapr1 libaprutil1
0 upgraded, 3 newly installed, 0 to remove and 1 not upgraded.
Need to get 259 kB of archives.
After this operation, 866 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu bionic/main amd64 libapr1 amd64 1.6.3-2 [90.9 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu bionic/main amd64 libaprutil1 amd64 1.6.1-2 [84.4 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-utils amd64 2.4.29-1ubuntu4.16 [84.0 kB]
Fetched 259 kB in 2s (129 kB/s)
Selecting previously unselected package libapr1:amd64.
(Reading database ... 158537 files and directories currently installed.)
Preparing to unpack .../libapr1_1.6.3-2_amd64.deb ...
Unpacking libapr1:amd64 (1.6.3-2) ...
Selecting previously unselected package libaprutil1:amd64.
Preparing to unpack .../libaprutil1_1.6.1-2_amd64.deb ...
Unpacking libaprutil1:amd64 (1.6.1-2) ...
Selecting previously unselected package apache2-utils.
Preparing to unpack .../apache2-utils_2.4.29-1ubuntu4.16_amd64.deb ...
Unpacking apache2-utils (2.4.29-1ubuntu4.16) ...
Setting up libapr1:amd64 (1.6.3-2) ...
Setting up libaprutil1:amd64 (1.6.1-2) ...
Setting up apache2-utils (2.4.29-1ubuntu4.16) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1.4) ...
Adding password for user helk
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
python3-click python3-colorama
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
libcurl4
The following NEW packages will be installed:
curl libcurl4
0 upgraded, 2 newly installed, 0 to remove and 1 not upgraded.
Need to get 378 kB of archives.
After this operation, 1,051 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libcurl4 amd64 7.58.0-2ubuntu3.14 [219 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 curl amd64 7.58.0-2ubuntu3.14 [159 kB]
Fetched 378 kB in 2s (155 kB/s)
Selecting previously unselected package libcurl4:amd64.
(Reading database ... 158586 files and directories currently installed.)
Preparing to unpack .../libcurl4_7.58.0-2ubuntu3.14_amd64.deb ...
Unpacking libcurl4:amd64 (7.58.0-2ubuntu3.14) ...
Selecting previously unselected package curl.
Preparing to unpack .../curl_7.58.0-2ubuntu3.14_amd64.deb ...
Unpacking curl (7.58.0-2ubuntu3.14) ...
Setting up libcurl4:amd64 (7.58.0-2ubuntu3.14) ...
Setting up curl (7.58.0-2ubuntu3.14) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1.4) ...
# Executing docker install script, commit: 93d2499759296ac1f9c510605fef85052a2c32be
+ sh -c apt-get update -qq >/dev/null
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
+ sh -c curl -fsSL "https://download.docker.com/linux/ubuntu/gpg" | gpg --dearmor --yes -o /usr/share/keyrings/docker-archive-keyring.gpg
gpg: WARNING: unsafe ownership on homedir '/home/user1/.gnupg'
+ sh -c echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu bionic stable" > /etc/apt/sources.list.d/docker.list
+ sh -c apt-get update -qq >/dev/null
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends docker-ce-cli docker-scan-plugin docker-ce >/dev/null
+ version_gte 20.10
+ [ -z ]
+ return 0
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq docker-ce-rootless-extras >/dev/null
+ sh -c docker version
Client: Docker Engine - Community
Version: 20.10.8
API version: 1.41
Go version: go1.16.6
Git commit: 3967b7d
Built: Fri Jul 30 19:54:08 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.8
API version: 1.41 (minimum version 1.12)
Go version: go1.16.6
Git commit: 75249d8
Built: Fri Jul 30 19:52:16 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.9
GitCommit: e25210fe30a0a703442421b0f60afac609f950a3
runc:
Version: 1.0.1
GitCommit: v1.0.1-0-g4144b63
docker-init:
Version: 0.19.0
GitCommit: de40ad0
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 633 100 633 0 0 1715 0 --:--:-- --:--:-- --:--:-- 1710
100 11.6M 100 11.6M 0 0 9756k 0 0:00:01 0:00:01 --:--:-- 9756k
Creating network "docker_helk" with driver "bridge"
Creating volume "docker_esdata" with local driver
Pulling helk-elasticsearch (docker.elastic.co/elasticsearch/elasticsearch:7.6.2)...
7.6.2: Pulling from elasticsearch/elasticsearch
Digest: sha256:59342c577e2b7082b819654d119f42514ddf47f0699c8b54dc1f0150250ce7aa
Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:7.6.2
Pulling helk-kibana (docker.elastic.co/kibana/kibana:7.6.2)...
7.6.2: Pulling from kibana/kibana
Digest: sha256:e8f3743e404462709663422056db2d5076a7a6bd6024f64aea1599b3014c63be
Status: Downloaded newer image for docker.elastic.co/kibana/kibana:7.6.2
Pulling helk-logstash (otrf/helk-logstash:7.6.2.1)...
7.6.2.1: Pulling from otrf/helk-logstash
Digest: sha256:b1135da506f40fc1d5861db7ba844486f3a08a57af3fdb8e301ab487f51a2ac1
Status: Downloaded newer image for otrf/helk-logstash:7.6.2.1
Pulling helk-nginx (otrf/helk-nginx:0.3.0)...
0.3.0: Pulling from otrf/helk-nginx
Digest: sha256:32eb6e39681849dc3bed36cfb95bd39b25f8c66d08965b6855f64eb2ee0668ba
Status: Downloaded newer image for otrf/helk-nginx:0.3.0
Pulling helk-zookeeper (otrf/helk-zookeeper:2.4.0)...
2.4.0: Pulling from otrf/helk-zookeeper
Digest: sha256:d8a7c57c03384f5ce2b6125505c1f8e2a020432de81bde3677fcc8009fc5cfd2
Status: Downloaded newer image for otrf/helk-zookeeper:2.4.0
Pulling helk-kafka-broker (otrf/helk-kafka-broker:2.4.0)...
2.4.0: Pulling from otrf/helk-kafka-broker
Digest: sha256:22b87b2e2c97157471af3db8a19e85c9184fa492fa8cd67cc57617c6abec6dce
Status: Downloaded newer image for otrf/helk-kafka-broker:2.4.0
Pulling helk-ksql-server (confluentinc/ksqldb-server:latest)...
latest: Pulling from confluentinc/ksqldb-server
Digest: sha256:a75f49a54d287356337f64dcf81d9ce8a8e1932e999904568b1abd867b3ca7c7
Status: Downloaded newer image for confluentinc/ksqldb-server:latest
Pulling helk-ksql-cli (confluentinc/ksqldb-cli:latest)...
latest: Pulling from confluentinc/ksqldb-cli
Digest: sha256:a75f49a54d287356337f64dcf81d9ce8a8e1932e999904568b1abd867b3ca7c7
Status: Downloaded newer image for confluentinc/ksqldb-cli:latest
Creating helk-elasticsearch ... done
Creating helk-kibana ... done
Creating helk-nginx ... done
Creating helk-logstash ... done
Creating helk-zookeeper ... done
Creating helk-kafka-broker ... done
Creating helk-ksql-server ... done
Creating helk-ksql-cli ... done
What version of HELK are you using
run the command from within the HELK repo run git log -1 --oneline
What version of Winlogbeat are you using if you are using Windows/WEF logs
stored logs imported into HELK using kcat
What steps did you take trying to fix the issue
jq shows no error on the imported json for either mordor sample or my own (attached)
I created a sample which had a format { "events": [ {evt1},\n{evt2},\n{evt3},\n ....]}. The key:value pairs in this sample are separated by NL/CR. Upon importing this sample using kcat, the HELK discovery page shows _jsonParse_failure and no data from the imported logs shows up here. Global dashboard for this sample shows number of lines in the sample log under global_count widget & the rest widgets, dashboards remained unpopulated.
I modified my sample logs to match mordor_project log format, i.e 1 line / event_log ex: {evt1}\n{evt2}\n{evt3}\n ..... This did not even got ingested (sample logs attached)
How could we replicate the issue
install HELK on ubuntu 18.04, choose option 1
import https://github.com/OTRF/Security-Datasets/blob/master/datasets/atomic/windows/lateral_movement/host/empire_wmic_add_user_backdoor.zip using kcat (winlogbeat as type)
Any additionally code or log context you would like to provide
see attached screen shots
Let me know, if you want the sample sysmon logs I tested with. Unable to attach here (github flagging me, even for a zipped attachment (3kb)
Describe the problem
I'm trying to learn this for my studies. I installed helk using
option 1. Downloaded Mordor dataset & uploaded to kibana using kcat. It populatesmitredashboards, but NOT global, sysmon or process Appreciate your help, if I'm missing somethingProvide the output of the following commands
Get operating system and version for linux (except Mac) use:
cat /etc/os-releaseGet disk space, memory, processor cores, and docker storage
echo -e "\nDocker Space:" && df -h /var/lib/docker; echo -e "\nMemory:" && free -g; echo -e "\nCores:" && getconf _NPROCESSORS_ONLNGet output of the HELK docker containers:
docker ps --filter "name=helk"Provide the HELK installation logs located at /var/log/helk-install.log if you are having install errors
What version of HELK are you using
run the command from within the HELK repo run
git log -1 --onelineWhat version of Winlogbeat are you using if you are using Windows/WEF logs
What steps did you take trying to fix the issue
{ "events": [ {evt1},\n{evt2},\n{evt3},\n ....]}. Thekey:valuepairs in this sample are separated by NL/CR. Upon importing this sample using kcat, the HELKdiscoverypage shows_jsonParse_failureand no data from the imported logs shows up here. Global dashboard for this sample shows number of lines in the sample log underglobal_countwidget & the rest widgets, dashboards remained unpopulated.How could we replicate the issue
https://github.com/OTRF/Security-Datasets/blob/master/datasets/atomic/windows/lateral_movement/host/empire_wmic_add_user_backdoor.zipusing kcat (winlogbeatas type)Any additionally code or log context you would like to provide
Any additional context or input you have
pictures, comments, etc.