search

Cyb3rWard0g / HELK

The Hunting ELK
GNU General Public License v3.0
3.73k stars 675 forks source link

Question: Zeek Installed On HELK #532

Open saroyaj opened 3 years ago

saroyaj commented 3 years ago

I have setup zeek running the host where HELK is running. The zeek logs are being populated as json under /opt/bro/logs/current/ no issues there.

root@helk:~# tail -1 /opt/bro/logs/current/conn.log {"ts":1608507941.199201,"uid":"CqhHBC4FK0Z49tBwF5","id.orig_h":"x.x.x.x","id.orig_p":49773,"id.resp_h":"y.y.y.y","id.resp_p":53,"proto":"udp","service":"dns","duration":0.0004978179931640625,"orig_bytes":31,"resp_bytes":106,"conn_state":"SF","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"Dd","orig_pkts":1,"orig_ip_bytes":59,"resp_pkts":1,"resp_ip_bytes":134}

I have installed filebeat to send these logs to kafka but i dont see them appear in kibana.

root@helk:~# cat /etc/filebeat/filebeat.yml [...trimmed...] output.kafka: hosts: ["HELK-IP:9092"] topic: "helkzeek" max_retries: 2 max_message_bytes: 1000000 [...trimmed...]

filebeat zeek module is enabled and no config issues in /etc/filebeat/modules.d/zeek.yml

Am I missing something ?

saroyaj commented 3 years ago

changed the topic name to 'filebeat' and i see the logs coming into kafka

kafkauser@e71f29748d57:~/scripts$ /opt/helk/kafka/bin/kafka-console-consumer.sh --bootstrap-server helk-kafka-broker:9092 --topic filebeat --from-beginning

but still noting in kibana.

saroyaj commented 3 years ago

Also read this topic: https://github.com/Cyb3rWard0g/HELK/issues/370

But the logs from filebeat dont appear under indexme-*

priamai commented 3 years ago

Could you dump in json files from kafkacat (as described here ) and I will try to replicate on my setup. I am assuming @jsinix also don't see the logs in indexme-* ? Cheers.

saroyaj commented 3 years ago

when i replayed some of the data using kafkacat manually it showed up in indexme-*

root@helk:~# tail -50 /opt/bro/logs/current/conn.log > file.json root@helk:~# kafkacat -b KAFKA-IP:9092 -t zeek -P -l file.json

Wondering if there is something wrong with filebeat config ?

filebeat.inputs:

filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false

setup.template.settings: index.number_of_shards: 1

setup.kibana:

output.kafka: codec.format: string: '%{[@timestamp]} %{[message]}' hosts: ["127.0.0.1:9092"] topic: "zeek" partition.round_robin: reachable_only: false required_acks: 1 compression: gzip max_message_bytes: 1000000

processors:

saroyaj commented 3 years ago

root@helk:~# filebeat test output -e -c /etc/filebeat/filebeat.yml 2020-12-22T14:09:14.737Z INFO instance/beat.go:645 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat] 2020-12-22T14:09:14.737Z INFO instance/beat.go:653 Beat ID: 8fef759b-78c5-4a26-be87-7e27816225e2 Kafka: 127.0.0.1:9092... parse host... OK dns lookup... OK addresses: 127.0.0.1 dial up... OK

root@helk:~# filebeat test output -e -c /etc/filebeat/modules.d/zeek.yml 2020-12-22T14:10:39.631Z INFO instance/beat.go:645 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat] 2020-12-22T14:10:39.631Z INFO instance/beat.go:653 Beat ID: 8fef759b-78c5-4a26-be87-7e27816225e2 Error initializing output: output type undefined

root@helk:~# cat /etc/filebeat/modules.d/zeek.yml

saroyaj commented 3 years ago

root@helk:~# filebeat -e -c /etc/filebeat/filebeat.yml [...trimmed...] /usr/local/go/src/runtime/asm_amd64.s:1373 2020-12-22T16:15:32.314Z ERROR [kafka] kafka/client.go:147 Dropping event: key not found github.com/elastic/beats/v7/libbeat/common.init /go/src/github.com/elastic/beats/libbeat/common/mapstr.go:41 runtime.doInit /usr/local/go/src/runtime/proc.go:5474 [...trimed...]

saroyaj commented 3 years ago

root@helk:~# systemctl status filebeat ● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch. Loaded: loaded (/lib/systemd/system/filebeat.service; disabled; vendor preset: enabled) Active: active (running) since Tue 2020-12-22 16:39:05 UTC; 39s ago Docs: https://www.elastic.co/products/beats/filebeat Main PID: 78434 (filebeat) Tasks: 11 (limit: 17927) CGroup: /system.slice/filebeat.service └─78434 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/file

Dec 22 16:39:05 helk filebeat[78434]: 2020-12-22T16:39:05.397Z INFO log/harvester.go:302 Harvester started for file: /opt/bro/logs/current/conn.log Dec 22 16:39:06 helk filebeat[78434]: 2020-12-22T16:39:06.397Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to kafka(192.168.66.3:9092) Dec 22 16:39:06 helk filebeat[78434]: 2020-12-22T16:39:06.397Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer Dec 22 16:39:06 helk filebeat[78434]: 2020-12-22T16:39:06.398Z INFO [publisher] pipeline/retry.go:223 done Dec 22 16:39:06 helk filebeat[78434]: 2020-12-22T16:39:06.398Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to kafka(192.168.66.3:9092) est Dec 22 16:39:15 helk filebeat[78434]: 2020-12-22T16:39:15.397Z INFO log/harvester.go:302 Harvester started for file: /opt/bro/logs/current/dns.log Dec 22 16:39:15 helk filebeat[78434]: 2020-12-22T16:39:15.397Z INFO log/harvester.go:302 Harvester started for file: /opt/bro/logs/current/ssl.log Dec 22 16:39:35 helk filebeat[78434]: 2020-12-22T16:39:35.259Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics" Dec 22 16:39:35 helk filebeat[78434]: 2020-12-22T16:39:35.397Z INFO log/harvester.go:302 Harvester started for file: /opt/bro/logs/current/files.log Dec 22 16:39:35 helk filebeat[78434]: 2020-12-22T16:39:35.398Z INFO log/harvester.go:302 Harvester started for file: /opt/bro/logs/current/x509.log

saroyaj commented 3 years ago

output.kafka: enabled: true hosts: ["127.0.0.1:9092"] topic: filebeat compression: gzip max_message_bytes: 1000000

priamai commented 3 years ago

Yeah just reading now: the fact that kafkacat is working and filebeat not is already a giveaway. I would like to focus on this error:

root@helk:~# filebeat -e -c /etc/filebeat/filebeat.yml
[...trimmed...]
/usr/local/go/src/runtime/asm_amd64.s:1373
2020-12-22T16:15:32.314Z ERROR [kafka] kafka/client.go:147 Dropping event: key not found

I think it is related to the codec format used that is not including the key in your last configuration.

Previously you defined:

codec.format:
string: '%{[@timestamp]} %{[message]}'

I think this is what Kafka expects where the timestamp is the key and message is the value.

Could you try to put that back and see if the key missing error disappear?

saroyaj commented 3 years ago

That error of "Dropping event: key not found" I had fixed. Forgot to post that. But funny thing, My last comment above was 9 hours ago. I did not touch the keyboard after that. The logs suddenly started appearing an hour ago.

image

saroyaj commented 3 years ago

Reading https://github.com/Cyb3rWard0g/HELK/blob/master/docker/helk-logstash/pipeline/0006-kafka-zeek-input.conf , I was assuming if I used the topic as zeek, then the zeek logstash parser would kick in. No ?

priamai commented 3 years ago

That is correct @jsinix but it could be dropped later on at this stage if the fields are not right then it will be marked as parsefailure and will be written in this index parse-failures-*.

I am a little confused by your progress, could you confirm that is now written in this index: logs-network-zeek-* ? This would mean it is parsed correctly.

saroyaj commented 3 years ago

No it’s written in indexme-* and that’s what I am trying to fix now.

Jasmeet

On Wed, Dec 23, 2020 at 2:50 AM priamai notifications@github.com wrote:

That is correct @jsinix https://github.com/jsinix but it could be dropped later on at this stage https://github.com/Cyb3rWard0g/HELK/blob/master/docker/helk-logstash/pipeline/3101-zeek_corelight-all-filter.conf if the fields are not right then it will be marked as parsefailure and will be written in this index https://github.com/Cyb3rWard0g/HELK/blob/master/docker/helk-logstash/pipeline/9998-catch_all-output.conf .

I am a little confused by your progress, could you confirm that is now written in this index: logs-network-zeek-* ? This would mean it is parsed correctly.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Cyb3rWard0g/HELK/issues/532#issuecomment-749992565, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZHH33UDJMBGCKFCMIYQ2TSWGOOFANCNFSM4VDPN3HQ .

robomotic commented 3 years ago

Great could you post the documents as JSON files, I want to check what ETL tags they have attached so we can guess where in Logstash they have been processed.

saroyaj commented 3 years ago

logs-from-kafka.txt

Attaching logs from here: kafkauser@e71f29748d57:~/scripts$ /opt/helk/kafka/bin/kafka-console-consumer.sh --bootstrap-server helk-kafka-broker:9092 --topic zeek

priamai commented 3 years ago

Looked into them a few comments:

Now there is a clear problem: the filter checks for the presence of a field "message" which I don't see in your logs. I am not sure whether we should interpret as message the entire json object or if is explicit field (this is something confusing in Logstash).

For further debug can you use this configuration and run the log again?

I have seen various attempts in your previous post but not sure what was your last configuration.

Cheers.

priamai commented 3 years ago

There is also an interesting tutorial here which outputs directly into ES in JSON format. I wonder if you should try that first to make sure everything is working and then attempt again to shift to HELK Logstash?

saroyaj commented 3 years ago

this is exactly what I am doing. Saving the zeek logs as json and using filebeat to send them to kafka. Let me check the "message" field

priamai commented 3 years ago

Please also check the Logstash viewer as described here. It will help us understand where the events flow.