HELK setup using option 4 running on CentOS 8. Winlogbeat receiving perfectly. Packet beat is not showing feed on Kibana dashboard.

Steps

Created one topic on Kafka kafkauser@bb0d306f7696:~/bin$ ./kafka-topics.sh --list --bootstrap-server 192.168.0.100:9092 SYSMON_JOIN __consumer_offsets _confluent-ksql-wardog_command_topic filebeat packetbeats winevent winlogbeat zeek

Packetbeat yml

` packetbeat.interfaces.device: any

packetbeat.interfaces.internal_networks:

private

packetbeat.protocols:

type: icmp

enabled: true
type: amqp

Configure the ports where to listen for AMQP traffic. You can disable

the AMQP protocol by commenting out the list of ports.

ports: [5672]
type: cassandra

Cassandra port for traffic monitoring.

ports: [9042]
type: dhcpv4

Configure the DHCP for IPv4 ports.

ports: [67, 68]
type: dns

Configure the ports where to listen for DNS traffic. You can disable

the DNS protocol by commenting out the list of ports.

ports: [53]
type: http

Configure the ports where to listen for HTTP traffic. You can disable

the HTTP protocol by commenting out the list of ports.

ports: [80, 8080, 8000, 5000, 8002]
type: memcache

ports: [11211]
type: mysql

ports: [3306,3307]
type: pgsql

ports: [5432]
type: redis

ports: [6379]
type: thrift

ports: [9090]
type: mongodb

ports: [27017]
type: nfs

ports: [2049]
type: tls

ports:
- 443 # HTTPS
- 993 # IMAPS
- 995 # POP3S
- 5223 # XMPP over SSL
- 8443
- 8883 # Secure MQTT
- 9243 # Elasticsearch
type: sip

ports: [5060]

setup.template.settings: index.number_of_shards: 1 setup.dashboards.enabled: true

=================================== Kibana ===================================

Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.

This requires a Kibana endpoint configuration.

setup.kibana:

Kibana Host

Scheme and port can be left out and will be set to the default (http and 5601)

In case you specify and additional path, the scheme is required: http://localhost:5601/path

IPv6 addresses should always be defined as: https://[2001:db8::1]:5601

host: "localhost:5601"

Kibana Space ID

ID of the Kibana Space into which the dashboards should be loaded. By default,

the Default Space will be used.

space.id:

=============================== Elastic Cloud ================================

These settings simplify using Packetbeat with the Elastic Cloud (https://cloud.elastic.co/).

The cloud.id setting overwrites the `output.elasticsearch.hosts` and

`setup.kibana.host` options.

You can find the `cloud.id` in the Elastic Cloud web UI.

cloud.id:

The cloud.auth setting overwrites the `output.elasticsearch.username` and

`output.elasticsearch.password` settings. The format is `<user>:<pass>`.

cloud.auth:

================================== Outputs ===================================

Configure what output to use when sending the data collected by the beat.

----------------------------- Kafka output --------------------------------

output.kafka:

initial brokers for reading cluster metadata

Place your HELK IP(s) here (keep the port).

If you only have one Kafka instance (default for HELK) then remove the 2nd IP that has port 9093

hosts: 192.168.0.100:9092 topic: 'packetbeats' partition.round_robin: reachable_only: false

required_acks: 1 compression: gzip max_message_bytes: 1000000

================================= Processors =================================

processors:

Add forwarded to tags when processing data from a network tap or mirror.

if.contains.tags: forwarded then:
- drop_fields: fields: [host] else:
- add_host_metadata: ~
add_cloud_metadata: ~
add_docker_metadata: ~
detect_mime_type: field: http.request.body.content target: http.request.mime_type
detect_mime_type: field: http.response.body.content target: http.response.mime_type

================================== Logging ===================================

Sets log level. The default log level is info.

Available log levels are: error, warning, info, debug

logging.level: debug

At debug level, you can selectively enable logging only for some components.

To enable all selectors use ["*"]. Examples of other selectors are "beat",

"publisher", "service".

logging.selectors: ["*"]

============================= X-Pack Monitoring ==============================

Packetbeat can export internal metrics to a central Elasticsearch monitoring

cluster. This requires xpack monitoring to be enabled in Elasticsearch. The

reporting is disabled by default.

Set to true to enable the monitoring reporter.

monitoring.enabled: false

Sets the UUID of the Elasticsearch cluster under which monitoring data for this

Packetbeat instance will appear in the Stack Monitoring UI. If output.elasticsearch

is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.

monitoring.cluster_uuid:

Uncomment to send the metrics to Elasticsearch. Most settings from the

Elasticsearch output are accepted here as well.

Note that the settings should point to your Elasticsearch monitoring cluster.

Any setting that is not set is automatically inherited from the Elasticsearch

output configuration, so if you have the Elasticsearch output configured such

that it is pointing to your Elasticsearch monitoring cluster, you can simply

uncomment the following line.

monitoring.elasticsearch:

============================== Instrumentation ===============================

Instrumentation support for the packetbeat.

instrumentation:

# Set to true to enable instrumentation of packetbeat.
#enabled: false

# Environment in which packetbeat is running on (eg: staging, production, etc.)
#environment: ""

# APM Server hosts to report instrumentation results to.
#hosts:
#  - http://localhost:8200

# API Key for the APM Server(s).
# If api_key is set then secret_token will be ignored.
#api_key:

# Secret token for the APM Server(s).
#secret_token:

================================= Migration ==================================

This allows to enable 6.7 migration aliases

migration.6_to_7.enabled: true

Kafka Topic packetbeat log [shortened]

kafkauser@bb0d306f7696:~/bin$ ./kafka-console-consumer.sh --topic packetbeats --from-beginning --bootstrap-server 192.168.0.100:9092 {"@timestamp":"2021-03-23T12:52:40.000Z","@metadata":{"beat":"packetbeat","type":"_doc","version":"7.11.2"},"destination":{"mac":"02:42:ac:12:00:0a","ip":"172.18.0.10","port":36664,"packets":27,"bytes":3274},"flow":{"id":"EQQA////DP//////FP8BAAECQqwSAAoCQs/HimOsEgAKwKgAZDiPhCM","final":false},"type":"flow","network":{"community_id":"1:1uxnyUMZPo1X/6t0uv1T4NqRrHE=","bytes":7408,"packets":79,"type":"ipv4","transport":"tcp"},"source":{"bytes":4134,"mac":"02:42:cf:c7:8a:63","ip":"192.168.0.100","port":9092,"packets":52},"event":{"end":"2021-03-23T12:52:39.739Z","duration":12096012558,"dataset":"flow","kind":"event","category":["network_traffic","network"],"action":"network_flow","start":"2021-03-23T12:52:27.643Z"},"ecs":{"version":"1.7.0"},"host":{"ip":["192.168.0.100","fe80::e540:873e:e4dd:e156","192.168.122.1","172.18.0.1","fe80::42:cfff:fec7:8a63","172.17.0.1","fe80::5c73:b1ff:fe20:e71e","fe80::3470:9fff:fe66:f057","fe80::7043:74ff:fe17:9a0","fe80::8c3b:50ff:fe38:3db2","fe80::b421:44ff:fed1:f04","fe80::ac2f:d6ff:fee8:11fe","fe80::f80e:34ff:feb7:f557","fe80::f41d:22ff:fea1:1d1","fe80::c422:1bff:fe08:ac3c","fe80::b0f1:edff:fe00:8d1e","fe80::b8b5:47ff:fe49:d9b2"],"mac":["c0:3f:d5:5d:9a:61","52:54:00:e5:61:b1","52:54:00:e5:61:b1","02:42:cf:c7:8a:63","02:42:d2:fc:d5:90","5e:73:b1:20:e7:1e","36:70:9f:66:f0:57","72:43:74:17:09:a0","8e:3b:50:38:3d:b2","b6:21:44:d1:0f:04","ae:2f:d6:e8:11:fe","fa:0e:34:b7:f5:57","f6:1d:22:a1:01:d1","c6:22:1b:08:ac:3c","b2:f1:ed:00:8d:1e","ba:b5:47:49:d9:b2"],"hostname":"localhost.localdomain","name":"localhost.localdomain","architecture":"x86_64","os":{"version":"8","family":"redhat","name":"CentOS Linux","kernel":"4.18.0-240.15.1.el8_3.x86_64","platform":"centos"},"id":"8382750050f545c1a273b99c555a4fd3","containerized":false},"agent":{"hostname":"localhost.localdomain","ephemeral_id":"1624d062-2c20-4f83-a674-7785d80b7922","id":"a50dc480-9fda-4d20-a68d-f69506299d44","name":"localhost.localdomain","type":"packetbeat","version":"7.11.2"}} {"@timestamp":"2021-03-23T12:52:40.000Z","@metadata":{"beat":"packetbeat","type":"_doc","version":"7.11.2"},"destination":{"port":37882,"packets":1,"bytes":66,"mac":"02:42:ac:12:00:0a","ip":"172.18.0.10"},"type":"flow","source":{"ip":"192.168.0.100","port":9092,"bytes":66,"packets":1,"mac":"02:42:cf:c7:8a:63"},"ecs":{"version":"1.7.0"},"event":{"dataset":"flow","kind":"event","category":["network_traffic","network"],"action":"network_flow","start":"2021-03-23T12:52:37.219Z","end":"2021-03-23T12:52:37.219Z","duration":8557},"flow":{"id":"EQQA////DP//////FP8BAAECQqwSAAoCQs/HimOsEgAKwKgAZPqThCM","final":false},"network":{"type":"ipv4","transport":"tcp","community_id":"1:4eQnBELLcg4Bjk9tJ6mktNEeGGo=","bytes":132,"packets":2},"host":{"id":"8382750050f545c1a273b99c555a4fd3","containerized":false,"ip":["192.168.0.100","fe80::e540:873e:e4dd:e156","192.168.122.1","172.18.0.1","fe80::42:cfff:fec7:8a63","172.17.0.1","fe80::5c73:b1ff:fe20:e71e","fe80::3470:9fff:fe66:f057","fe80::7043:74ff:fe17:9a0","fe80::8c3b:50ff:fe38:3db2","fe80::b421:44ff:fed1:f04","fe80::ac2f:d6ff:fee8:11fe","fe80::f80e:34ff:feb7:f557","fe80::f41d:22ff:fea1:1d1","fe80::c422:1bff:fe08:ac3c","fe80::b0f1:edff:fe00:8d1e","fe80::b8b5:47ff:fe49:d9b2"],"name":"localhost.localdomain","mac":["c0:3f:d5:5d:9a:61","52:54:00:e5:61:b1","52:54:00:e5:61:b1","02:42:cf:c7:8a:63","02:42:d2:fc:d5:90","5e:73:b1:20:e7:1e","36:70:9f:66:f0:57","72:43:74:17:09:a0","8e:3b:50:38:3d:b2","b6:21:44:d1:0f:04","ae:2f:d6:e8:11:fe","fa:0e:34:b7:f5:57","f6:1d:22:a1:01:d1","c6:22:1b:08:ac:3c","b2:f1:ed:00:8d:1e","ba:b5:47:49:d9:b2"],"hostname":"localhost.localdomain","architecture":"x86_64","os":{"platform":"centos","version":"8","family":"redhat","name":"CentOS Linux","kernel":"4.18.0-240.15.1.el8_3.x86_64"}},"agent":{"id":"a50dc480-9fda-4d20-a68d-f69506299d44","name":"localhost.localdomain","type":"packetbeat","version":"7.11.2","hostname":"localhost.localdomain","ephemeral_id":"1624d062-2c20-4f83-a674-7785d80b7922"}} Dashboard not showing these any suggestions how what changes to be made on Kafka to show these on Kibana?

Cyb3rWard0g / HELK

packetbeat and HELK #549