search

PaloAltoNetworks / SafeNetworking

Read only mirror. To contribute or submit issues, please go to the website link --->
https://gitlab.com/panw-gse/as/SafeNetworking/
Apache License 2.0
12 stars 10 forks source link

Use pipeline functionality in logstash to separate listeners to individual pipelines #48

Closed punisherVX closed 5 years ago

punisherVX commented 5 years ago

Since we are starting to add more logging types, we need to break out the logstash filtering for each type individually. Since logstash 6.0.0, the ability to use separate pipelines allows for this. It still runs under the same logstash process, but now each is it's own thread and they can individually be started/stopped and each is configurable at the worker level so we can balance how much processor and memory each uses if needed.

punisherVX commented 5 years ago

Broke out each logging type to own configuration file. Fixed in commit d52e96be0ee2106898d1194e04fb36dfc2590cc7

threat.conf contains input/filter/output for THREAT and TRAFFIC listening on *port 5514** iot.conf* contains input/filter/output for external IP iot edge routers listening on port 5510 customer.conf contains input/filter/output for customer DB info listening on port 5515***

Added /etc/logstash/pipelines.yml file that describes each above for logstash to use in pipeline functionality

Modified setup.sh to backup/move/install config files and start pipelines properly

punisherVX commented 5 years ago

Added in v3.5