search

pfelk / ansible

Ansible playbook automation for pfelk
Apache License 2.0
24 stars 4 forks source link

no syslogs from pfSense; Couldn't find any Elasticsearch data #8

Closed vap0rtranz closed 4 years ago

vap0rtranz commented 4 years ago

Describe the bug Sounds similar to issue 3ilson/pfelk#118

To Reproduce Steps to reproduce the behavior:

  1. install Ubuntu 18
  2. install pfElk via ansible
  3. access Kibana frontend
  4. setup pfSense to forward logs per Git doc
  5. Kibana frontend complains "Couldn't find any Elasticsearch data"

Firewall System (please complete the following information):

Operating System (please complete the following information):

NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Installation method (manual, ansible-playbook, docker, script): ansible-playbook

Elasticsearch, Logstash, Kibana (please complete the following information):

Additional context The logstash port is not up and listening: netstat --listen | grep 5140

**Attach the pfELK Error Log (error.pfelk), for Better Assistance*** "error.pfelk" doesn't exist on filesystem

sudo find / -type f -name error.pfelk
justin@pfelk1:/$
revere521 commented 4 years ago

This is the error - logstash looks like its not starting becasue its looking for the geoip template: 

      # This setting must be a path
      # File does not exist or cannot be opened /etc/logstash/conf.d/template/pf-geoip-template.json
      template => "/etc/logstash/conf.d/template/pf-geoip-template.json"

on you box navigate to /etc/logstash/conf.d/template/ and run sudo wget https://raw.githubusercontent.com/3ilson/pfelk/master/etc/logstash/conf.d/templates/pf-geoip-template.json

I cant remember if the ansible playbook configures everything, but you may need to download and setup GeoIP https://github.com/3ilson/pfelk/blob/master/install/ubuntu.md#7-install-maxmind

then do the remainder of the config https://github.com/3ilson/pfelk/blob/master/install/ubuntu.md#configuration

vap0rtranz commented 4 years ago

Looks like a typo in the name of the directory.

Ansible had created:

root@pfelk1:/etc/logstash/conf.d/template# ls -lrt
total 8
-rw-r--r-- 1 root root 2035 Jun  4 23:34 pf-geoip-template.json

The manual doc says to create a "templates" directory so I manually created it:

mkdir /etc/logstash/conf.d/templates

And put the pf-geoip-template.json file in it.

Also, from the manual doc: "download and setup GeoIP"

Line #18 in /etc/GeoIP.conf was missing, though the other two lines were already done, so I added #18.

DatabaseDirectory /usr/share/GeoIP/
vap0rtranz commented 4 years ago

The connection to port 5140 from pfSense still fails.

I gathered the pfELK logs by running error-data.sh. Logstash had the same error.

So I manually re-downloaded the conf.d files for logstash and found Ansible had downloaded files with differences. For example:

diff 01-inputs.conf 01-inputs.conf.1 
7d6
< 
12c11
<   if [host] =~ /192\.168\.0\.1/ {
---
>   if [host] == "192.168.0.1" {
15c14
<       add_field => [ "[observer][type]", "router" ]
---
>       add_field => [ "[observer][type]", "firewall" ]
21c20
< #if [host] =~ /172\.2\.22\.1/ {
---
> #if [host] == "10.0.0.1" {
31c30
<       #match => { "message" => "<(?<[event][id]>.*)>%{SYSLOGTIMESTAMP:[event][created]} %{SYSLOGHOST:[observer][name]} %{DATA:labels}(?:\[%{POSINT:pf_pid}\])?: %{GREEDYDATA:pf_message}" }
---
>       #OPN# match => { "message" => "<(?<[event][id]>.*)>%{SYSLOGTIMESTAMP:[event][created]} %{SYSLOGHOST:[observer][name]} %{DATA:labels}(?:\[%{POSINT:pf_pid}\])?: %{GREEDYDATA:pf_message}" }
34c33
<        match => { "message" => "<(?<[event][id]>.*)>%{SYSLOGTIMESTAMP:[event][created]} %{DATA:labels}(?:\[%{POSINT:[event][id]}\])?: %{GREEDYDATA:pf_message}" }
---
>       #pf# match => { "message" => "<(?<[event][id]>.*)>%{SYSLOGTIMESTAMP:[event][created]} %{DATA:labels}(?:\[%{POSINT:[event][id]}\])?: %{GREEDYDATA:pf_message}" }

It looks like logstash has started now, but it is hard to tell. The latest pfELK error log ends the logstash section with:

[2020-06-05T22:11:47,605][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Cou
ld not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2020-06-05T22:11:48,060][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-06-05T22:11:52,824][INFO ][logstash.runner          ] Logstash shut down.
vap0rtranz commented 4 years ago

I've fixed this issue after one more change.

The final change was in 01-inputs.conf. My conf now has these lines:

if "pf" in [tags] {
    grok {
      # OPNsense - Enable/Disable the line below based on firewall platform
      #OPN# match => { "message" => "<(?<[event][id]>.*)>%{SYSLOGTIMESTAMP:[event][created]} %{SYSLOGHOST:[observer][name]} %{DATA:labels}(?:\[%{POSINT:pf_pid}\])?: %{GREEDYDATA:pf_message}" }
      ########################################################################################################################################
      # pfSense - Enable/Disable the line below based on firewall platform
      match => { "message" => "<(?<[event][id]>.*)>%{SYSLOGTIMESTAMP:[event][created]} %{DATA:labels}(?:\[%{POSINT:[event][id]}\])?: %{GREEDYDATA:pf_message}" }
    }

To verify, the listening port for syslog is now open:

sudo netstat --listen | grep 5140
udp        0      0 0.0.0.0:5140            0.0.0.0:*

Hopefully this helps others.

fktkrt commented 4 years ago

Right, sorry for the inconvenience, the wrongly named folder is resolved now, thank you for checking it! About the other thing: yes, the input file is set up for OPNSense by default. I'll update the README with a note to that. I might be good, if this could be set with custom tags/templates when using the ansible-playbook.

EDIT: It looks like the README was up-to-date actually.

vap0rtranz commented 4 years ago

No worries. Happy to be pointed in the right direction. ty!

fktkrt commented 4 years ago

Closing this, since it's resolved. Feel free to reach out if you have any further issues in one of the repositories.