pfelk / pfelk

pfSense/OPNsense + Elastic Stack
https://pfelk.github.io/pfelk/
Other
1.07k stars 192 forks source link

Elastic v8.0.0+ Security Fails To Install Templates/Saved Objects <-- must be done manually...for now #415

Closed ghost closed 2 years ago

ghost commented 2 years ago

Describe the bug The install steps do not work as written, it errors out at the dashboard script saying one of the files does not exist. They are also incomplete as there are no instructions for an elastic token or the default username and password

To Reproduce Steps to reproduce the behavior:

These are the commands I used, in order, to install from the readme.

wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/scripts/pfelk-installer.sh sudo chmod +x pfelk-installer.sh sudo ./pfelk-installer.sh wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/scripts/pfelk-template-installer.sh sudo chmod +x pfelk-template-installer.sh sudo ./pfelk-template-installer.sh wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/scripts/pfelk-dashboard-installer.sh sudo chmod +x pfelk-dashboard-installer.sh sudo ./pfelk-dashboard-installer.sh

Firewall System (please complete the following information):

Operating System (please complete the following information):

Elasticsearch, Logstash, Kibana (please complete the following information):

Additional context Add any other context about the problem here.

**Attach the pfELK Error Log (error.pfelk), for Better Assistance***

a3ilson commented 2 years ago

@nwdean

pfelk-install.sh contains and runs both the pfelk-template-installer.sh and pfelk-dashboard-installer.sh scripts.

a3ilson commented 2 years ago

@nwdean

More testing is needed but this should get you running:

Reference:

ghost commented 2 years ago

@a3ilson

Thank you for such a quick reply, much appreciated!

Just clarifying I am running this AFTER I run the install script right?

(just blew away the OS and am redoing it all)

a3ilson commented 2 years ago

That is correct. The pfelk-install.sh will install everything to include the two subsequent scripts (pfelk-template-installer.sh & pfelk-dashboard-installer.sh). However, the pfelk-dashboard-installer.sh will fail (working on a fix) as with v8 (Elastic), security is now implemented by default (good thing).

After everything is installed (you would have been fine without reinstalling). You'll need to run:

a3ilson commented 2 years ago

Screen Shot 2022-02-26 at 12 38 13 AM

Screen Shot 2022-02-26 at 12 40 29 AM

Manually:

a3ilson commented 2 years ago

Reference: https://www.elastic.co/guide/en/elasticsearch/reference/8.0/reset-password.html

a3ilson commented 2 years ago

Reference: https://www.elastic.co/guide/en/kibana/current/using-kibana-with-security.html

a3ilson commented 2 years ago

Unfortunately, the save objects and index patterns will need to be installed manually (for now until the scripts are amended to accommodate for the security features)

Templates Manual Method: https://github.com/pfelk/pfelk/blob/main/install/configuration.md#a-manual-method

Saved Objects Manual Method: https://github.com/pfelk/pfelk/blob/main/install/configuration.md#a-manual-method-1

header_red() { clear clear echo -e "${RED}#####################################################################################################${RESET}\n" } #

Check for root (sudo)

if [[ "$EUID" -ne 0 ]]; then header_red echo -e "${WHITE_R}#${RESET} The script need to be run as root...\n\n" echo -e "${WHITE_R}#${RESET} For Ubuntu based systems run the command below to login as root" echo -e "${GREEN}#${RESET} sudo -i\n" echo -e "${WHITE_R}#${RESET} For Debian based systems run the command below to login as root" echo -e "${GREEN}#${RESET} su\n\n" exit 1 fi # ###################################################################################################################################################################################################

pfELK - Download Saved Objects

################################################################################################################################################################################################### wget -q https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/dashboard/22.01-captive.ndjson -P /tmp/pfELK/ wget -q https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/dashboard/22.01-dhcp.ndjson -P /tmp/pfELK/ wget -q https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/dashboard/22.01-firewall.ndjson -P /tmp/pfELK/ wget -q https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/dashboard/22.01-haproxy.ndjson -P /tmp/pfELK/ wget -q https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/dashboard/22.01-nginx.ndjson -P /tmp/pfELK/ wget -q https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/dashboard/22.01-snort.ndjson -P /tmp/pfELK/ wget -q https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/dashboard/22.01-squid.ndjson -P /tmp/pfELK/ wget -q https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/dashboard/22.01-suricata.ndjson -P /tmp/pfELK/ wget -q https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/dashboard/22.01-unbound.ndjson -P /tmp/pfELK/ ###################################################################################################################################################################################################

pfELK - Installing Saved Objects

################################################################################################################################################################################################### curl -X POST localhost:5601/api/saved_objects/_import -H "kbn-xsrf: true" --form file=@/tmp/pfELK/22.01-captive.ndjson -H 'kbn-xsrf: true' -u elastic:passwordgoeshere curl -X POST localhost:5601/api/saved_objects/_import -H "kbn-xsrf: true" --form file=@/tmp/pfELK/22.01-dhcp.ndjson -H 'kbn-xsrf: true' -u elastic:passwordgoeshere curl -X POST localhost:5601/api/saved_objects/_import -H "kbn-xsrf: true" --form file=@/tmp/pfELK/22.01-firewall.ndjson -H 'kbn-xsrf: true' -u elastic:passwordgoeshere curl -X POST localhost:5601/api/saved_objects/_import -H "kbn-xsrf: true" --form file=@/tmp/pfELK/22.01-haproxy.ndjson -H 'kbn-xsrf: true' -u elastic:passwordgoeshere curl -X POST localhost:5601/api/saved_objects/_import -H "kbn-xsrf: true" --form file=@/tmp/pfELK/22.01-nginx.ndjson -H 'kbn-xsrf: true' -u elastic:passwordgoeshere curl -X POST localhost:5601/api/saved_objects/_import -H "kbn-xsrf: true" --form file=@/tmp/pfELK/22.01-snort.ndjson -H 'kbn-xsrf: true' -u elastic:passwordgoeshere curl -X POST localhost:5601/api/saved_objects/_import -H "kbn-xsrf: true" --form file=@/tmp/pfELK/22.01-squid.ndjson -H 'kbn-xsrf: true' -u elastic:passwordgoeshere curl -X POST localhost:5601/api/saved_objects/_import -H "kbn-xsrf: true" --form file=@/tmp/pfELK/22.01-suricata.ndjson -H 'kbn-xsrf: true' -u elastic:passwordgoeshere curl -X POST localhost:5601/api/saved_objects/_import -H "kbn-xsrf: true" --form file=@/tmp/pfELK/22.01-unbound.ndjson -H 'kbn-xsrf: true' -u elastic:passwordgoeshere

ghost commented 2 years ago

@a3ilson

heads up more privileges are required to run those templates than are found in the kibana admin role you said to assign to the pfelk user. I added the super user role to pfelk but I am not sure that is a great long-term solution, got it working though

{ "error" : { "root_cause" : [ { "type" : "security_exception", "reason" : "action [cluster:admin/component_template/put] is unauthorized for user [pfelk] with roles [kibana_admin], this action is granted by the cluster privileges [manage_index_templates,manage,all]" } ], "type" : "security_exception", "reason" : "action [cluster:admin/component_template/put] is unauthorized for user [pfelk] with roles [kibana_admin], this action is granted by the cluster privileges [manage_index_templates,manage,all]" }, "status" : 403 }

a3ilson commented 2 years ago

Thanks! I should have specified that those actions should have been completed as the elastic user.

ghost commented 2 years ago

@a3ilson ah ok cool, so elastic is for admin stuff and pfelk is analyst/user. Got it!

a3ilson commented 2 years ago

@nwdean - in addition to the above, you'll need to configure logstash for security.

Thanks again for highlighting the issues with this repo and the v8.0.0 default security. If you're looking to get it up and running a bit quicker (easier). I would recommend utilizing the prior script here which will run the Elastic stack in version 7.16.X.

We're working to update the repo to accommodate the new default security and apologize for the inconvenience.

Reference: https://www.elastic.co/guide/en/logstash/current/ls-security.html

ghost commented 2 years ago

@a3ilson

I tried installing from that installer script and it all seems to exit clean but when I go to elastic I get the following error for all of my dashboards: image

Also if I go to stack management it shows no indexes or data streams. Templates are there though. I verified all of my settings in pfsense are good and it is sending the syslog data, just seems that either logstash isn't listening OR there are no indexes to put the incoming data. Thoughts?

a3ilson commented 2 years ago

@nwdean the issue is configuring logstash for security. Appears that you have everything configured except logstash. Again apologize as I haven’t had time to update the repo with the new security settings. However, logstash is unable to communicate with elastic until it’s configured (security) and that is why you are not seeing any data.

ghost commented 2 years ago

@a3ilson your last message said to use the older install script that used the elastic version before the security was introduced. So that message above was me using installer script 22.01 and I got that error.

Have you guys tried testing this out on blank 18.04 installs? I am following the guide exactly and stuff isn't working. Not trying to be a jerk, just trying to get his installed and help you guys work out any bugs

a3ilson commented 2 years ago

@a3ilson your last message said to use the older install script that used the elastic version before the security was introduced. So that message above was me using installer script 22.01 and I got that error.

Have you guys tried testing this out on blank 18.04 installs? I am following the guide exactly and stuff isn't working. Not trying to be a jerk, just trying to get his installed and help you guys work out any bugs

Gotcha - give this a try and if it doesn't work send me the logs.

edit /etc/logstash/pipelines.yml

# pipelines.yml
################################################################################
# Version: 22.01                                                               #
# Required: Yes                                                                #
# Description: This is a required file for a pfelk installation                #
# This file is where you define your pipelines. You can define multiple.       #
# For more information on multiple pipelines, see the documentation:           #
#   https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html   #
#                                                                              #
################################################################################
#
- pipeline.id: pfelk
  path.config: "/etc/pfelk/conf.d/*.conf"

Restart logstash (systemctl restart logstash)

if that doesn't do the trick, send me the logs from running /etc/pfelk/scripts/error-data.sh

ghost commented 2 years ago

@a3ilson

That was pretty much the pipelines.yml that was on there, I just commented out the compatibility line. But still didn't work, no indexes or data streams or anything. This looked like it had issues running:

pfelk@pfelk:/etc/pfelk/scripts$ sudo ./error-data.sh pfelk: Generating pfelk Error Data mkdir: cannot create directory ‘/etc/pfelk/logs’: File exists rm: cannot remove '/etc/pfelk/logs/error.pfelk.log': No such file or directory find: ‘/var/lib/GeoIP/’: No such file or directory cat: '/etc/logstash/conf.d/*.conf': No such file or directory Error Data Collected Successfully Attach the contents of /etc/pfelk/logs/error.pfelk.log as a error.pfelk.log file to attache and include with your issue in github pfelk@pfelk:/etc/pfelk/scripts$

a3ilson commented 2 years ago

It appears suricata has a conflict with one or field. Try disabling suricata logs. That should work and then we can look into suricate.

ghost commented 2 years ago

@a3ilson trying that now but the logstash service was stalled on shutdown. You guys should edit the service script to have it start with "--pipeline.unsafe_shutdown" so that if it stalls it can still be force closed.

got it from here: https://stackoverflow.com/questions/49423897/logstash-shutdown-stalling-when-starting-from-bash-script

Had to reboot the box, will update when it comes back up

ghost commented 2 years ago

@a3ilson

logstash started up and loaded the pipeline like it should have. Getting some errors in the dashboards regarding the interface attribute though. Not sure what that is caused by but some of the windows aren't populating with data. Also any idea what the issue is with suricata? That was one of the main reasons I stood this up

a3ilson commented 2 years ago

@nwdean

Suricata failed as a result of logstash attempting to parse suricata.eve.http.content_range as object but found a concrete value

This repo was also incorporated as an experimental feature within Integrations>>pfSense Logs but not as robust as this repo. Screen Shot 2022-03-01 at 2 48 31 PM

a3ilson commented 2 years ago

@nwdean - Updated the script to disable the newly implemented default security. This will allow for the installation with no issues (tested) until we incorporate an easier solution to for the defaulted security.

Kudos to @revere521 for the suggestion!

TildeSlashC0re commented 2 years ago

it might appear like an off-topic comment or even a slight... but have you considered setting up some sort of CI/CD so issues like these become obsolete, since you'd see beforehand (read: before the release), that various methods of installation run into issues that'd need manual intervention?

I, for one, will roll back to a snapshot of pfelk 7.16, which was a whole hassle in and of itself to set up, with loads of manual intervention until it'd behave as expected.

This isn't a critique of the pfelk stack either... that's actually quite fine and useful even. But the deployment is ... atrocious.

just my 2 cents here.

a3ilson commented 2 years ago

@TildeSlashC0re I agree and we looked into this a few years ago but CI/CD was incapable with the current structure and variables required for install. From that, the docker and ansible repo's were built out.

We wanted to merge (collapse) the docker repo and will look into that again as it would provide a solution for CI/CD