Closed ghost closed 2 years ago
@nwdean
pfelk-install.sh
contains and runs both the pfelk-template-installer.sh
and pfelk-dashboard-installer.sh
scripts.
[ ] The pfelk-dashboard-installer.sh
fails to import due to Elastic's default security with v8.0.0
[ ] Configuring/Guide for default user configuration (security)
@nwdean
More testing is needed but this should get you running:
sudo chown kibana /etc/kibana/kibana.yml
sudo ./usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana --url "https://10.0.0.30:9200"
sudo ./usr/share/kibana/bin/kibana-verificaiton-code
Reference:
@a3ilson
Thank you for such a quick reply, much appreciated!
Just clarifying I am running this AFTER I run the install script right?
(just blew away the OS and am redoing it all)
That is correct. The pfelk-install.sh
will install everything to include the two subsequent scripts (pfelk-template-installer.sh
& pfelk-dashboard-installer.sh
). However, the pfelk-dashboard-installer.sh
will fail (working on a fix) as with v8 (Elastic), security is now implemented by default (good thing).
After everything is installed (you would have been fine without reinstalling). You'll need to run:
sudo chown kibana /etc/kibana/kibana.yml
sudo ./usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana --url "https://10.0.0.30:9200"
sudo ./usr/share/kibana/bin/kibana-verificaiton-code
Manually:
sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana --url "https://10.0.0.30:9200"
sudo /usr/share/kibana/bin/kibana-verification-code
[ ] Reset the default elastic
user password
sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
Reference: https://www.elastic.co/guide/en/elasticsearch/reference/8.0/reset-password.html
{
"password" : "$up3r$@f#",
"roles" : [ "kibana_admin" ]
}
Login with new user credentials
Reference: https://www.elastic.co/guide/en/kibana/current/using-kibana-with-security.html
Unfortunately, the save objects and index patterns will need to be installed manually (for now until the scripts are amended to accommodate for the security features)
Templates Manual Method: https://github.com/pfelk/pfelk/blob/main/install/configuration.md#a-manual-method
Saved Objects Manual Method: https://github.com/pfelk/pfelk/blob/main/install/configuration.md#a-manual-method-1
#!/bin/bash
#
# Version | 22.02
# Email | support@pfelk.com
# Website | https://pfelk.com
#
###################################################################################################################################################################################################
# #
# Color Codes #
# #
###################################################################################################################################################################################################
#
RESET='\033[0m'
WHITE_R='\033[39m'
RED='\033[1;31m' # Light Red.
GREEN='\033[1;32m' # Light Green.
#
header() {
clear
clear
echo -e "${GREEN}#####################################################################################################${RESET}\\n"
}
header_red() { clear clear echo -e "${RED}#####################################################################################################${RESET}\n" } #
if [[ "$EUID" -ne 0 ]]; then header_red echo -e "${WHITE_R}#${RESET} The script need to be run as root...\n\n" echo -e "${WHITE_R}#${RESET} For Ubuntu based systems run the command below to login as root" echo -e "${GREEN}#${RESET} sudo -i\n" echo -e "${WHITE_R}#${RESET} For Debian based systems run the command below to login as root" echo -e "${GREEN}#${RESET} su\n\n" exit 1 fi # ###################################################################################################################################################################################################
################################################################################################################################################################################################### wget -q https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/dashboard/22.01-captive.ndjson -P /tmp/pfELK/ wget -q https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/dashboard/22.01-dhcp.ndjson -P /tmp/pfELK/ wget -q https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/dashboard/22.01-firewall.ndjson -P /tmp/pfELK/ wget -q https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/dashboard/22.01-haproxy.ndjson -P /tmp/pfELK/ wget -q https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/dashboard/22.01-nginx.ndjson -P /tmp/pfELK/ wget -q https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/dashboard/22.01-snort.ndjson -P /tmp/pfELK/ wget -q https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/dashboard/22.01-squid.ndjson -P /tmp/pfELK/ wget -q https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/dashboard/22.01-suricata.ndjson -P /tmp/pfELK/ wget -q https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/dashboard/22.01-unbound.ndjson -P /tmp/pfELK/ ###################################################################################################################################################################################################
################################################################################################################################################################################################### curl -X POST localhost:5601/api/saved_objects/_import -H "kbn-xsrf: true" --form file=@/tmp/pfELK/22.01-captive.ndjson -H 'kbn-xsrf: true' -u elastic:passwordgoeshere curl -X POST localhost:5601/api/saved_objects/_import -H "kbn-xsrf: true" --form file=@/tmp/pfELK/22.01-dhcp.ndjson -H 'kbn-xsrf: true' -u elastic:passwordgoeshere curl -X POST localhost:5601/api/saved_objects/_import -H "kbn-xsrf: true" --form file=@/tmp/pfELK/22.01-firewall.ndjson -H 'kbn-xsrf: true' -u elastic:passwordgoeshere curl -X POST localhost:5601/api/saved_objects/_import -H "kbn-xsrf: true" --form file=@/tmp/pfELK/22.01-haproxy.ndjson -H 'kbn-xsrf: true' -u elastic:passwordgoeshere curl -X POST localhost:5601/api/saved_objects/_import -H "kbn-xsrf: true" --form file=@/tmp/pfELK/22.01-nginx.ndjson -H 'kbn-xsrf: true' -u elastic:passwordgoeshere curl -X POST localhost:5601/api/saved_objects/_import -H "kbn-xsrf: true" --form file=@/tmp/pfELK/22.01-snort.ndjson -H 'kbn-xsrf: true' -u elastic:passwordgoeshere curl -X POST localhost:5601/api/saved_objects/_import -H "kbn-xsrf: true" --form file=@/tmp/pfELK/22.01-squid.ndjson -H 'kbn-xsrf: true' -u elastic:passwordgoeshere curl -X POST localhost:5601/api/saved_objects/_import -H "kbn-xsrf: true" --form file=@/tmp/pfELK/22.01-suricata.ndjson -H 'kbn-xsrf: true' -u elastic:passwordgoeshere curl -X POST localhost:5601/api/saved_objects/_import -H "kbn-xsrf: true" --form file=@/tmp/pfELK/22.01-unbound.ndjson -H 'kbn-xsrf: true' -u elastic:passwordgoeshere
@a3ilson
heads up more privileges are required to run those templates than are found in the kibana admin role you said to assign to the pfelk user. I added the super user role to pfelk but I am not sure that is a great long-term solution, got it working though
{ "error" : { "root_cause" : [ { "type" : "security_exception", "reason" : "action [cluster:admin/component_template/put] is unauthorized for user [pfelk] with roles [kibana_admin], this action is granted by the cluster privileges [manage_index_templates,manage,all]" } ], "type" : "security_exception", "reason" : "action [cluster:admin/component_template/put] is unauthorized for user [pfelk] with roles [kibana_admin], this action is granted by the cluster privileges [manage_index_templates,manage,all]" }, "status" : 403 }
Thanks! I should have specified that those actions should have been completed as the elastic user.
@a3ilson ah ok cool, so elastic is for admin stuff and pfelk is analyst/user. Got it!
@nwdean - in addition to the above, you'll need to configure logstash for security.
Thanks again for highlighting the issues with this repo and the v8.0.0 default security. If you're looking to get it up and running a bit quicker (easier). I would recommend utilizing the prior script here which will run the Elastic stack in version 7.16.X.
We're working to update the repo to accommodate the new default security and apologize for the inconvenience.
Reference: https://www.elastic.co/guide/en/logstash/current/ls-security.html
@a3ilson
I tried installing from that installer script and it all seems to exit clean but when I go to elastic I get the following error for all of my dashboards:
Also if I go to stack management it shows no indexes or data streams. Templates are there though. I verified all of my settings in pfsense are good and it is sending the syslog data, just seems that either logstash isn't listening OR there are no indexes to put the incoming data. Thoughts?
@nwdean the issue is configuring logstash for security. Appears that you have everything configured except logstash. Again apologize as I haven’t had time to update the repo with the new security settings. However, logstash is unable to communicate with elastic until it’s configured (security) and that is why you are not seeing any data.
@a3ilson your last message said to use the older install script that used the elastic version before the security was introduced. So that message above was me using installer script 22.01 and I got that error.
Have you guys tried testing this out on blank 18.04 installs? I am following the guide exactly and stuff isn't working. Not trying to be a jerk, just trying to get his installed and help you guys work out any bugs
@a3ilson your last message said to use the older install script that used the elastic version before the security was introduced. So that message above was me using installer script 22.01 and I got that error.
Have you guys tried testing this out on blank 18.04 installs? I am following the guide exactly and stuff isn't working. Not trying to be a jerk, just trying to get his installed and help you guys work out any bugs
Gotcha - give this a try and if it doesn't work send me the logs.
edit /etc/logstash/pipelines.yml
# pipelines.yml
################################################################################
# Version: 22.01 #
# Required: Yes #
# Description: This is a required file for a pfelk installation #
# This file is where you define your pipelines. You can define multiple. #
# For more information on multiple pipelines, see the documentation: #
# https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html #
# #
################################################################################
#
- pipeline.id: pfelk
path.config: "/etc/pfelk/conf.d/*.conf"
Restart logstash (systemctl restart logstash
)
if that doesn't do the trick, send me the logs from running /etc/pfelk/scripts/error-data.sh
@a3ilson
That was pretty much the pipelines.yml that was on there, I just commented out the compatibility line. But still didn't work, no indexes or data streams or anything. This looked like it had issues running:
pfelk@pfelk:/etc/pfelk/scripts$ sudo ./error-data.sh pfelk: Generating pfelk Error Data mkdir: cannot create directory ‘/etc/pfelk/logs’: File exists rm: cannot remove '/etc/pfelk/logs/error.pfelk.log': No such file or directory find: ‘/var/lib/GeoIP/’: No such file or directory cat: '/etc/logstash/conf.d/*.conf': No such file or directory Error Data Collected Successfully Attach the contents of /etc/pfelk/logs/error.pfelk.log as a error.pfelk.log file to attache and include with your issue in github pfelk@pfelk:/etc/pfelk/scripts$
It appears suricata has a conflict with one or field. Try disabling suricata logs. That should work and then we can look into suricate.
@a3ilson trying that now but the logstash service was stalled on shutdown. You guys should edit the service script to have it start with "--pipeline.unsafe_shutdown" so that if it stalls it can still be force closed.
got it from here: https://stackoverflow.com/questions/49423897/logstash-shutdown-stalling-when-starting-from-bash-script
Had to reboot the box, will update when it comes back up
@a3ilson
logstash started up and loaded the pipeline like it should have. Getting some errors in the dashboards regarding the interface attribute though. Not sure what that is caused by but some of the windows aren't populating with data. Also any idea what the issue is with suricata? That was one of the main reasons I stood this up
@nwdean
Suricata failed as a result of logstash attempting to parse suricata.eve.http.content_range
as object but found a concrete value
This repo was also incorporated as an experimental feature within Integrations>>pfSense Logs but not as robust as this repo.
@nwdean - Updated the script to disable the newly implemented default security. This will allow for the installation with no issues (tested) until we incorporate an easier solution to for the defaulted security.
Kudos to @revere521 for the suggestion!
it might appear like an off-topic comment or even a slight... but have you considered setting up some sort of CI/CD so issues like these become obsolete, since you'd see beforehand (read: before the release), that various methods of installation run into issues that'd need manual intervention?
I, for one, will roll back to a snapshot of pfelk 7.16, which was a whole hassle in and of itself to set up, with loads of manual intervention until it'd behave as expected.
This isn't a critique of the pfelk stack either... that's actually quite fine and useful even. But the deployment is ... atrocious.
just my 2 cents here.
@TildeSlashC0re I agree and we looked into this a few years ago but CI/CD was incapable with the current structure and variables required for install. From that, the docker and ansible repo's were built out.
We wanted to merge (collapse) the docker repo and will look into that again as it would provide a solution for CI/CD
Describe the bug The install steps do not work as written, it errors out at the dashboard script saying one of the files does not exist. They are also incomplete as there are no instructions for an elastic token or the default username and password
To Reproduce Steps to reproduce the behavior:
These are the commands I used, in order, to install from the readme.
wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/scripts/pfelk-installer.sh sudo chmod +x pfelk-installer.sh sudo ./pfelk-installer.sh wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/scripts/pfelk-template-installer.sh sudo chmod +x pfelk-template-installer.sh sudo ./pfelk-template-installer.sh wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/scripts/pfelk-dashboard-installer.sh sudo chmod +x pfelk-dashboard-installer.sh sudo ./pfelk-dashboard-installer.sh
Firewall System (please complete the following information):
Operating System (please complete the following information):
OS (
printf "$(uname -srm)\n$(cat /etc/os-release)\n"
):Installation method (manual, ansible-playbook, docker, script):
Elasticsearch, Logstash, Kibana (please complete the following information):
dpkg -l [elasticsearch]|[logstash]|[kibana]
)no idea, it is a scripted install
Elasticsearch, Logstash, Kibana logs:
tail -f /var/log/elasticsearch/[your-elk-cluster-name].log
)tail -f /var/log/logstash/logstash-plain.log
)journalctl -u kibana.service
)Additional context Add any other context about the problem here.
**Attach the pfELK Error Log (error.pfelk), for Better Assistance***