Grafana 7.4.3
Influxdb 1.8.3
pfSense 2.6 - No Stats from pfBlocker - https://github.com/VictorRobellini/pfSense-Dashboard/issues/58
This is just a summary, for more details look at the commits.
Due to the update in the Gateway plugin (move from py to php), you may need to drop your gateways measurement.
In the recent commits I updated the telegraf config to use the Tails Input Plugin in place of the Logparser Input Plugin since it's been deprecated.
I renamed many of the columns to reflect what's being logged by pfBlockerNG-devel and fixed some parsing bugs that cause lines to be skipped due to inconsistent log formatting. As a result, the measurements ip_block_log and dnsbl_log have been replaced with tail_ip_block_log and tail_dnsbl_log respectively.
I dropped the old measurements: ip_block_log, dnsbl_log
If you cannot live without this data, you could use the panels from this commit and not update the config. Read my note about the Logparser Input Plugin above!
If you want to load the complete logs files, you could probably change the telegraf config to:
from_beginning = false
to
from_beginning = true
I'm sure you can even rename the measurements, columns and update the tags, but that's beyond my influx capabilities.
Kubernetes deployed locally with these instructions
Stripped yaml templates used to deploy my homelab (including Influx and Grafana) are here
grafana-pfSense:
image: "grafana/grafana:7.4.3"
container_name: grafana
hostname: grafana
mem_limit: 4gb
ports:
- "3000:3000"
environment:
TZ: "America/New_York"
GF_INSTALL_PLUGINS: "grafana-clock-panel,grafana-simple-json-datasource,grafana-piechart-panel,grafana-worldmap-panel"
GF_PATHS_DATA: "/var/lib/grafana"
GF_DEFAULT_INSTANCE_NAME: "home"
GF_ANALYTICS_REPORTING_ENABLED: "false"
GF_SERVER_ENABLE_GZIP: "true"
GF_SERVER_DOMAIN: "home.mydomain"
volumes:
- '/share/ContainerData/grafana:/var/lib/grafana'
logging:
driver: "json-file"
options:
max-size: "100M"
network_mode: bridge
influxdb-pfsense:
image: "influxdb:1.8.3-alpine"
container_name: influxdb
hostname: influxdb
mem_limit: 10gb
ports:
- "2003:2003"
- "8086:8086"
environment:
TZ: "America/New_York"
INFLUXDB_DATA_QUERY_LOG_ENABLED: "false"
INFLUXDB_REPORTING_DISABLED: "true"
INFLUXDB_HTTP_AUTH_ENABLED: "true"
INFLUXDB_ADMIN_USER: "admin"
INFLUXDB_ADMIN_PASSWORD: "adminpassword"
INFLUXDB_USER: "pfsense"
INFLUXDB_USER_PASSWORD: "pfsenseuserpassword"
INFLUXDB_DB: "pfsense"
volumes:
- '/share/ContainerData/influxdb:/var/lib/influxdb'
logging:
driver: "json-file"
options:
max-size: "100M"
network_mode: bridge
Make sure you are using pfBlockerNG-devel
The Config for the dashboard relies on the variables defined within the dashboard in Grafana. When importing the dashboard, make sure to select your datasource.
Dashboard Settings -> Variables
WAN - $WAN is a static variable defined so that a separate dashboard panel can be created for WAN interfaces stats. Use a comma-separated list for multiple WAN interfaces.
LAN_Interfaces - $LAN_Interfaces uses a regex to remove any interfaces you don't want to be grouped as LAN. The filtering happens in the "Regex" field. I use a negative lookahead regex to match the interfaces I want excluded. It should be pretty easy to understand what you need to do here. I have excluded igb0 (WAN) and igb1,igb2,igb3 (only used to host vlans).
After writing this up, I realize I need to change this variable name, it's just not going to happen right now.
In the /config directory you will find all of the additional telegraf config. In pfSense, under Services -> Telegraf, at the bottom of the page with the teeny tiny text box is where you paste in the included config.
I also included the config for Unbound DNS and it's commented out. I'm not currently using it, but it's fully functional, just uncomment if you want to use it.
Plugins get copied to your pfSense system
I put all my plugins in /usr/local/bin and set them to 555
I also included a wrapper script for Unbound DNS. I'm not currently using it, but it's fully functional.
# cat -e /usr/local/bin/telegraf_pfinterface.php
If you get no good output from running the plugin directly, try the following command before moving to the below step.
# telegraf --test --config /usr/local/etc/telegraf.conf
To troubleshoot plugins further, add the following lines to the agent block in /usr/local/etc/telegraf.conf and send a HUP to the telegraf pid. You're going to need to do this from a ssh shell. One you update the config you are going to need to tell telegraf to read the new configs. If you restart telegraf from pfSense, this will not work since it will overwrite your changes.
debug = true
quiet = false
logfile = "/var/log/telegraf/telegraf.log"
# ps aux | grep '[t]elegraf.conf'
# kill -HUP <pid of telegraf proces>
Now go read /var/log/telegraf/telegraf.log
When in doubt, run a few queries to see if the data you are looking for is being populated.
bash-4.4# influx
Connected to http://localhost:8086 version 1.8.3
InfluxDB shell version: 1.8.3
> auth
username: admin
password:
> show databases
name: databases
name
----
pfsense
_internal
> use pfsense
Using database pfsense
> show measurements
name: measurements
name
----
cpu
disk
diskio
gateways
interface
mem
net
netstat
pf
processes
swap
system
tail_dnsbl_log
tail_ip_block_log
temperature
> select * from system limit 20
name: system
time host load1 load15 load5 n_cpus n_users uptime uptime_format
---- ---- ----- ------ ----- ------ ------- ------ -------------
1585272640000000000 pfSense.home 0.0615234375 0.07861328125 0.0791015625 4 1 196870 2 days, 6:41
1585272650000000000 pfSense.home 0.05126953125 0.07763671875 0.076171875 4 1 196880 2 days, 6:41
1585272660000000000 pfSense.home 0.04296875 0.07666015625 0.0732421875 4 1 196890 2 days, 6:41
1585272670000000000 pfSense.home 0.03564453125 0.07568359375 0.0703125 4 1 196900 2 days, 6:41
1585272680000000000 pfSense.home 0.02978515625 0.07470703125 0.0673828125 4 1 196910 2 days, 6:41
1585272690000000000 pfSense.home 0.02490234375 0.07373046875 0.064453125 4 1 196920 2 days, 6:42
...
How to drop influx measurement
bash-4.4# influx
Connected to http://localhost:8086 version 1.8.3
InfluxDB shell version: 1.8.3
> auth
username: admin
password:
> use pfsense
Using database pfsense
> drop measurement ip_block_log
I was going to post this in the thread made by /u/seb6596 since this is based on their dashboard, but I made quite a few changes and wanted to include information that would get lost in the thread.
What I updated: