search

molu8bits / squid-filebeat-kibana

Filebeat module for Squid access.log + Kibana dashboards. ELK 7.x
Apache License 2.0
17 stars 14 forks source link
elasticsearch elk filebeat kibana kibana-dashboard squid squid-access squid-filebeat-kibana visualisation visualisations

readme

squid-filebeat-kibana

Filebeat module for Squid access logs + Kibana dashboards. ELK 7.8

TL;DR

Collect your squid access.log with Filebeat, send directly to Elasticsearch. Get overview of Squid access log using Kibana dashboard.

Elastichsearch and Kibana

  1. Elasticsearch and Kibana

    a.) Install Elasticsearch and Kibana.

    b.) Configure firewall to allow access from filebeat host to elasticsearch service.

Filebeat + module squid installation

  1. Configuration Filebeat (7.x recommended. Older versions may not work)

    a.) copy filebeat/module/squid into /usr/share/filebeat/module

    b.) copy filebeat/etc/filebeat/modules.d/squid.yml.disabled into /etc/filebeat/modules.d

    c.) configure /etc/filebeat/filebeat.yml - reference file placed in /etc/filebeat/filebeat.yml (change hosts ["elasticsearch.local"] in section output.elastichsearch to elastichsarch instance listening from filebeat host

    d.) enable Filebeat squid module by command "filebeat modules enable squid" (or just rename /etc/filebeat/modules.d/squid.yml.disabled to /etc/filename/modules.d/squid.yml

    e.) Replace /etc/filebeat/fields.yml with filebeat/etc/fields.yml from repo. (Important! - This must be done before run filebeat). This file is a compiled version from 7.3 so the rest of functionality should work. If filebeat with newer definitions is needed then just cut squid section and join into newer field.yml"

    f.) restart Filebeat service - "systemctl restart filebeat"

Kibana configuration

  1. Import Dashboard definition from kibana folder Kibana (*.ndjson file via GUI: Management -> Stack Management -> Kibana -> Saved Objects -> Import)

Check Dashboard view on Kibana

  1. Go to the Dashboard section and find "[Filebeat Squid] Access log". Set Time-Range according to expected log entries.

Troubleshooting

Elasticsearch needs to know what types should be applied to particular fields during processing logs. For all the modules they are available in /etc/filebeat/fields.yml and this repo contains of modified fields.yml filebeat 7.3 with compiled into mappings required for the module squid. If they are not applied automatically on the Filebeat Index (e.g. index filebeat-* already exist and) then try to remove index, filebeat ingest pipeline and filebeat mapping. ```bash cat filebeat/etc/squid-fields.yml >> /etc/filebeat/fields.yml curl -XDELETE elasticsearch.local:9200/filebeat-index-name curl -XDELETE elasticsearch.local:9200/_ingest/pipeline/filebeat*squid* systemctl restart filebeat ```

Module has been tested with latest Elasticsearch/Kibana 7.9 based on docker-compose stack: https://www.elastic.co/guide/en/elasticsearch/reference/7.9/docker.html In case of trouble such quick clean installation of ELK is recommended to test filebeat configuration.