a3ilson
commented
3 years ago @SaiKiran449 - thanks for the inquiry!
The squid portion was built utilizing OPNsense. The logs are sent in the json format thus not requiring a grok pattern. However, i see that you are using pfSense which I believe may be sending the squid logs differently.
Are you able to provide setup steps and options? Additionally, if you don’t mind are are able to send squid logs in their raw format (sanitized...i.e. replace ip with non routeable) and I’ll be happy to amend to support pfSenses output.
SaiKiran449
Describe the bug I have deployed the pfelk through docker containers. I am able to receive the data from my pfsense to the elastic and I can see the indices created in Kibana and the data is being added to the index. However, I can't see the visualization in the Squid dashboard
To Reproduce Steps to reproduce the behavior:
Screenshots
Operating System (please complete the following information):
OS (
printf "$(uname -srm)\n$(cat /etc/os-release)\n"): Linux 5.4.0-59-generic x86_64 NAME="Ubuntu" VERSION="18.04.5 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.5 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionicVersion of Docker (
docker -v): Docker version 20.10.1, build 831ebeaVersion of Docker-Compose (
docker-compose -v): docker-compose version 1.24.0, build 0aa59064Elasticsearch, Logstash, Kibana (please complete the following information):
Version of ELK (
cat /docker-pfelk/.env): ELK_VERSION=7.10.1Service logs
docker-compose logs pfelk01docker-compose logs pfelk02es03 | {"type": "server", "timestamp": "2021-01-08T09:31:27,993Z", "level": "INFO", "component": "o.e.x.s.a.TokenService", "cluster.name": "es-docker-cluster", "node.name": "es03", "message": "refresh keys", "cluster.uuid": "XuOhpVCQ-4ePdZ68tELQ", "node.id": "u2PpmSdrSAmFsXOE7ZxWwg" } es03 | {"type": "server", "timestamp": "2021-01-08T09:31:28,126Z", "level": "INFO", "component": "o.e.x.s.a.TokenService", "cluster.name": "es-docker-cluster", "node.name": "es03", "message": "refreshed keys", "cluster.uuid": "XuOhpVCQ-4ePdZ68tELQ", "node.id": "u2PpmSdrSAmFsXOE7ZxWwg" } es03 | {"type": "server", "timestamp": "2021-01-08T09:31:28,199Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "es-docker-cluster", "node.name": "es03", "message": "added {{es01}{5xp6gCnEQvG8Ez3j5X_6og}{ClUVqoGoREuh12Y7eees-Q}{172.18.0.3}{172.18.0.3:9300}{cdhilmrstw}{ml.machine_memory=16646402048, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}}, term: 79, version: 1263, reason: ApplyCommitRequest{term=79, version=1263, sourceNode={es02}{3AmE-f8oSlepBeIuHRWaOQ}{UIdhHD5_TAOx_JkhP4rH6g}{172.18.0.4}{172.18.0.4:9300}{cdhilmrstw}{ml.machine_memory=16646402048, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}}", "cluster.uuid": "XuOhpVCQ-4ePdZ68tELQ", "node.id": "u2PpmSdrSAmFsXOE7ZxWwg" }
logstash | [INFO ] 2021-01-08 09:31:40.606 [[main]<udp] udp - UDP listener started {:address=>"0.0.0.0:5190", :receive_buffer_bytes=>"106496", :queue_size=>"2000"} logstash | [INFO ] 2021-01-08 09:31:40.607 [[main]<udp] udp - UDP listener started {:address=>"0.0.0.0:5140", :receive_buffer_bytes=>"106496", :queue_size=>"2000"} logstash | [INFO ] 2021-01-08 09:31:40.610 [[main]<udp] udp - UDP listener started {:address=>"0.0.0.0:5141", :receive_buffer_bytes=>"106496", :queue_size=>"2000"} logstash | [INFO ] 2021-01-08 09:31:40.653 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
kibana | {"type":"log","@timestamp":"2021-01-08T09:31:32Z","tags":["info","plugins","watcher"],"pid":10,"message":"Your basic license does not support watcher. Please upgrade your license."} kibana | {"type":"log","@timestamp":"2021-01-08T09:31:32Z","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":10,"message":"Starting monitoring stats collection"} kibana | {"type":"log","@timestamp":"2021-01-08T09:31:33Z","tags":["listening","info"],"pid":10,"message":"Server running at http://0:5601"} kibana | {"type":"log","@timestamp":"2021-01-08T09:31:34Z","tags":["info","http","server","Kibana"],"pid":10,"message":"http server running at http://0:5601"} kibana | {"type":"log","@timestamp":"2021-01-08T09:31:34Z","tags":["error","elasticsearch","data"],"pid":10,"message":"[version_conflict_engine_exception]: [task:endpoint:user-artifact-packager:1.0.0]: version conflict, document already exists (current version [9190])"} kibana | {"type":"log","@timestamp":"2021-01-08T09:31:34Z","tags":["error","elasticsearch","data"],"pid":10,"message":"[version_conflict_engine_exception]: [task:Lens-lens_telemetry]: version conflict, document already exists (current version [16])"} kibana | {"type":"log","@timestamp":"2021-01-08T09:31:34Z","tags":["error","elasticsearch","data"],"pid":10,"message":"[version_conflict_engine_exception]: [task:Actions-actions_telemetry]: version conflict, document already exists (current version [16])"} kibana | {"type":"log","@timestamp":"2021-01-08T09:31:34Z","tags":["error","elasticsearch","data"],"pid":10,"message":"[version_conflict_engine_exception]: [task:apm-telemetry-task]: version conflict, document already exists (current version [22])"} kibana | {"type":"log","@timestamp":"2021-01-08T09:31:34Z","tags":["error","elasticsearch","data"],"pid":10,"message":"[version_conflict_engine_exception]: [task:Alerting-alerting_telemetry]: version conflict, document already exists (current version [16])"} kibana | {"type":"log","@timestamp":"2021-01-08T09:31:35Z","tags":["warning","plugins","reporting"],"pid":10,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}