making filebeat work with our lancache

[billthecatt]

Hello,

bntjah, can you please post a copy of your /etc/filebeat/filebeat.yml?

Mine looks like this: name: "lancache" filebeat.prospectors:

• type: log paths:
  • /nginx_cache/logs/*.log

output.logstash: hosts: ["server:5044"]

And something isn't quite working right.. Guessing my paths are goofed up?

Thanks!

[bntjah]

Quick Q is Filebeat active through ps aux | grep filebeat and is there something listening on this port?

My setup is a bith different in the area that I am running filebeat on the lancache and then shipping it through an internal vm only network to my elk vm. I am starting the vm up so I can copy / paste the config here

[billthecatt]

I too am running filebeat on the lancache and shipping it over to the elk server. My question relates to how your filebeat is getting in to the logs. We have our split out into Access/Errors/Keys and I'm curious if i point filebeat at the root /nginx_cache/logs or at /nginx_cache/logs/Access or use multiple lines?

ps -ef indicates: /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat

Not sure it'd be listening on this port, since it's shipping stuff off to the lancache-elk server but I did: lsof -i :5044 on the ELK server and it's listening as expected: lsof -i :5044 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 10510 logstash 235u IPv4 93006582 0t0 TCP *:5044 (LISTEN)

[bntjah]

name: "lancache" filebeat.prospectors:

• type: log paths:
  • /srv/lancache/logs/*.log

output.logstash: hosts: ["172.16.0.250:5044"]

[billthecatt]

TYVM. Mine matches yours now.

Guessing you must have modified how your logs are being written as well? Ours was configured as something like /srv/lancache/Keys/steam.log /srv/lancache/Access/steam.log /srv/lancache/Errors/steam.log

it looks like we'll need to change to $CACHE_LOGS_DIRECTORY/steam-access.log $CACHE_LOGS_DIRECTORY/steam-error.log $CACHE_LOGS_DIRECTORY/steam-keys.log

And have all the log files dump into /srv/lancache/logs instead of into individual folders instead?

[billthecatt]

Modified my /etc/nginx/vhosts-enabled/lancache-steam.conf as follows:

access_log /nginx_cache/logs/steam-access.log main buffer=128k flush=1m;
access_log /nginx_cache/logs/steam-keys.log keys_uri;
error_log /nginx_cache/logs/steam-error.log;

And I'm now getting data all the way into kibana.

TYVM for your help on this today!

[bntjah]

Yeah sorry I did change this how it was formatted; I copied the settings over from the guys over @ zeropingheroes

[billthecatt]

Hey no worries, It's working solid now. I commented out the keys log file as the new log format appears to write that data to the access file and I figured it'd cut down on performance impacting disk writes. I also reduced my log buffer time from 1m to 5s to mirror the refresh time in ELK. I'm sure that impacts disk performance but my nginx logs are written to a separate disk so I think I'm ok.

Thanks again for pointing me in the right direction on this. I really like having the analytics that help me figure out when someone is pulling data that's not already in the cache..