search

bntjah / lancache

Improve download speeds and reduce strain on your Internet connection at LAN parties. Locally cache game installs and updates from the largest distributors: Steam, RIOT, Blizard, Hirez, Origin, Sony, Microsoft, Tera, GOG, ArenaNetworks, WarGaming, and Uplay. Super easy to setup with auto installer script!
174 stars 44 forks source link

making filebeat work with our lancache #128

Open billthecatt opened 5 years ago

billthecatt commented 5 years ago

Hello,

bntjah, can you please post a copy of your /etc/filebeat/filebeat.yml?

Mine looks like this: name: "lancache" filebeat.prospectors:

output.logstash: hosts: ["server:5044"]

And something isn't quite working right.. Guessing my paths are goofed up?

Thanks!

bntjah commented 5 years ago

Quick Q is Filebeat active through ps aux | grep filebeat and is there something listening on this port?

My setup is a bith different in the area that I am running filebeat on the lancache and then shipping it through an internal vm only network to my elk vm. I am starting the vm up so I can copy / paste the config here

billthecatt commented 5 years ago

I too am running filebeat on the lancache and shipping it over to the elk server. My question relates to how your filebeat is getting in to the logs. We have our split out into Access/Errors/Keys and I'm curious if i point filebeat at the root /nginx_cache/logs or at /nginx_cache/logs/Access or use multiple lines?

ps -ef indicates: /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat

Not sure it'd be listening on this port, since it's shipping stuff off to the lancache-elk server but I did: lsof -i :5044 on the ELK server and it's listening as expected: lsof -i :5044 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 10510 logstash 235u IPv4 93006582 0t0 TCP *:5044 (LISTEN)

bntjah commented 5 years ago

name: "lancache" filebeat.prospectors:

output.logstash: hosts: ["172.16.0.250:5044"]

billthecatt commented 5 years ago

TYVM. Mine matches yours now.

Guessing you must have modified how your logs are being written as well? Ours was configured as something like /srv/lancache/Keys/steam.log /srv/lancache/Access/steam.log /srv/lancache/Errors/steam.log

it looks like we'll need to change to $CACHE_LOGS_DIRECTORY/steam-access.log $CACHE_LOGS_DIRECTORY/steam-error.log $CACHE_LOGS_DIRECTORY/steam-keys.log

And have all the log files dump into /srv/lancache/logs instead of into individual folders instead?

billthecatt commented 5 years ago

Modified my /etc/nginx/vhosts-enabled/lancache-steam.conf as follows:

access_log /nginx_cache/logs/steam-access.log main buffer=128k flush=1m;
access_log /nginx_cache/logs/steam-keys.log keys_uri;
error_log /nginx_cache/logs/steam-error.log;

And I'm now getting data all the way into kibana.

TYVM for your help on this today!

bntjah commented 5 years ago

Yeah sorry I did change this how it was formatted; I copied the settings over from the guys over @ zeropingheroes

billthecatt commented 5 years ago

Hey no worries, It's working solid now. I commented out the keys log file as the new log format appears to write that data to the access file and I figured it'd cut down on performance impacting disk writes. I also reduced my log buffer time from 1m to 5s to mirror the refresh time in ELK. I'm sure that impacts disk performance but my nginx logs are written to a separate disk so I think I'm ok.

Thanks again for pointing me in the right direction on this. I really like having the analytics that help me figure out when someone is pulling data that's not already in the cache..