search

bntjah / lancache

Improve download speeds and reduce strain on your Internet connection at LAN parties. Locally cache game installs and updates from the largest distributors: Steam, RIOT, Blizard, Hirez, Origin, Sony, Microsoft, Tera, GOG, ArenaNetworks, WarGaming, and Uplay. Super easy to setup with auto installer script!
175 stars 43 forks source link

Lancache isn't working #138

Open ramzess1 opened 5 years ago

ramzess1 commented 5 years ago

Hello,

I have installed the Lancache server. Lancache, Unbound and NGINX are activated. Forwarding addresses in Unbound file is working well. But when i add an entry to test the cache for example : local-zone: "download.microsoft.com." redirect local-data: "download.microsoft.com. 600 IN A 1..."

Lancache is doing nothing, there is no logs and i got a request time out from the PC client. The dns also isn't resolving the download.microsoft.com from my pc.

For the other forwarded sites the dns is resolving.

Thank you for your response.

nexusofdoom commented 5 years ago

I may be able to help tonight team-viewer ssh access to box?

bntjah commented 5 years ago

From what I can tell from your error; I think you should look into your DNS. Might be an interesting test to try on a Windows System to change the host file with the correct DNS and check if this directs to lancache?

Op do 10 jan. 2019 om 22:54 schreef nexusofdoom notifications@github.com:

I may be able to help tonight team-viewer ssh access to box?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bntjah/lancache/issues/138#issuecomment-453268411, or mute the thread https://github.com/notifications/unsubscribe-auth/AGE_0TNHAKyBL6Bbt3vhy04vzBPn1zi_ks5vB7aEgaJpZM4Z5yrK .

ramzess1 commented 5 years ago

Problem solved, it was the ipv6 NIC enabled so i unchecked it.

I have another issue using this test link : http://download.microsoft.com/download/0/9/5/0953E553-3BB6-44B1-8973-106F1B7E5049/wmp11-windowsxp-x86-enu.exe

It gives me: 404 Not Found nginx/1.13.4

This problem is lnked to the name download.microsoft.com added in the unbound file.

ramzess1 commented 5 years ago

capture

ramzess1 commented 5 years ago

capture

bntjah commented 5 years ago

This looks more like the config of microsoft isn't working as nginx reverse proxy isn't hooking to the download.microsoft.com or the bind ip...

Op vr 11 jan. 2019 om 10:29 schreef ramzess1 notifications@github.com:

[image: capture] https://user-images.githubusercontent.com/46570020/51025256-3761a500-1594-11e9-80a2-2d7c87292b28.PNG

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bntjah/lancache/issues/138#issuecomment-453449742, or mute the thread https://github.com/notifications/unsubscribe-auth/AGE_0e3AlYpJD32-e9S1djRj5w2EvV6Sks5vCFmVgaJpZM4Z5yrK .

ramzess1 commented 5 years ago

Do you mean it should work for other sites? because i am only testing the "download.microsoft.com" on my windows PC

bntjah commented 5 years ago

Have a look into your config that are loaded by nginx. See line 3 and 4 of https://github.com/bntjah/lancache/blob/master/conf/vhosts-enabled/lancache-microsoft.conf lancache-microsoft is defined in the hosts file so any traffic going over port 80 on the ip defined in hosts will be picked up by nginx and sent through the reverse proxy...

Op vr 11 jan. 2019 om 10:44 schreef ramzess1 notifications@github.com:

Do you mean it should work for other sites? because i am only testing the "download.microsoft.com" on my windows PC

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

ramzess1 commented 5 years ago

There's no logs in: access_log /srv/lancache/logs/Access/ access_log /srv/lancache/logs/Keys/ error_log /srv/lancache/logs/Errors/

And find next the microsoft host file and the hosts file.

And thank you for you help.

ramzess1 commented 5 years ago

microsoft

ramzess1 commented 5 years ago

hosts

bntjah commented 5 years ago

Hey Ramzess,

Looking through your screenshots I see no where where NGINX should listen on the 10.126.14.9 so I guess this is where you issue is situated. Logs won't be generated as no traffic is actually reaching the NGINX config used for Microsoft...

Op vr 11 jan. 2019 om 11:14 schreef ramzess1 notifications@github.com:

[image: hosts] https://user-images.githubusercontent.com/46570020/51027772-6549e800-159a-11e9-8ace-cf77b5027a5e.PNG

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bntjah/lancache/issues/138#issuecomment-453465699, or mute the thread https://github.com/notifications/unsubscribe-auth/AGE_0RPYSViRUib5OeN6IuAS0tdvhk86ks5vCGPzgaJpZM4Z5yrK .

ramzess1 commented 5 years ago

I got this log error in NGINX logs: 2019/01/11 12:22:23 [error] 7092#7092: *1 open() "/usr/local/nginx/html/download/0/9/5/0953E553-3BB6-44B1-8973-106F1B7E5049/wmp11-windowsxp-x86-enu.exe" failed (2: No such file or directory)

bntjah commented 5 years ago

Ramzess could you give me a screenshot of your nginx.conf and the lancache-microsoft.conf complete content?

Op vr 11 jan. 2019 om 11:45 schreef ramzess1 notifications@github.com:

I got this log error in NGINX logs: 2019/01/11 12:22:23 [error] 7092#7092: *1 open() "/usr/local/nginx/html/download/0/9/5/0953E553-3BB6-44B1-8973-106F1B7E5049/wmp11-windowsxp-x86-enu.exe" failed (2: No such file or directory)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bntjah/lancache/issues/138#issuecomment-453477261, or mute the thread https://github.com/notifications/unsubscribe-auth/AGE_0aZfTJDT4QMDorGxLAFqn2yKa9f-ks5vCGtkgaJpZM4Z5yrK .

ramzess1 commented 5 years ago

capture

ramzess1 commented 5 years ago

capture

ramzess1 commented 5 years ago

capture

ramzess1 commented 5 years ago

capture

ramzess1 commented 5 years ago

I only changed the proxy bind variable in lancache-microsoft file

bntjah commented 5 years ago

Hey Ramzess, don't follow this commands blindly (it will most likely fix your issue, but see why I suggest this): mv /usr/local/nginx/nginx.conf /usr/local/nginx/nginx.conf.bak wget https://raw.githubusercontent.com/bntjah/lancache/master/conf/nginx.conf

Reasoning: comparing your nginx.conf towards https://raw.githubusercontent.com/bntjah/lancache/master/conf/nginx.conf I see alot of differences; it seems like your nginx.conf seems to be the default one that nginx installed with and not the one from the github repo...

Op vr 11 jan. 2019 om 12:09 schreef ramzess1 notifications@github.com:

I only changed the proxy bind variable in lancache-microsoft file

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

ramzess1 commented 5 years ago

The screenshots sent of nginx.conf are for the executable file given by nginx -t.

The other one has the same config as in Github.

The error seems to be a location error for the get http links.

ramzess1 commented 5 years ago

Everything is activated but the problem nginx isn't downloading the requested webpage.

The location inside the executor of nginx in the screenshot sent is showing "root: html" but in that folder there is only index.html the nginx default page.

I don't know if i should change the location inside the nginx executor file or there's anything else.

nexusofdoom commented 5 years ago

are you downloading as http or https ?

nexusofdoom commented 5 years ago

my setup is working run this sudo httpry -i ethX "X is what your network is setup as" what do you see?

example of what I see

2019-01-11 11:00:48 10.0.2.232 192.168.30.232 > GET download.microsoft.com /download/0/9/5/0953E553-3BB6-44B1-8973-106F1B7E5049/wmp11-windowsxp-x86-enu.exe HTTP/1.1 - - 2019-01-11 11:00:48 192.168.30.232 10.0.2.232 < - - - HTTP/1.1 200 OK

my logs

/srv/lancache/logs/Keys/micrsoft.log [11/Jan/2019:11:00:42 -0600] - REQUEST="microsoft/download/0/9/5/0953E553-3BB6-44B1-8973-106F1B7E5049/wmp11-windowsxp-x86-enu.exe" - KEY="microsoft/download/0/9/5/0953E553-3BB6-44B1-8973-106F1B7E5049/wmp11-windowsxp-x86-enu.exe -" - CACHE="HIT" [11/Jan/2019:11:00:48 -0600] - REQUEST="microsoft/download/0/9/5/0953E553-3BB6-44B1-8973-106F1B7E5049/wmp11-windowsxp-x86-enu.exe" - KEY="microsoft/download/0/9/5/0953E553-3BB6-44B1-8973-106F1B7E5049/wmp11-windowsxp-x86-enu.exe -" - CACHE="HIT"

ramzess1 commented 5 years ago

Ok nexusofdoom i will try it on Monday.

It is an http link and was always showing 404 not found because it is pointing to etc/local/nginx/html/"the requested page file" which doesn't exist. I figured this by looking at nginx/logs/error.log as sent in one of the last comments.

This location only contains the default page index.html and the 505.. error file.

So i think nginx isn't downloading the file.

Https downloads doesn't work with lancache right? Because while downloading https from microsoft it dowloads the file normally but it doesn't cache it.

Thank you all for your replies.

nexusofdoom commented 5 years ago

If you have a spare pc box or vm I would try reloading linux and lancache and see how it works.

raz3r83 commented 5 years ago

what's the output of: service lancache status

ramzess1 commented 5 years ago

Lancache is activated.

Nexusofdoom i will try to reinstall nginx, maybe there's something in the installation but on Monday because i don't have my pc now.

Can you send me the steps to install the full nginx please?

raz3r83 commented 5 years ago

Easiest way to install is using the install script https://github.com/nexusofdoom/lancache-installer Written by nexusofdoom :)

Its been a while since I tried a manual install but you have cant just run apt-get install nginx as you have to load additional modules so have to run a compile, instructions are in the readme I believe.

ramzess1 commented 5 years ago

These are the steps that i made:

1) Configure the network interface

2)apt-get install curl git unbound build-essential libpcre3 zlib1g-dev libreadline-dev libncurses5-dev libssl-dev httpry libudns0 libudns-dev libev4 libev-dev devscripts automake libtool autoconf autotools-dev cdbs debhelper dh-autoreconf dpkg-dev gettext pkg-config fakeroot libpcre3-dev libgd2-xpm-dev libgeoip-dev tcpdump -y

3)curl http://nginx.org/download/nginx-1.13.4.tar.gz | tar zx

cd nginx-1.13.4 curl "http://labs.frickle.com/files/ngx_cache_purge-2.3.tar.gz" | tar zx git clone https://github.com/multiplay/nginx-range-cache/ $PWD/nginx-range-cache curl "https://codeload.github.com/wandenberg/nginx-push-stream-module/tar.gz/0.5.1?dummy=/wandenberg-nginx-push-stream-module-0.5.1_GH0.tar.gz" | tar zx

4)git clone -b master http://github.com/bntjah/lancache

5) cd nginx-1.13.4 patch -p1 <$PWD/nginx-range-cache/range_filter.patch ./configure --modules-path=$PWD --with-cc-opt='-I /usr/local/include' --with-ld-opt='-L /usr/local/lib' --conf-path=/usr/local/nginx/nginx.conf --sbin-path=/usr/local/sbin/nginx --pid-path=/var/run/nginx.pid --with-file-aio --add-module=$PWD/ngx_cache_purge-2.3 --with-http_flv_module --with-http_geoip_module=dynamic --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --add-module=$PWD/nginx-range-cache --with-http_realip_module --with-http_slice_module --with-http_stub_status_module --with-pcre --with-http_v2_module --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_ssl_module --add-module=$PWD/nginx-push-stream-module-0.5.1 --with-threads make make install

6) Adding the virtual interfaces with ips in the same network as network interface's ip

7)adduser --system --no-create-home lancache addgroup --system lancache usermod -aG lancache lancache

8)mkdir -p /srv/lancache/data/{microsoft,installs,other,tmp,hirez,origin,riot,gog,sony,steam,wargaming,arenanetworks,uplay,glyph,zenimax,digitalextremes,pearlabyss, blizzard, apple} mkdir -p /srv/lancache/logs/{Errors,Keys,Access}

9)chown -R lancache:lancache /srv/lancache

10)cp -R ~/lancache/conf /usr/local/nginx/

11)Replace the proxy-bind variable in each lancache-"service-name" file and in the hosts file

12)cp ~/lancache/init.d/lancache /etc/init.d/

13)chmod +x /etc/init.d/lancache

14)update-rc.d lancache defaults

15)cp ~/lancache/limits.conf /etc/security/

16)cp ~/lancache/hosts /etc/ and changed the virtual ip near each lancache service

17)echo "net.ipv6.conf.all.disable_ipv6=1" >/etc/sysctl.d/disable-ipv6.conf sysctl -p /etc/sysctl.d/disable-ipv6.conf

18) git clone https://github.com/dlundquist/sniproxy curl https://raw.githubusercontent.com/OpenSourceLAN/origin-docker/master/sniproxy/sniproxy.conf -o /etc/sniproxy.conf cd sniproxy ./autogen.sh && ./configure && make check && make install cp ~/lancache/init.d/sniproxy /etc/init.d/ chmod +x /etc/init.d/sniproxy update-rc.d sniproxy defaults

19) cp ~/lancache/unbound/unbound.conf /etc/unbound Added the virtual ips of each service added the ip of network interface

20)service lancache start

21)service unbound start

Is there anything left?

ramzess1 commented 5 years ago

Nexusofdoom this script "https://github.com/nexusofdoom/lancache-installer"installs automatically the full lancache service ??

ramzess1 commented 5 years ago

I saw it, greatt!

So i just have to change the variables of ips, dns.. in the files and the run the script?

Thankss

raz3r83 commented 5 years ago

there's really very little you would need to change in nexus's script though it needs a little bit of an update. 1 warning it makes pretty drastic changes to the netplan which might cause you issues if you are bonding network ports together.

  1. issue its using ifconfig to try and identify the interface name for later use which has been depreciated and isn't installed in a new install.
  2. issue its using route to identify the network gateway which doesn't seem to be installed as default either.
  3. another check for interface name is using ip link show but its searching for ens and my network device names eth so doesn't pickup a value.

The only real affect the 3 issues have are that I had to go and update the netplan yaml to set interface name and gateway, everything else appears to be working without having changed anything.

This was on a fresh ubuntu 18.04 container running on a proxmox host.

nexusofdoom commented 5 years ago

I know that it works with "Ubuntu 18.04.1 Server" http://cdimage.ubuntu.com/releases/18.04.1/release/ubuntu-18.04.1-server-amd64.iso

Did you install 18.04 or 18.04.1?

ramzess1 commented 5 years ago

I executed the script of nexusofdomm and i got this error of unbound file

-- Unit unbound.service has finished shutting down. Jan 14 17:25:14 LANCache systemd[1]: unbound.service: Start request repeated too quickly. Jan 14 17:25:14 LANCache systemd[1]: unbound.service: Failed with result 'exit-code'. Jan 14 17:25:14 LANCache systemd[1]: Failed to start Unbound DNS server. -- Subject: Unit unbound.service has failed -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit unbound.service has failed. -- -- The result is RESULT. Jan 14 17:25:14 LANCache systemd[1]: unbound.service: Start request repeated too quickly. Jan 14 17:25:14 LANCache systemd[1]: unbound.service: Failed with result 'exit-code'. Jan 14 17:25:14 LANCache systemd[1]: Failed to start Unbound DNS server.

nexusofdoom commented 5 years ago

Are you running Ubuntu 18.04.1 Server?

ramzess1 commented 5 years ago

yess

nexusofdoom commented 5 years ago

just did a fresh install and unbound is working no issues. image

nexusofdoom commented 5 years ago

image

ramzess1 commented 5 years ago

Did you do apt-get update and apt-get upgrade before the installation?

nexusofdoom commented 5 years ago

The script does the updates

Update packages

echo "Installing package updates..." universeCheck=$(apt-cache policy |grep universe) if [[ -z $universeCheck ]]; then echo "Adding universe repository..." apt-add-repository universe else apt -y update fi apt -y upgrade

nexusofdoom commented 5 years ago

I tried this on two other computers and one of the computers the network card was having issues replaced and the installer worked great.

ramzess1 commented 5 years ago

Trying to rerun the script on a new ubuntu 18.04.1, it gives

Installing package updates... Adding universe repository... 'universe' distribution component is already enabled for all sources. Reading package lists... Done Building dependency tree
Reading state information... Done Calculating upgrade... Done 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Reading package lists... Done Building dependency tree
Reading state information... Done Calculating upgrade... Done 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Installing required updates... Reading package lists... Done Building dependency tree

nexusofdoom commented 5 years ago

Sorry don't know what to say it updates and it also updates the kernel. I tried again today and its working. fresh install of ubuntu

Download url for Ubuntu 18.04.1 Server http://cdimage.ubuntu.com/releases/18.04.1/release/ubuntu-18.04.1-server-amd64.iso

You will need 18 avaliable IP's example 192.168.0.2 - 192.168.0.20 used for lancache Clone the git repo git clone -b master http://github.com/nexusofdoom/lancache-installer

cd lancache-installer

Run scripts with sudo Run sudo ./install-lancache.sh

ramzess1 commented 5 years ago

Configuration file '/etc/default/sniproxy' ==> File on system created by you or by a script. ==> File also in package provided by package maintainer. What would you like to do about it ? Your options are: Y or I : install the package maintainer's version N or O : keep your currently-installed version D : show the differences between the versions Z : start a shell to examine the situation The default action is to keep your current version. *** sniproxy (Y/I/N/O/D/Z) [default=N] ?

which one please?

ramzess1 commented 5 years ago

Thank you all for your help, it works correctly.

Steps for new lancache installers : -Ubuntu 18.04.1 with OpenSSH only -apt -y update -Run the script -"Y" for any asked question -Reboot at the end