search

bntjah / lancache

Improve download speeds and reduce strain on your Internet connection at LAN parties. Locally cache game installs and updates from the largest distributors: Steam, RIOT, Blizard, Hirez, Origin, Sony, Microsoft, Tera, GOG, ArenaNetworks, WarGaming, and Uplay. Super easy to setup with auto installer script!
175 stars 43 forks source link

Issues when testing Windows Update #37

Closed nexusofdoom closed 7 years ago

nexusofdoom commented 7 years ago

[crit] 686#0: 271 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.179, server: microsoft, request: "GET /d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_3_aadae06fd0885db7996e702e453f784b4683d978.psf HTTP/1.1", upstream: "http://[2001:1900:2302:2f04::ff]:80/d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_3_aadae06fd0885db7996e702e453f784b4683d978.psf", host: "download.windowsupdate.com" 2017/03/30 21:27:44 [crit] 692#0: 274 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.179, server: microsoft, request: "GET /d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_1_9881d994bf0f17a1120e12723450b6e54e499075.psf HTTP/1.1", upstream: "http://[2001:1900:2304:8f0a::ff]:80/d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_1_9881d994bf0f17a1120e12723450b6e54e499075.psf", host: "download.windowsupdate.com" 2017/03/30 21:27:51 [crit] 692#0: *269 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.179, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_3_aadae06fd0885db7996e702e453f784b4683d978.psf HTTP/1.1", upstream: "http://[2001:1900:2304:af07::ff]:80/d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_3_aadae06fd0885db7996e702e453f784b4683d978.psf", host: "download.windowsupdate.com"

nexusofdoom commented 7 years ago

this might be the issue for ms updates

http://mailman.nginx.org/pipermail/nginx/2007-May/000961.html

nexusofdoom commented 7 years ago

Hi guys Im just trying to work out what the cause of this error is

crit] 29494#0: *788901 sendfile() failed (22: Invalid argument) while sending response to client It only appears to happen on 1 3gig file the file gets downloaded over 1000 times a day this happens about 5 times.

What nginx version? The latest changelog (0.5.19) includes this:

*) Bugfix: the files more than 2G could not be transferred using sendfile and limit_rate on 64-bit platforms.

*) Bugfix: the files more than 2G could not be transferred using sendfile on 64-bit Linux.

Possibly related? If you are running 0.5.19, I don't have a fix but I do have a workaround. Put "sendfile off;" in your config, and nginx will use a different system call to send the dat

wendelj commented 7 years ago

Change this line in nginx.conf: resolver 8.8.8.8;

To this: resolver 8.8.8.8 ipv6=off;

nexusofdoom commented 7 years ago

ok will give a try this week and let you know

nexusofdoom commented 7 years ago

still getting this when doing updates , note it looks like a windows 10 updates are the one having the issues , I do not get any errors when doing updates with windows 7

017/04/03 18:31:35 [crit] 688#0: 3450 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.158, server: microsoft, request: "HEAD /d/msdownload/update/software/crup/2017/03/windows10.0-kb4013418-x64_e35782f290df71c9f00a52b0203a772b876d6e97.psf HTTP/1.1", upstream: "http://[2a01:111:2003::50]:80/d/msdownload/update/software/crup/2017/03/windows10.0-kb4013418-x64_e35782f290df71c9f00a52b0203a772b876d6e97.psf", host: "au.download.windowsupdate.com" 2017/04/03 18:31:36 [crit] 685#0: 3412 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.159, server: microsoft, request: "GET /c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe HTTP/1.1", upstream: "http://[2a01:111:2003::50]:80/c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe", host: "au.download.windowsupdate.com" 2017/04/03 18:31:38 [crit] 690#0: 3457 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.158, server: microsoft, request: "HEAD /c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe HTTP/1.1", upstream: "http://[2a01:111:2003::50]:80/c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe", host: "au.download.windowsupdate.com" 2017/04/03 18:31:41 [crit] 685#0: 3352 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.158, server: microsoft, request: "GET /d/msdownload/update/software/secu/2017/03/windows10.0-kb4013429-x64-express_f112f73709fe89085542b03bf2b0332ca2c3b9b8.cab HTTP/1.1", upstream: "http://[2a01:111:2003::50]:80/d/msdownload/update/software/secu/2017/03/windows10.0-kb4013429-x64-express_f112f73709fe89085542b03bf2b0332ca2c3b9b8.cab", host: "au.download.windowsupdate.com" 2017/04/03 18:31:43 [crit] 690#0: 3460 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.158, server: microsoft, request: "HEAD /c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe HTTP/1.1", upstream: "http://[2a01:111:2003::50]:80/c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe", host: "au.download.windowsupdate.com" 2017/04/03 18:31:43 [crit] 690#0: 3461 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.158, server: microsoft, request: "HEAD /c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe HTTP/1.1", upstream: "http://[2a01:111:2003::50]:80/c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe", host: "au.download.windowsupdate.com" 2017/04/03 18:31:44 [crit] 685#0: 3421 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.159, server: microsoft, request: "GET /c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe HTTP/1.1", upstream: "http://[2a01:111:2003::50]:80/c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe", host: "au.download.windowsupdate.com" ^C travus@lancache:/srv/lancache/logs/Errors$ tail microsoft.log -f 2017/04/03 18:31:25 [crit] 688#0: 3446 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.158, server: microsoft, request: "HEAD /d/msdownload/update/software/crup/2017/03/windows10.0-kb4013418-x64_e35782f290df71c9f00a52b0203a772b876d6e97.psf HTTP/1.1", upstream: "http://[2a01:111:2003::50]:80/d/msdownload/update/software/crup/2017/03/windows10.0-kb4013418-x64_e35782f290df71c9f00a52b0203a772b876d6e97.psf", host: "au.download.windowsupdate.com" 2017/04/03 18:31:26 [crit] 685#0: 3426 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.158, server: microsoft, request: "GET /c/msdownload/update/software/uprl/2017/03/windows-kb890830-x64-v5.46_dda7f98e0f03bcab3bdba5eb0ca799771c79f141.exe HTTP/1.1", upstream: "http://[2a01:111:2003::50]:80/c/msdownload/update/software/uprl/2017/03/windows-kb890830-x64-v5.46_dda7f98e0f03bcab3bdba5eb0ca799771c79f141.exe", host: "au.download.windowsupdate.com" 2017/04/03 18:31:31 [crit] 688#0: 3402 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.158, server: microsoft, request: "GET /c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe HTTP/1.1", upstream: "http://[2a01:111:2003::50]:80/c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe", host: "au.download.windowsupdate.com" 2017/04/03 18:31:35 [crit] 688#0: 3450 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.158, server: microsoft, request: "HEAD /d/msdownload/update/software/crup/2017/03/windows10.0-kb4013418-x64_e35782f290df71c9f00a52b0203a772b876d6e97.psf HTTP/1.1", upstream: "http://[2a01:111:2003::50]:80/d/msdownload/update/software/crup/2017/03/windows10.0-kb4013418-x64_e35782f290df71c9f00a52b0203a772b876d6e97.psf", host: "au.download.windowsupdate.com" 2017/04/03 18:31:36 [crit] 685#0: 3412 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.159, server: microsoft, request: "GET /c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe HTTP/1.1", upstream: "http://[2a01:111:2003::50]:80/c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe", host: "au.download.windowsupdate.com" 2017/04/03 18:31:38 [crit] 690#0: 3457 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.158, server: microsoft, request: "HEAD /c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe HTTP/1.1", upstream: "http://[2a01:111:2003::50]:80/c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe", host: "au.download.windowsupdate.com" 2017/04/03 18:31:41 [crit] 685#0: 3352 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.158, server: microsoft, request: "GET /d/msdownload/update/software/secu/2017/03/windows10.0-kb4013429-x64-express_f112f73709fe89085542b03bf2b0332ca2c3b9b8.cab HTTP/1.1", upstream: "http://[2a01:111:2003::50]:80/d/msdownload/update/software/secu/2017/03/windows10.0-kb4013429-x64-express_f112f73709fe89085542b03bf2b0332ca2c3b9b8.cab", host: "au.download.windowsupdate.com" 2017/04/03 18:31:43 [crit] 690#0: 3460 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.158, server: microsoft, request: "HEAD /c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe HTTP/1.1", upstream: "http://[2a01:111:2003::50]:80/c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe", host: "au.download.windowsupdate.com" 2017/04/03 18:31:43 [crit] 690#0: 3461 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.158, server: microsoft, request: "HEAD /c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe HTTP/1.1", upstream: "http://[2a01:111:2003::50]:80/c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe", host: "au.download.windowsupdate.com" 2017/04/03 18:31:44 [crit] 685#0: 3421 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.159, server: microsoft, request: "GET /c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe HTTP/1.1", upstream: "http://[2a01:111:2003::50]:80/c/msdownload/update/software/defu/2017/03/am_base_a831173843a3f8112c305c588effcd399968b87c.exe", host: "au.download.windowsupdate.com" 2017/04/03 18:31:53 [crit] 688#0: 3471 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.158, server: microsoft, request: "GET /d/msdownload/update/software/secu/2017/03/windows10.0-kb4013429-x64-express_f112f73709fe89085542b03bf2b0332ca2c3b9b8.cab HTTP/1.1", upstream: "http://[2a01:111:2003::50]:80/d/msdownload/update/software/secu/2017/03/windows10.0-kb4013429-x64-express_f112f73709fe89085542b03bf2b0332ca2c3b9b8.cab", host: "au.download.windowsupdate.com" 2017/04/03 18:31:53 [crit] 690#0: *3472 bind(10.0.4.21) failed (22: Invalid argument) while connecting to upstream, client: 10.0.4.158, server: microsoft, request: "GET /d/msdownload/update/software/secu/2017/03/windows10.0-kb4013429-x64-express_f112f73709fe89085542b03bf2b0332ca2c3b9b8.cab HTTP/1.1", upstream: "http://[2a01:111:2003::50]:80/d/msdownload/update/software/secu/2017/03/windows10.0-kb4013429-x64-express_f112f73709fe89085542b03bf2b0332ca2c3b9b8.cab", host: "au.download.windowsupdate.com"

nexusofdoom commented 7 years ago

new dns looks like new dns hosts needs to be added to the configs

Microsoft Windows Updates °|-lc-host-vint:7

so I added this to the unbound.conf and it --- still testing !!! local-zone: "fe2.update.microsoft.com." redirect local-data: "fe2.update.microsoft.com. 600 IN A 10.0.4.28" local-zone: "download.windowsupdate.com." redirect local-data: "download.windowsupdate.com. 600 IN A 10.0.4.28"

nexusofdoom commented 7 years ago

this is the ip that all the updates are coming from

13.107.4.50 it looks like a cdn for MS updates .. still trying to find out what the DNS name or cname is for this ip

nexusofdoom commented 7 years ago

more on this IP Address https://www.rtsak.com/dns-lookup/tlu.graystore.msedge.net

nexusofdoom commented 7 years ago

more on this IP 13.107.4.50

https://www.robtex.com/dns-lookup/7.tlu.dl.delivery.mp.microsoft.com

nexusofdoom commented 7 years ago

Names pointing to same IP address as this host name (45 shown) Which hostnames and domains point to the same IP address as this host name? download.windowsupdate.com au.au-msedge.net b1fgds.au-msedge.net b1fgv4.au-msedge.net b1ns.au-msedge.net b1w7dl.au-msedge.net b1w8au.au-msedge.net edges.au-msedge.net c-0001.c-msedge.net graystore.msedge.net 7.download.windowsupdate.com au.download.windowsupdate.com b1.download.windowsupdate.com www.download.windowsupdate.com tlu.graystore.au-msedge.net au.c-0001.c-msedge.net b1fgds.c-0001.c-msedge.net b1fgv4.c-0001.c-msedge.net b1ns.c-0001.c-msedge.net b1w7dl.c-0001.c-msedge.net b1w8au.c-0001.c-msedge.net graystore.c-0001.c-msedge.net 2-01-3cf7-0009.cdx.cedexis.net 2-01-3cf7-000c.cdx.cedexis.net 2-01-3cf7-000d.cdx.cedexis.net nontlu.graystore.msedge.net tlu.graystore.msedge.net 5.perf.msedge.net audownload.windowsupdate.nsatc.net dl.delivery.mp.microsoft.com au.b1.download.windowsupdate.com au.v4.download.windowsupdate.com au.www.download.windowsupdate.com 7.dl.delivery.mp.microsoft.com tlu.dl.delivery.mp.microsoft.com au.ds.b1.download.windowsupdate.com fg.ds.b1.download.windowsupdate.com aupl.v4.b1.download.windowsupdate.com b1.dl.windowsupdate.com.nsatc.net graystore.msedge.net.redirect.media.akamaihd.net au.b1.dl.windowsupdate.com.nsatc.net bg.v4.dl.windowsupdate.com.nsatc.net au.ds.b1.dl.windowsupdate.com.nsatc.net au.v4.b1.dl.windowsupdate.com.nsatc.net aupl.v4.b1.dl.windowsupdate.com.nsatc.net

bntjah commented 7 years ago

Let me know how it goes; I still haven't had the chance to set up my lancache as I'm debugging an error on my internet wich is causing random disconnects on bf1 :-S

nexusofdoom commented 7 years ago

So what I see on the client pc running Windows 10 is this wireshark flags reset set when that happens it ask to redownload and it looks in nginx in the data tmp folder.

On Apr 4, 2017 12:59 AM, "bntjah" notifications@github.com wrote:

Let me know how it goes; I still haven't had the chance to set up my lancache as I'm debugging an error on my internet wich is causing random disconnects on bf1 :-S

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bntjah/lancache/issues/37#issuecomment-291401233, or mute the thread https://github.com/notifications/unsubscribe-auth/APwRgeh0Ofl8qQh0rXelYyFjqbkK2LNSks5rsdwlgaJpZM4MvJqX .

nexusofdoom commented 7 years ago

http://serverfault.com/questions/679705/why-is-my-web-server-dropping-connections-with-a-tcp-reset-at-high-load

On Apr 4, 2017 7:27 AM, "Travus Elm" travus@discoverpc.net wrote:

So what I see on the client pc running Windows 10 is this wireshark flags reset set when that happens it ask to redownload and it looks in nginx in the data tmp folder.

On Apr 4, 2017 12:59 AM, "bntjah" notifications@github.com wrote:

Let me know how it goes; I still haven't had the chance to set up my lancache as I'm debugging an error on my internet wich is causing random disconnects on bf1 :-S

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bntjah/lancache/issues/37#issuecomment-291401233, or mute the thread https://github.com/notifications/unsubscribe-auth/APwRgeh0Ofl8qQh0rXelYyFjqbkK2LNSks5rsdwlgaJpZM4MvJqX .

nexusofdoom commented 7 years ago

http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream

On Apr 4, 2017 8:13 AM, "Travus Elm" travus@discoverpc.net wrote:

http://serverfault.com/questions/679705/why-is-my-web-server-dropping- connections-with-a-tcp-reset-at-high-load

On Apr 4, 2017 7:27 AM, "Travus Elm" travus@discoverpc.net wrote:

So what I see on the client pc running Windows 10 is this wireshark flags reset set when that happens it ask to redownload and it looks in nginx in the data tmp folder.

On Apr 4, 2017 12:59 AM, "bntjah" notifications@github.com wrote:

Let me know how it goes; I still haven't had the chance to set up my lancache as I'm debugging an error on my internet wich is causing random disconnects on bf1 :-S

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bntjah/lancache/issues/37#issuecomment-291401233, or mute the thread https://github.com/notifications/unsubscribe-auth/APwRgeh0Ofl8qQh0rXelYyFjqbkK2LNSks5rsdwlgaJpZM4MvJqX .

nexusofdoom commented 7 years ago

so at this point disabled proxybind in nginx config for mswindows and this is the error I am getting now

2017/04/04 14:14:31 [error] 859#0: 1 connect() to [2001:1900:2302:6f01::ff]:80 failed (101: Network is unreachable) while connecting to upstream, client: 192.168.30.211, server: microsoft, request: "GET /c/msdownload/update/sof tware/secu/2017/03/windows6.1-kb4012215-x64_31cc0bb569162c3ec6c1f992476f1ad9f58e0f16.psf HTTP/1.1", upstream: "http://[2001:1900:2302:6f01::ff]:80/c/msdownload/update/software/secu/2017/03/windows6.1-kb4012215-x64_31cc0bb569162c 3ec6c1f992476f1ad9f58e0f16.psf", host: "download.windowsupdate.com" 2017/04/04 14:14:31 [error] 859#0: 1 connect() to [2001:1900:2304:8f05::ff]:80 failed (101: Network is unreachable) while connecting to upstream, client: 192.168.30.211, server: microsoft, request: "GET /c/msdownload/update/sof tware/secu/2017/03/windows6.1-kb4012215-x64_31cc0bb569162c3ec6c1f992476f1ad9f58e0f16.psf HTTP/1.1", upstream: "http://[2001:1900:2304:8f05::ff]:80/c/msdownload/update/software/secu/2017/03/windows6.1-kb4012215-x64_31cc0bb569162c 3ec6c1f992476f1ad9f58e0f16.psf", host: "download.windowsupdate.com" 2017/04/04 14:14:33 [error] 859#0: 1 connect() to [2001:1900:2302:2f05::ff]:80 failed (101: Network is unreachable) while connecting to upstream, client: 192.168.30.211, server: microsoft, request: "GET /c/msdownload/update/sof tware/secu/2017/03/windows6.1-kb4012215-x64_31cc0bb569162c3ec6c1f992476f1ad9f58e0f16.psf HTTP/1.1", upstream: "http://[2001:1900:2302:2f05::ff]:80/c/msdownload/update/software/secu/2017/03/windows6.1-kb4012215-x64_31cc0bb569162c 3ec6c1f992476f1ad9f58e0f16.psf", host: "download.windowsupdate.com" 2017/04/04 14:14:38 [error] 859#0: 1 connect() to [2001:1900:2304:8f0c::ff]:80 failed (101: Network is unreachable) while connecting to upstream, client: 192.168.30.211, server: microsoft, request: "GET /c/msdownload/update/sof tware/secu/2017/03/windows6.1-kb4012215-x64_31cc0bb569162c3ec6c1f992476f1ad9f58e0f16.psf HTTP/1.1", upstream: "http://[2001:1900:2304:8f0c::ff]:80/c/msdownload/update/software/secu/2017/03/windows6.1-kb4012215-x64_31cc0bb569162c 3ec6c1f992476f1ad9f58e0f16.psf", host: "download.windowsupdate.com" 2017/04/04 14:14:38 [error] 859#0: *1 connect() to [2001:1900:2302:2f05::ff]:80 failed (101: Network is unreachable) while connecting to upstream, client: 192.168.30.211, server: microsoft, request: "GET /c/msdownload/update/sof tware/secu/2017/03/windows6.1-kb4012215-x64_31cc0bb569162c3ec6c1f992476f1ad9f58e0f16.psf HTTP/1.1", upstream: "http://[2001:1900:2302:2f05::ff]:80/c/msdownload/update/software/secu/2017/03/windows6.1-kb4012215-x64_31cc0bb569162c 3ec6c1f992476f1ad9f58e0f16.psf", host: "download.windowsupdate.com"

nexusofdoom commented 7 years ago

ok did this and testing now

Re: disable ipV6 solves lookup slowness for Debian but not for Ubuntu sysctl net.ipv6.conf.all.disable_ipv6=1 ref https://ubuntuforums.org/showthread.php?t=1668964

wendelj commented 7 years ago

I missed where the resolver was set in a second place. /usr/local/nginx/conf/lancache/resolver

Change this: resolver 8.8.8.8 4.2.2.2;

To this: resolver 8.8.8.8 4.2.2.2 ipv6=off;

nexusofdoom commented 7 years ago

I did find the other location for ipv6=off in lancache resolver and did the change but it was still looking up ipv6

the errors only stopped when I did the command sudo sysctl net.ipv6.conf.all.disable_ipv6=1

bntjah commented 7 years ago

@nexusofdoom did this actually solve the issue? If so we'll add it to the readme.md & installer but just want to make sure that this resolved the problem

2017-04-05 13:19 GMT+02:00 nexusofdoom notifications@github.com:

I did find the other location for ipv6=off in lancache resolver and did the change but it was still looking up ipv6

the errors only stopped when I did the command sudo sysctl net.ipv6.conf.all.disable_ipv6=1

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bntjah/lancache/issues/37#issuecomment-291830788, or mute the thread https://github.com/notifications/unsubscribe-auth/AGE_0fdCGP8R0r0_rM-W9_zDqdbcUH3oks5rs3i_gaJpZM4MvJqX .

nexusofdoom commented 7 years ago

still doing some tweaks for windows 10 and testing and testing :-)

this is what i have so for for nginx settings

Windows Updates

server { listen lancache-microsoft bind ipv6only=off deferred default; servername microsoft ; access_log /srv/lancache/logs/Access/microsoft.log main buffer=128k flush=1m; access_log /srv/lancache/logs/Keys/microsoft.log keys_range buffer=128k flush=1m; error_log /srv/lancache/logs/Errors/microsoft.log;

    # Default Node
    include lancache/resolver;
    include lancache/cache-key-default;
location / {
     set $no_cache "";
    if ($request_uri ~* \.filestreamingservice$) {
        set $no_cache "1";
    }
    proxy_no_cache $no_cache;
    proxy_cache_bypass $no_cache;
    proxy_connect_timeout 600;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Host $http_host;
    # Some downloads are very large so we cache based on range to keep single downloads quick and hence ensure interactivity is good.
    #proxy_bind 192.168.30.99;
    proxy_cache_valid 200 206 60d;
    proxy_cache microsoft;
    proxy_read_timeout 150;
    include lancache/proxy-cache;
    include lancache/cache-range;
}

}

once I am all done I will attache a tar with all the changes

nexusofdoom commented 7 years ago

trying to get this folder to not cache

set $no_cache ""; if ($request_uri ~* .filestreamingservice$) { set $no_cache "1"; } proxy_no_cache $no_cache; proxy_cache_bypass $no_cache;

something that windows 10 is doing with updates i think its something like a update list it pushes and it pulls it down over and over

there might also be a way to block it so still checking in ref url below

ref. https://social.technet.microsoft.com/Forums/en-US/e9531028-4bfe-40c0-af5c-8873c42b6bae/massive-download?forum=win10itprogeneral

nexusofdoom commented 7 years ago

more fun :-)

https://answers.microsoft.com/en-us/insider/forum/insider_wintp-insider_update/windows10-windows-automatic-update-and-store/17f48b91-c12c-41de-8c3a-63954cd40639

Hello, Yes, the URL tlu.dl.delivery.mp.microsoft.com is from the app store. I had the same problem with high bandwidth usage. In my case it was caused because the proxy (privoxy) was filtering the HTTP Range header from the HTTP requests. This caused the app store download everything in an endless loop. Maybe your proxy is doing something similar? Be the first person to like this

nexusofdoom commented 7 years ago

so don't know if i need to remove this uri from caching? URI="/filestreamingservice/files/ --- so anything in this folder to not cache ..

any one know how I could do this ? Thanks

REQUEST="GET /filestreamingservice/files/3787205b-0105-4bfb-a64e-985d61a84e3b?P1=1491419171&P2=301&P3=2&P4=7L1llVny%2fZSfMprTfHkBkhZ0hiELzvopzaXFgiJ%2frIU%3d HTTP/1.1" - REQUEST_URI="/filestreamingservice/files/3787205b-0105-4bfb-a64e-985d61a84e3b?P1=1491419171&P2=301&P3=2&P4=7L1llVny%2fZSfMprTfHkBkhZ0hiELzvopzaXFgiJ%2frIU%3d" - URI="/filestreamingservice/files/3787205b-0105-4bfb-a64e-985d61a84e3b"

wendelj commented 7 years ago

After setting ipv6=off for both resolver definitions and restarting nginx I was able to update a fresh install of Windows 10 without seeing those errors about Invalid Argument.

nexusofdoom commented 7 years ago

current setup

Windows Updates

server { listen lancache-microsoft bind ipv6only=off deferred default; servername microsoft ; access_log /srv/lancache/logs/Access/microsoft.log main buffer=128k flush=1m; access_log /srv/lancache/logs/Keys/microsoft.log keys_range buffer=128k flush=1m; error_log /srv/lancache/logs/Errors/microsoft.log;

    # Default Node
    include lancache/resolver;
    include lancache/cache-key-default;
location / {
    proxy_connect_timeout 300;
    set $no_cache 0;
    if ($request_uri ~* "(filestreamingservice/files)") {
        set $no_cache 1;
    }
    proxy_no_cache $no_cache;
    proxy_cache_bypass $no_cache;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Host $http_host;
    # Some downloads are very large so we cache based on range to keep single downloads quick and hence ensure interactivity is good.
    #proxy_bind 192.168.30.99;
    #proxy_cache_valid 200 206 120d;
    proxy_cache microsoft;
    proxy_read_timeout 150;
    include lancache/proxy-cache;
    include lancache/cache-range;
}

}

Microsoft Windows Updates °|-lc-host-vint:7

    local-zone: "windowsupdate.com." redirect
    local-data: "windowsupdate.com. 600 IN A 192.168.30.106"
    #local-zone: "dlassets.xboxlive.com." redirect
    #local-data: "dlassets.xboxlive.com. 600 IN A 192.168.30.106"
    #local-zone: "msxbassets.loris.llnwd.net." redirect
    #local-data: "msxbassets.loris.llnwd.net. 600 IN A 192.168.30.106"
    #local-zone: "xboxone.loris.llnwd.net." redirect
    #local-data: "xboxone.loris.llnwd.net. 600 IN A 192.168.30.106"
    #local-zone: "xboxone.vo.llnwd.net." redirect
    #local-data: "xboxone.vo.llnwd.net. 600 IN A 192.168.30.106"
    #local-zone: "images-eds.xboxlive.com." redirect
    #local-data: "images-eds.xboxlive.com. 600 IN A 192.168.30.106"
    #local-zone: "xbox-mbr.xboxlive.com." redirect
    #local-data: "xbox-mbr.xboxlive.com. 600 IN A 192.168.30.106"
    #local-zone: "assets1.xboxlive.com.nsatc.net." redirect
    #local-data: "assets1.xboxlive.com.nsatc.net. 600 IN A 192.168.30.106"
    #local-zone: "assets1.xboxlive.com." redirect
    #local-data: "assets1.xboxlive.com. 600 IN A 192.168.30.106"
    local-zone: "audownload.windowsupdate.nsatc.net." redirect
    local-data: "audownload.windowsupdate.nsatc.net. 600 IN A 192.168.30.106"
    local-zone: "windowsupdate.com.nsatc.net." redirect
    local-data: "windowsupdate.com.nsatc.net. 600 IN A 192.168.30.106"
    #local-zone: "delivery.mp.microsoft.com." redirect
    #local-data: "delivery.mp.microsoft.com. 600 IN A 192.168.30.106"
    local-zone: "graystore.msedge.net.redirect.media.akamaihd.net." redirect
    local-data: "graystore.msedge.net.redirect.media.akamaihd.net. 600 IN A 192.168.30.106"
    local-zone: "msedge.net." redirect
    local-data: "msedge.net. 600 IN A 192.168.30.106"
    local-zone: "c-msedge.net." redirect
    local-data: "c-msedge.net. 600 IN A 192.168.30.106"
    local-zone: "au-msedge.net." redirect
    local-data: "au-msedge.net. 600 IN A 192.168.30.106"

also did keep the ipv6=off in both configs -- and also have a "bind ipv6only=off" on the top of the lancahce-microsoft.conf

nexusofdoom commented 7 years ago

wendelj when you do all the windows updates

if you are looking at nload do you still see it pulling down data.. what I am getting is it keeps pulling down data once all the windows updates are done and it does not show it caching

image

nexusofdoom commented 7 years ago

if I use all the defaults along with all the ipv6 disabled not every update for windows comes over the lancache it routes over the internet direct

nexusofdoom commented 7 years ago

going to attache wireshark logs this image is showing it pulling from another location not just from MS domains. some type of cdn or other locations I do see that it pulls some updates from adobe in the windows update but there is some other windows updates getting pulled from another domain image

nexusofdoom commented 7 years ago

dns wireshark file csv on pc that was doing windows updates

DNS.txt

nexusofdoom commented 7 years ago

might be how windows 10 does the updates https://community.spiceworks.com/topic/1400908-what-is-being-delivered-from-these-microsoft-hosts

nexusofdoom commented 7 years ago

where I am at now

is this error messages

2017/04/06 15:51:29 [error] 551#0: 2160 upstream prematurely closed connection while reading upstream, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_4_867ccb99f9ecf836844f1567a25db6f91d6070c3.psf HTTP/1.1", upstream: "http://8.253.45.248:80/d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_4_867ccb99f9ecf836844f1567a25db6f91d6070c3.psf", host: "download.windowsupdate.com" 2017/04/06 15:51:29 [alert] 551#0: 2160 header already sent while sending to client, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_4_867ccb99f9ecf836844f1567a25db6f91d6070c3.psf HTTP/1.1", upstream: "http://8.253.45.248:80/d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_4_867ccb99f9ecf836844f1567a25db6f91d6070c3.psf", host: "download.windowsupdate.com" 2017/04/06 15:51:41 [error] 547#0: 1146 upstream prematurely closed connection while reading upstream, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_3_aadae06fd0885db7996e702e453f784b4683d978.psf HTTP/1.1", upstream: "http://8.253.45.248:80/d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_3_aadae06fd0885db7996e702e453f784b4683d978.psf", host: "download.windowsupdate.com" 2017/04/06 15:51:41 [alert] 547#0: 1146 header already sent while sending to client, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_3_aadae06fd0885db7996e702e453f784b4683d978.psf HTTP/1.1", upstream: "http://8.253.45.248:80/d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_3_aadae06fd0885db7996e702e453f784b4683d978.psf", host: "download.windowsupdate.com" 2017/04/06 15:51:46 [error] 549#0: 2374 upstream prematurely closed connection while reading upstream, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_5_c03a4a4111de297c7c400c8e59af9fb04a5f4be1.psf HTTP/1.1", upstream: "http://8.253.45.248:80/d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_5_c03a4a4111de297c7c400c8e59af9fb04a5f4be1.psf", host: "download.windowsupdate.com" 2017/04/06 15:51:46 [alert] 549#0: 2374 header already sent while sending to client, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_5_c03a4a4111de297c7c400c8e59af9fb04a5f4be1.psf HTTP/1.1", upstream: "http://8.253.45.248:80/d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_5_c03a4a4111de297c7c400c8e59af9fb04a5f4be1.psf", host: "download.windowsupdate.com" 2017/04/06 15:51:58 [error] 547#0: 1143 upstream prematurely closed connection while reading upstream, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_4_867ccb99f9ecf836844f1567a25db6f91d6070c3.psf HTTP/1.1", upstream: "http://8.253.110.112:80/d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_4_867ccb99f9ecf836844f1567a25db6f91d6070c3.psf", host: "download.windowsupdate.com" 2017/04/06 15:51:58 [alert] 547#0: 1143 header already sent while sending to client, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_4_867ccb99f9ecf836844f1567a25db6f91d6070c3.psf HTTP/1.1", upstream: "http://8.253.110.112:80/d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_4_867ccb99f9ecf836844f1567a25db6f91d6070c3.psf", host: "download.windowsupdate.com" 2017/04/06 15:53:07 [error] 547#0: 1138 upstream prematurely closed connection while reading upstream, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_2_4d9901b63ecc4abfe32a6bb352d3312d26eef527.psf HTTP/1.1", upstream: "http://8.253.45.248:80/d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_2_4d9901b63ecc4abfe32a6bb352d3312d26eef527.psf", host: "download.windowsupdate.com" 2017/04/06 15:53:07 [alert] 547#0: 1138 header already sent while sending to client, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_2_4d9901b63ecc4abfe32a6bb352d3312d26eef527.psf HTTP/1.1", upstream: "http://8.253.45.248:80/d/msdownload/update/software/updt/2017/03/windows10.0-kb4015438-x64_2_4d9901b63ecc4abfe32a6bb352d3312d26eef527.psf", host: "download.windowsupdate.com"

nexusofdoom commented 7 years ago

http://stackoverflow.com/questions/23844761/upstream-sent-too-big-header-while-reading-response-header-from-upstream

for this type of error alert] 547#0: 1138 header already sent while sending to client, 2017/04/06 16:15:23 [error] 548#0: 4286 readv() failed (104: Connection reset by peer) while reading upstream, client: 192.168.30.178, server: microsoft, request: "GET /c/Upgr/2017/03/15063.0.170317-1834.rs2_release_clientcombinedsl_ret_x64fre_en-us_64317f9f897ab3cab7e45cbcafd139d30396c81f.esd HTTP/1.1", upstream: "http://13.107.4.50:80/c/Upgr/2017/03/15063.0.170317-1834.rs2_release_clientcombinedsl_ret_x64fre_en-us_64317f9f897ab3cab7e45cbcafd139d30396c81f.esd", host: "fg.ds.b1.download.windowsupdate.com"

travispk commented 7 years ago

I'm glad you guys are working on this issue. I had an event last weekend with 500 attendees. LANCache worked awesome for everything but Arena Networks and Microsoft. I had to disable DNS redirects for those and had no time to troubleshoot. Arena Networks was just not working, and Microsoft was causing LANCache to just bury our Internet connection and fill the ./tmp folder with files.

nexusofdoom commented 7 years ago

it looks like its just winodws 10 that is making it download over and over and fill up the tmp folder

nexusofdoom commented 7 years ago

2017/04/06 15:50:23 [alert] 551#0: 2223 header already sent while sending to client, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10$ 2017/04/06 15:50:27 [error] 551#0: 2163 upstream prematurely closed connection while reading upstream, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt$ 2017/04/06 15:50:27 [alert] 551#0: 2163 header already sent while sending to client, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10$ 2017/04/06 15:50:36 [error] 547#0: 1157 upstream prematurely closed connection while reading upstream, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt$ 2017/04/06 15:50:36 [alert] 547#0: 1157 header already sent while sending to client, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10$ 2017/04/06 15:50:39 [error] 548#0: 1433 upstream prematurely closed connection while reading upstream, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt$ 2017/04/06 15:50:39 [alert] 548#0: 1433 header already sent while sending to client, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10$ 2017/04/06 15:50:40 [error] 551#0: 2599 upstream prematurely closed connection while reading upstream, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt$ 2017/04/06 15:50:40 [alert] 551#0: 2599 header already sent while sending to client, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10$ 2017/04/06 15:51:01 [error] 547#0: 1154 upstream prematurely closed connection while reading upstream, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt$ 2017/04/06 15:51:01 [alert] 547#0: 1154 header already sent while sending to client, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10$ 2017/04/06 15:51:03 [error] 551#0: 2226 upstream prematurely closed connection while reading upstream, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt$ 2017/04/06 15:51:03 [alert] 551#0: 2226 header already sent while sending to client, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10$ 2017/04/06 15:51:09 [error] 547#0: 1150 upstream prematurely closed connection while reading upstream, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt$ 2017/04/06 15:51:09 [alert] 547#0: 1150 header already sent while sending to client, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10$ 2017/04/06 15:51:13 [error] 551#0: 2228 upstream prematurely closed connection while reading upstream, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt$ 2017/04/06 15:51:13 [alert] 551#0: 2228 header already sent while sending to client, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10$ 2017/04/06 15:51:14 [error] 552#0: 1241 upstream prematurely closed connection while reading upstream, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt$ 2017/04/06 15:51:14 [alert] 552#0: 1241 header already sent while sending to client, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10$ 2017/04/06 15:51:14 [error] 547#0: 1139 upstream prematurely closed connection while reading upstream, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt$ 2017/04/06 15:51:14 [alert] 547#0: 1139 header already sent while sending to client, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10$ 2017/04/06 15:51:29 [error] 551#0: 2160 upstream prematurely closed connection while reading upstream, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt$ 2017/04/06 15:51:29 [alert] 551#0: 2160 header already sent while sending to client, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10$ 2017/04/06 15:51:41 [error] 547#0: 1146 upstream prematurely closed connection while reading upstream, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt$ 2017/04/06 15:51:41 [alert] 547#0: 1146 header already sent while sending to client, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10$ 2017/04/06 15:51:46 [error] 549#0: 2374 upstream prematurely closed connection while reading upstream, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt$ 2017/04/06 15:51:46 [alert] 549#0: 2374 header already sent while sending to client, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10$ 2017/04/06 15:51:58 [error] 547#0: 1143 upstream prematurely closed connection while reading upstream, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt$ 2017/04/06 15:51:58 [alert] 547#0: 1143 header already sent while sending to client, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10$ 2017/04/06 15:53:07 [error] 547#0: 1138 upstream prematurely closed connection while reading upstream, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt$ 2017/04/06 15:53:07 [alert] 547#0: 1138 header already sent while sending to client, client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2017/03/windows10$ 2017/04/06 16:05:04 [error] 548#0: 4270 readv() failed (104: Connection reset by peer) while reading upstream, client: 192.168.30.178, server: microsoft, request: "GET /c/Upgr/2017/03/15063.0.170$ 2017/04/06 16:15:23 [error] 548#0: *4286 readv() failed (104: Connection reset by peer) while reading upstream, client: 192.168.30.178, server: microsoft, request: "GET /c/Upgr/2017/03/15063.0.170$

wendelj commented 7 years ago

I switched the Microsoft cache over to Slice instead of Range and it stopped the constant download issue with Windows 10.

nexusofdoom commented 7 years ago

Do you have the changes you made ? So I can test out on my sever thanks.

On Apr 6, 2017 10:38 PM, "Jason Wendel" notifications@github.com wrote:

I switched the Microsoft cache over to Slice instead of Range and it stopped the constant download issue with Windows 10.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bntjah/lancache/issues/37#issuecomment-292421643, or mute the thread https://github.com/notifications/unsubscribe-auth/APwRgc4jlL-cw_AluQWTVHUNcyetGh1rks5rta-tgaJpZM4MvJqX .

nexusofdoom commented 7 years ago

wendelj so did you just change this ?

old include lancache/proxy-cache; include lancache/cache-range; }

new

include lancache/proxy-cache; include lancache/cache-slice; }

or can you attacke your config thanks

also do you have any more dns names in unbound.conf ?

nexusofdoom commented 7 years ago

after changing from lancache/cache-range to slice it looks like working better but on some updates that it pulls down its not caching ? do we need to add more dns names to unbound.conf ? p_20170406_181854_1_p

this is what I have in my unbound.conf p_20170407_131616_1_p

some of the errors

2017/04/07 14:04:49 [error] 865#0: 6157 download.windowsupdate.com could not be resolved (110: Operation timed out), client: 192.168.30.178, server: microsoft, request: "GET /c/msdownload/update/software/updt/2013/06/windows6.1-kb2808679-x64_09ca2df24bdf9829ed87c03657c381a16d629d89.psf HTTP/1.1", host: "download.w$ 2017/04/07 14:04:54 [error] 866#0: 6035 au.download.windowsupdate.com could not be resolved (110: Operation timed out), client: 192.168.30.178, server: microsoft, request: "HEAD /msdownload/update/software/ftpk/2011/08/dotnetfx40_client_x86_x64_a96cef357b16f0dea3423bf6c515a96a3c2233de.exe HTTP/1.1", host: "au.down$ 2017/04/07 14:04:54 [error] 865#0: 5919 download.windowsupdate.com could not be resolved (110: Operation timed out), client: 192.168.30.178, server: microsoft, request: "GET /msdownload/update/software/secu/2012/12/windows6.1-kb2742599-x64_f89a680cd53a05216dea658036a3385eab4a2812.psf HTTP/1.1", host: "download.win$ 2017/04/07 14:04:54 [error] 865#0: 5892 download.windowsupdate.com could not be resolved (110: Operation timed out), client: 192.168.30.178, server: microsoft, request: "GET /d/msdownload/update/software/updt/2016/03/windows6.1-kb3147071-x64_18992bd5fa0424684c4872e1c10f5663b6d3aa52.psf HTTP/1.1", host: "download.w$ 2017/04/07 14:04:54 [error] 865#0: 6157 download.windowsupdate.com could not be resolved (110: Operation timed out), client: 192.168.30.178, server: microsoft, request: "GET /c/msdownload/update/software/updt/2013/06/windows6.1-kb2808679-x64_09ca2df24bdf9829ed87c03657c381a16d629d89.psf HTTP/1.1", host: "download.w$ 2017/04/07 14:04:54 [error] 867#0: 5773 upstream timed out (110: Connection timed out) while connecting to upstream, client: 192.168.30.178, server: microsoft, request: "GET /msdownload/update/software/secu/2011/03/windows6.1-kb2509553-x64_3cd4024668e5f505956b53ade86485ba322cb893.psf HTTP/1.1", upstream: "http://8$ 2017/04/07 14:04:54 [error] 867#0: 5743 download.windowsupdate.com could not be resolved (110: Operation timed out), client: 192.168.30.178, server: microsoft, request: "GET /msdownload/update/software/crup/2011/03/windows6.1-kb2506014-x64_0c57cb4a8c84d1931f8a80b419463819e5832532.psf HTTP/1.1", host: "download.win$ 2017/04/07 14:04:55 [error] 868#0: 5941 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.30.178, server: microsoft, request: "GET /c/msdownload/update/software/secu/2014/06/windows6.1-kb2973351-x64_b550d0384dda837a0b3f543f62f0bd1851082498.psf HTTP/1.1", ups$ 2017/04/07 14:04:58 [error] 865#0: 5935 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.30.178, server: microsoft, request: "GET /d/msdownload/update/software/secu/2015/11/windows6.1-kb3109103-x64_66e00af753e3faae5d558534711af7dc29a9160d.psf HTTP/1.1", up$ 2017/04/07 14:04:58 [error] 865#0: 5937 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.30.178, server: microsoft, request: "GET /c/msdownload/update/software/updt/2015/03/windows6.1-kb3013531-v2-x64_5ab9d694766b788f6be938b08fbcc1e0013d78c6.psf HTTP/1.1",$ 2017/04/07 14:04:59 [error] 866#0: 6027 au.download.windowsupdate.com could not be resolved (110: Operation timed out), client: 192.168.30.178, server: microsoft, request: "HEAD /d/msdownload/update/software/updt/2015/01/windows6.1-kb3021917-x64-express_7a8a3134f9fe596b652870942ace28f5249d7298.cab HTTP/1.1", host:$ 2017/04/07 14:04:59 [error] 865#0: 5919 download.windowsupdate.com could not be resolved (110: Operation timed out), client: 192.168.30.178, server: microsoft, request: "GET /d/msdownload/update/software/updt/2016/03/windows6.1-kb3147071-x64_18992bd5fa0424684c4872e1c10f5663b6d3aa52.psf HTTP/1.1", host: "download.w$ 2017/04/07 14:04:59 [error] 866#0: 5691 upstream timed out (110: Connection timed out) while connecting to upstream, client: 192.168.30.178, server: microsoft, request: "GET /c/msdownload/update/software/secu/2015/01/windows6.1-kb3004375-v3-x64_690d03c38a087b846ac2e6a2e5ad4383413420fe.psf HTTP/1.1", upstream: "htt$ 2017/04/07 14:05:00 [error] 865#0: 5899 download.windowsupdate.com could not be resolved (110: Operation timed out), client: 192.168.30.178, server: microsoft, request: "GET /d/msdownload/update/software/secu/2013/05/windows6.1-kb2813430-x64_c6b25c32a059874e89f6197c35aa12b75a7dd6d4.psf HTTP/1.1", host: "download.w$ 2017/04/07 14:05:00 [error] 865#0: 5892 download.windowsupdate.com could not be resolved (110: Operation timed out), client: 192.168.30.178, server: microsoft, request: "GET /c/msdownload/update/software/secu/2015/09/windows6.1-kb3093513-x64_02d98c20d0f54b3543ca542f6482960f1db2666c.psf HTTP/1.1", host: "download.w$ 2017/04/07 14:05:01 [error] 865#0: 6157 download.windowsupdate.com could not be resolved (110: Operation timed out), client: 192.168.30.178, server: microsoft, request: "GET /c/msdownload/update/software/secu/2014/05/windows6.1-kb2957189-x64_cc41fc5982c710162eb834bb7537d58cac2cb8dd.psf HTTP/1.1", host: "download.w$ 2017/04/07 14:05:03 [error] 866#0: 5692 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.30.178, server: microsoft, request: "GET /d/msdownload/update/software/secu/2015/11/windows6.1-kb3109103-x64_66e00af753e3faae5d558534711af7dc29a9160d.psf HTTP/1.1", upstream: "http://8.254$ 2017/04/07 14:05:03 [error] 867#0: 5772 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.30.178, server: microsoft, request: "GET /msdownload/update/software/updt/2012/11/windows6.1-kb2750841-x64_4e9cebb7c3b6684b7a83d69ccd6f798ec4d07d17.psf HTTP/1.1", upstream: "http://8.254.2$ 2017/04/07 14:05:07 [error] 865#0: 5919 download.windowsupdate.com could not be resolved (110: Operation timed out), client: 192.168.30.178, server: microsoft, request: "GET /d/msdownload/update/software/secu/2013/05/windows6.1-kb2813430-x64_c6b25c32a059874e89f6197c35aa12b75a7dd6d4.psf HTTP/1.1", host: "download.w$ 2017/04/07 14:05:07 [error] 865#0: 5899 download.windowsupdate.com could not be resolved (110: Operation timed out), client: 192.168.30.178, server: microsoft, request: "GET /msdownload/update/software/secu/2012/12/windows6.1-kb2742599-x64_f89a680cd53a05216dea658036a3385eab4a2812.psf HTTP/1.1", subrequest: "/msdow$ 2017/04/07 14:05:07 [error] 865#0: *5899 unexpected status code 502 in slice response whi

wendelj commented 7 years ago

Here's my current config for the Microsoft cache:

Windows Updates

server { listen lancache-microsoft deferred default; servername microsoft ; access_log /srv/lancache/logs/Access/microsoft.log main buffer=128k flush=1m; ?? access_log /srv/lancache/logs/Keys/microsoft.log keys_range buffer=128k flush=1m;

looking this over does keys need to be slice also?

error_log /srv/lancache/logs/Errors/microsoft.log;
    # Default Node
    include lancache/resolver;
location / {
    # Some downloads are very large so we cache based on range to keep single downloads quick and hen$
    proxy_bind lc-host-proxybind;
    proxy_cache_valid 206 90d;
    proxy_cache microsoft;
    proxy_read_timeout 150;
    include lancache/proxy-cache;
    include lancache/cache-slice;
}

}

Tested with back to back runs of Windows 10 updates. Everything hit the cache and no errors written to the microsoft cache error log. Probably need to make sure this still works okay for Windows 7 and 8. Haven't done that testing yet.

Edited to uncomment the proxy_cache_valid 206 line.

nexusofdoom commented 7 years ago

can I get a copy of your unbound config for Microsoft?

I will post my configs next week still testing with windows 7,8.1 and 10 so far so good see how it goes..

thanks for the help

On Apr 8, 2017 11:55 AM, "Jason Wendel" notifications@github.com wrote:

Here's my current config for the Microsoft cache:

Windows Updates

server { listen lancache-microsoft deferred default; servername microsoft ; access_log /srv/lancache/logs/Access/microsoft.log main buffer=128k flush=1m; access_log /srv/lancache/logs/Keys/microsoft.log keys_range buffer=128k flush=1m; error_log /srv/lancache/logs/Errors/microsoft.log;

Default Node

include lancache/resolver; location / {

Some downloads are very large so we cache based on range to keep single

downloads quick and hen$ proxy_bind lc-host-proxybind;

proxy_cache_valid 206 90d;

proxy_cache microsoft; proxy_read_timeout 150; include lancache/proxy-cache; include lancache/cache-slice; } }

Tested with back to back runs of Windows 10 updates. Everything hit the cache. Probably need to make sure this still works okay for Windows 7 and

  1. Haven't done that testing yet.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bntjah/lancache/issues/37#issuecomment-292730671, or mute the thread https://github.com/notifications/unsubscribe-auth/APwRgdQHk-iehypiee_ZpkvB37hXna9Nks5rt7wagaJpZM4MvJqX .

wendelj commented 7 years ago

I didn't change the config for unbound. Using Wireshark I was able to verify >99% of Bytes transferred went through the cache while updating Windows 10. The next largest by bytes was 2.5 MB from emdl.ws.microsoft.com to download some JSON files. I'm not too concerned about not caching those little files.

Ran the Win 10 update test a few more times, it does sporadically get a cache miss on a file here and there for some reason.

nexusofdoom commented 7 years ago

this is what I have in my unbound file for Microsoft

Microsoft Windows Updates °|-lc-host-vint:7

    local-zone: "windowsupdate.com." redirect
    local-data: "windowsupdate.com. 600 IN A 10.0.4.28"
    local-zone: "do.dsp.mp.microsoft.com." redirect
    local-data: "do.dsp.mp.microsoft.com. 600 IN A 10.0.4.28"
    local-zone: "v10.vortex-win.data.microsoft.com." redirect
    local-data: "v10.vortex-win.data.microsoft.com. 600 IN A 10.0.4.28"
    local-zone: "update.microsoft.com." redirect
    local-data: "update.microsoft.com. 600 IN A 10.0.4.28"
    local-zone: "download.microsoft.com." redirect
    local-data: "download.microsoft.com. 600 IN A 10.0.4.28"
    local-zone: "go.microsoft.com." redirect
    local-data: "go.microsoft.com. 600 IN A 10.0.4.28"

tested with windows 7 , 8.1 and windows 10 along with file downloads from microsoft.com like direct KB updates take come down from go.microsoft.com url.

unbound caching the name lookup timeouts tim? can we keep resent lookup's so it does not have to re lookup dns names over and over?

Minimum lifetime of cache entries in seconds. Default is 0.

cache-min-ttl: 60

Maximum lifetime of cached entries. Default is 86400 seconds (1 day).

cache-max-ttl: 172800

what about tcp and udp ?

do-udp: yes do-tcp: yes

what about this in nginx config ref url https://www.nginx.com/blog/tuning-nginx/ keepalive_requests – The number of requests a client can make over a single keepalive connection. The default is 100, but a much higher value can be especially useful for testing with a load-generation tool, which generally sends a large number of requests from a single client.

The following directive relates to upstream keepalives:

keepalive – The number of idle keepalive connections to an upstream server that remain open for each worker process. There is no default value. To enable keepalive connections to upstream servers you must also include the following directives in the configuration:

proxy_http_version 1.1; proxy_set_header Connection "";

nexusofdoom commented 7 years ago

unbound setting so far

Microsoft Windows Updates °|-lc-host-vint:7

    local-zone: "download.windowsupdate.com." redirect
    local-data: "download.windowsupdate.com. 600 IN A x.x.x.x"
nexusofdoom commented 7 years ago

lancache config ms office

Windows Updates

server { listen lancache-microsoft deferred default; servername microsoft ; access_log /srv/lancache/logs/Access/microsoft.log main buffer=128k flush=1m; access_log /srv/lancache/logs/Keys/microsoft.log keys_range buffer=128k flush=1m; error_log /srv/lancache/logs/Errors/microsoft.log;

    # Default Node
    include lancache/resolver;
    include lancache/cache-key-default;
location / {
    #proxy_send_timeout 300s;
    proxy_connect_timeout 300;
    # Some downloads are very large so we cache based on range to keep single downloads quic$
    proxy_bind 192.168.30.99;
    #proxy_cache_valid 200 400 206 320d;
    proxy_cache microsoft;
    proxy_read_timeout 300;
    include lancache/proxy-cache;
    include lancache/cache-slice;
}

}

nexusofdoom commented 7 years ago

when getting new updates I get this error when using slice.. testing again with range and some other settings.

8a385923ff85ddc95ecd75ea41.psf HTTP/1.1", upstream: "http://8.254.220.190:80/c/msdownload/update/software/secu/2017/03/windows10.0-kb4013198-x64_1_b8835f35c094b78a385923ff85ddc95ecd75ea41.psf", host: "download.windowsupdate.com" 2017/04/10 22:02:15 [error] 3375#0: 169 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.30.247, server: microsoft, request: "GET /c/msdownload/update/software/secu/2017/03/windows10.0-kb4013198-x64_2_f96caff54f77da6bd303d7672fcb2ed6f61005a5.psf HTTP/1.1", upstream: "http://8.253.185.249:80/c/msdownload/update/software/secu/2017/03/windows10.0-kb4013198-x64_2_f96caff54f77da6bd303d7672fcb2ed6f61005a5.psf", host: "download.windowsupdate.com" 2017/04/10 22:02:15 [error] 3377#0: 176 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.30.247, server: microsoft, request: "GET /c/msdownload/update/software/secu/2017/03/windows10.0-kb4013198-x64_1_b8835f35c094b78a385923ff85ddc95ecd75ea41.psf HTTP/1.1", upstream: "http://8.12.206.30:80/c/msdownload/update/software/secu/2017/03/windows10.0-kb4013198-x64_1_b8835f35c094b78a385923ff85ddc95ecd75ea41.psf", host: "download.windowsupdate.com" 2017/04/10 22:02:32 [error] 3378#0: 42 upstream prematurely closed connection while reading upstream, client: 192.168.30.247, server: microsoft, request: "GET /c/msdownload/update/software/secu/2017/03/windows10.0-kb4013198-x64_4_a3e35c6ecf8fcf7dff322cd1ef1c4fc7afcbbc88.psf HTTP/1.1", upstream: "http://8.253.32.174:80/c/msdownload/update/software/secu/2017/03/windows10.0-kb4013198-x64_4_a3e35c6ecf8fcf7dff322cd1ef1c4fc7afcbbc88.psf", host: "download.windowsupdate.com" 2017/04/10 22:02:34 [error] 3378#0: 211 upstream prematurely closed connection while reading upstream, client: 192.168.30.247, server: microsoft, request: "GET /c/msdownload/update/software/secu/2017/03/windows10.0-kb4013198-x64_1_b8835f35c094b78a385923ff85ddc95ecd75ea41.psf HTTP/1.1", upstream: "http://8.254.250.78:80/c/msdownload/update/software/secu/2017/03/windows10.0-kb4013198-x64_1_b8835f35c094b78a385923ff85ddc95ecd75ea41.psf", host: "download.windowsupdate.com"

nexusofdoom commented 7 years ago

trying this setup again

Windows Updates

server { listen lancache-microsoft deferred default; servername microsoft ; access_log /srv/lancache/logs/Access/microsoft.log main buffer=128k flush=1m; access_log /srv/lancache/logs/Keys/microsoft.log keys_range buffer=128k flush=1m; error_log /srv/lancache/logs/Errors/microsoft.log;

    # Default Node
    include lancache/resolver;
    include lancache/cache-key-default;
location / {
    #proxy_set_header   X-Real-IP $remote_addr;
    #proxy_set_header   Host      $http_host;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_send_timeout 1000s;
    proxy_connect_timeout 1000;
    # Some downloads are very large so we cache based on range to keep single downloads quick and hence ensure interactivity is good.
    proxy_bind 192.168.30.99;
    #proxy_cache_valid 200 400 206 320d;
    proxy_cache microsoft;
    proxy_read_timeout 1000;
    include lancache/proxy-cache;
    include lancache/cache-range;
}

}

nexusofdoom commented 7 years ago

got a message like this

unexpected status code 400 in slice response while reading response header from upstream

[error] 5801#0: *954 unexpected status code 400 in slice response while reading response header from upstream, client: 192.168.30.150, server: microsoft, request: "GET /c/msdownload/update/software/secu/2017/04/windows6.1-kb4015549-x64_bf8033c1dbede1fcd8b09e29328fddea5c44f8f4.psf HTTP/1.1", subrequest: "/c/msdownload/update/software/secu/2017/04/windows6.1-kb4015549-x64_bf8033c1dbede1fcd8b09e29328fddea5c44f8f4.psf", upstream: "http://13.107.4.50:80/c/msdownload/update/software/secu/2017/04/windows6.1-kb4015549-x64_bf8033c1dbede1fcd8b09e29328fddea5c44f8f4.psf", host: "au.download.windowsupdate.com"

nexusofdoom commented 7 years ago

maybe try this ? If nginx is running as a proxy / reverse proxy

that is, for users of ngx_http_proxy_module

In addition to fastcgi, the proxy module also saves the request header in a temporary buffer.

So you may need also to increase the proxy_buffer_size and the proxy_buffers, or disable it totally (Please read the nginx documentation).

Example of proxy buffering configuration

http { proxy_buffer_size 128k; proxy_buffers 4 256k; proxy_busy_buffers_size 256k; } Example of disabling your proxy buffer (recommended for long polling servers)

http { proxy_buffering off; } For more information: Nginx proxy module documentation

nexusofdoom commented 7 years ago

# sudo service nginx status ● lancache.service - LSB: starts the lancache nginx web server Loaded: loaded (/etc/init.d/lancache) Active: active (running) since Tue 2017-04-11 19:45:37 CDT; 35s ago Process: 6085 ExecStop=/etc/init.d/lancache stop (code=exited, status=0/SUCCESS) Process: 6088 ExecStart=/etc/init.d/lancache start (code=exited, status=0/SUCCESS) CGroup: /system.slice/lancache.service ├─6090 nginx: master process /usr/local/nginx/sbin/nginx ├─6091 nginx: worker process ├─6092 nginx: worker process ├─6093 nginx: worker process ├─6094 nginx: worker process ├─6095 nginx: cache manager process └─6096 nginx: cache loader process

Apr 11 19:45:37 lancache-server lancache[6088]: Starting : nginx: [warn] could not build optimal proxy_headers_hash, you should increase either proxy_headers_hash_max_size: 512 or pr..._bucket_size Apr 11 19:45:37 lancache-server lancache[6088]: nginx: [warn] could not build optimal proxy_headers_hash, you should increase either proxy_headers_hash_max_size: 512 or proxy_headers..._bucket_size Apr 11 19:45:37 lancache-server lancache[6088]: lancache. Apr 11 19:45:37 lancache-server systemd[1]: Started LSB: starts the lancache nginx web server.

there is a "warn" about proxy header hash size

change this in my ms-config "proxy_headers_hash_bucket_size 512;"

Windows Updates

server { listen lancache-microsoft deferred default; servername microsoft ; access_log /srv/lancache/logs/Access/microsoft.log main buffer=256k flush=1m; access_log /srv/lancache/logs/Keys/microsoft.log keys_slice buffer=256k flush=1m; error_log /srv/lancache/logs/Errors/microsoft.log debug; proxy_ignore_client_abort on;

Default Node

    include lancache/resolver;
    include lancache/cache-key-default;
location / {
    proxy_set_header    Host            $host;
    proxy_set_header    X-Real-IP       $remote_addr;
    proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header    Accept-Encoding "";
    proxy_headers_hash_bucket_size 512;
    #proxy_set_header   X-Real-IP $remote_addr;
    #proxy_set_header   Host      $http_host;
    #fastcgi_buffers 16 16k;
    #fastcgi_buffer_size 32k;
    #proxy_buffer_size   128k;
    #proxy_buffers   4 256k;
    #proxy_busy_buffers_size   256k;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    #proxy_send_timeout 1000s;
    #proxy_connect_timeout 1000;
    # Some downloads are very large so we cache based on range to keep single downloads quick and hence ensure interactivity is good.
    proxy_bind 192.168.30.99;
    proxy_cache_valid 206 200 320d;
    proxy_buffering off;
    proxy_cache microsoft;
    proxy_read_timeout 1000;
    include lancache/proxy-cache;
    include lancache/cache-slice;
}

}