search

mmd123 / squid-cache-dynamic_refresh-list

list of dynamic_refresh patterns to use with squid3, I had a hard time finding things online, so I figured I'd make a collaborative github repo for just this purpose
8 stars 4 forks source link

Nvidia updates via geforce experience not working #2

Closed ageekhere closed 2 years ago

ageekhere commented 4 years ago

So at the moment Nvidia updates when using geforce experience does not hit the cache. To test you can open geforce experience download the update, then before installing it rename the install file and then update again.

Suggestion maybe using wireshark to trackdown the correct server could be used.

mmd123 commented 4 years ago

I am looking into this now, thanks for the heads up...also, HUGE thank you for the suggestion of using wireshark, I'm not the most knowledgeable on this stuff, so that extra tip is more valuable than gold. thank you.

also, going by the download speeds I'm getting on the update right now...its for sure not being cached, though, admittedly I don't know if it was downloaded already on my pc with nvidia drivers yet or not, will report back after I know for sure.

mmd123 commented 4 years ago

yea, definitely not working, I have pfsense on my end limited for wan downloads to 4MBps and I'm getting far less than that, meaning that it if were properly caching, then I'd be getting far in excess of 3MBps that I'm getting now, so yea, your right, its definitely not working right, got wireshark installed and gonna have it scan now and report updates after they're found and committed. thanks @ageekhere

mmd123 commented 4 years ago

.....not sure what to make of this...

https://mxtoolbox.com/SuperTool.aspx?action=ptr%3a192.229.211.70&run=toolpage https://www.ultratools.com/tools/ipWhoisLookupResult https://www.whois.com/whois/192.229.211.70 https://www.hashemian.com/tools/reverse-whois.php thoughts @ageekhere ?? seems to be a CDN, but outside of that obvious little tidbit, I'm gonna be totally frank, I'm not sure what to make of this and how to adapt it that said.

hindsight....I might have just realized a band-aid fix for this for the time being?? just increase the caching capacity for .exe files, given that the cdn is downloading it as a standard .exe....still not a full fix, but its a band-aid fix for the time being at least.... I'll commit that, while I ponder, and wait your reply, on how to truly fix this issue.

mmd123 commented 4 years ago

direct from nvidia support, the list of url's they use for geforce experience downloads

https://international-gfe.download.nvidia.com/Windows/451.67/451.67-desktop-win8-win7-64bit-international-whql.exe

https://international-gfe.download.nvidia.com/Windows/451.67/451.67-desktop-win7-64bit-international-whql.exe

https://international-gfe.download.nvidia.com/Windows/451.67/451.67-desktop-win10-64bit-international-whql.exe

https://international-gfe.download.nvidia.com/Windows/451.67/451.67-notebook-win8-win7-64bit-international-whql.exe

https://international-gfe.download.nvidia.com/Windows/451.67/451.67-notebook-win7-64bit-international-whql.exe

https://international-gfe.download.nvidia.com/Windows/451.67/451.67-notebook-win10-64bit-international-whql.exe

https://international-gfe.download.nvidia.com/Windows/451.67/451.67-desktop-win10-64bit-international-dch-whql.exe

https://international-gfe.download.nvidia.com/Windows/451.67/451.67-notebook-win10-64bit-international-dch-whql.exe

https://international-gfe.download.nvidia.com/Windows/451.67/451.67-desktop-win8-win7-64bit-international-whql-g.exe

https://international-gfe.download.nvidia.com/Windows/451.67/451.67-desktop-win7-64bit-international-whql-g.exe

https://international-gfe.download.nvidia.com/Windows/451.67/451.67-desktop-win10-64bit-international-whql-g.exe

https://international-gfe.download.nvidia.com/Windows/451.67/451.67-notebook-win8-win7-64bit-international-whql-g.exe

https://international-gfe.download.nvidia.com/Windows/451.67/451.67-notebook-win7-64bit-international-whql-g.exe

https://international-gfe.download.nvidia.com/Windows/451.67/451.67-notebook-win10-64bit-international-whql-g.exe

https://international-gfe.download.nvidia.com/Windows/451.67/451.67-desktop-win10-64bit-international-dch-whql-g.exe

https://international-gfe.download.nvidia.com/Windows/451.67/451.67-notebook-win10-64bit-international-dch-whql-g.exe

ageekhere commented 4 years ago

Ok after doing a bit of reading on the matter it looks like it SQUID cannot cache any content from HTTPS sites when using HTTPS/SSL Interception "SPLICE ALL" Note sure if Splice Whitelist, Bump Otherwise works (you have to install a certificate on all devices)

However there is a new Feature called SslBump Peek and Splice https://wiki.squid-cache.org/Features/SslPeekAndSplice Maybe that could work? https://forum.netgate.com/topic/155265/squid-s-new-sslbump-peek-and-splice-for-https-caching

mmd123 commented 4 years ago

I have squid on my end set up exactly that way with my pfsense ssl certificate installed on all my own devices, and it's indeed still not caching nvidias updates, so that definitely doesn't seem to fix this unfortunately... I'll loom into that new feature tomorrow and see if it changes anything and report back.

mmd123 commented 4 years ago

@ageekhere thoughts on whats contained in this video, and if the contents are applicable to this situation??

https://youtu.be/2hVZ5sZ-nNw

ageekhere commented 4 years ago

having a look, also looks like there are some more refresh patterns for windows updates

refresh_pattern -i windowsupdate.com/..(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims refresh_pattern -i microsoft.com/..(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims refresh_pattern -i windows.com/..(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims refresh_pattern -i microsoft.com.akadns.net/..(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims refresh_pattern -i deploy.akamaitechnologies.com/.*.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims

Check for any missing patterns

UPDATE What the video is showing is you can setup custom rules for MITM The video shows making exceptions to either splice or bump So if you splice or do a splice all then it becomes a TCP tunnel without decrypting proxied traffic.

However things like windows updates have issues when using bump so an exception is made for that traffic to be spliced. So it is mainly used to fix broken sites. I use a WPAD by default so I do not have to set up these exceptions.

The issue is https content need to be decrypted in order for it to be cached however this seems ssl_bump breaks that.

Hmm some more research is needed

ageekhere commented 4 years ago

maybe a good time to ask in squid mailing list http://www.squid-cache.org/Support/mailing-lists.html#squid-users

ghost commented 4 years ago

Hello guys, hope you are well !!

I new to squid ... and i trying to cache Windows Update updates , saw that you have a specific part for this... Do I add the code to squid.conf? or create a new file?

Another question ...I need to configure cache_dir, to save updates.

Thanks and Regars.

Your code

windows update NEW UPDATE 0.04

refresh_pattern windowsupdate.com/..(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims refresh_pattern update.microsoft.com/..(cab|exe) 43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims refresh_pattern download.microsoft.com/..(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims refresh_pattern -i microsoft.com/..(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims refresh_pattern -i windowsupdate.com/..(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims refresh_pattern -i windows.com/..(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims refresh_pattern ([^.]+.)?(download|(windows)?update).(microsoft.)?com/..(cab|exe|msi|msp|psf) 4320 100% 43200 reload-into-ims refresh_pattern update.microsoft.com/..(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims refresh_pattern windowsupdate.com/..(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims refresh_pattern download.microsoft.com/..(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims refresh_pattern www.microsoft.com/..(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims refresh_pattern au.download.windowsupdate.com/..(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims refresh_pattern bg.v4.pr.dl.ws.microsoft.com/..(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims refresh_pattern -i .windowsupdate.com/..(cab|exe) 259200 100% 259200 ignore-no-store ignore-reload reload-into-ims refresh_pattern -i .update.microsoft.com/..(cab|exe|dll|msi|psf) 259200 100% 259200 ignore-no-store ignore-reload reload-into-ims refresh_pattern au.download.windowsupdate.com/..(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims refresh_pattern bg.v4.pr.dl.ws.microsoft.com/..(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims refresh_pattern -i .update.microsoft.com/..(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth refresh_pattern -i .windowsupdate.com/..(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth refresh_pattern -i .download.microsoft.com/..(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth refresh_pattern -i .ws.microsoft.com/.*.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth

new refresh patterns 3

acl Windows_Update dstdomain windowsupdate.microsoft.com acl Windows_Update dstdomain .update.microsoft.com acl Windows_Update dstdomain download.windowsupdate.com acl Windows_Update dstdomain www.download.windowsupdate.com acl Windows_Update dstdomain au.download.windowsupdate.com acl Windows_Update dstdomain bg.v4.pr.dl.ws.microsoft.com

mmd123 commented 4 years ago

Hello guys, hope you are well !!

I new to squid ... and i trying to cache Windows Update updates , saw that you have a specific part for this... Do I add the code to squid.conf? or create a new file?

Another question ...I need to configure cache_dir, to save updates.

Thanks and Regars.

Your code

windows update NEW UPDATE 0.04

refreshpattern windowsupdate.com/..(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims refreshpattern update.microsoft.com/..(cab|exe) 43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims refreshpattern download.microsoft.com/..(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims refreshpattern -i microsoft.com/..(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims refreshpattern -i windowsupdate.com/..(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims refreshpattern -i windows.com/..(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims refreshpattern ([^.]+.)?(download|(windows)?update).(microsoft.)?com/..(cab|exe|msi|msp|psf) 4320 100% 43200 reload-into-ims refreshpattern update.microsoft.com/..(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims refreshpattern windowsupdate.com/..(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims refreshpattern download.microsoft.com/..(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims refresh_pattern www.microsoft.com/.*.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims refreshpattern au.download.windowsupdate.com/..(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims refreshpattern bg.v4.pr.dl.ws.microsoft.com/..(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims refresh_pattern -i .windowsupdate.com/..(cab|exe) 259200 100% 259200 ignore-no-store ignore-reload reload-into-ims refresh_pattern -i .update.microsoft.com/..(cab|exe|dll|msi|psf) 259200 100% 259200 ignore-no-store ignore-reload reload-into-ims refreshpattern au.download.windowsupdate.com/..(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims refreshpattern bg.v4.pr.dl.ws.microsoft.com/..(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims refreshpattern -i .update.microsoft.com/..(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth refreshpattern -i .windowsupdate.com/..(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth refreshpattern -i .download.microsoft.com/..(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth refreshpattern -i .ws.microsoft.com/..(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth

new refresh patterns 3

acl Windows_Update dstdomain windowsupdate.microsoft.com acl Windows_Update dstdomain .update.microsoft.com acl Windows_Update dstdomain download.windowsupdate.com acl Windows_Update dstdomain www.download.windowsupdate.com acl Windows_Update dstdomain au.download.windowsupdate.com acl Windows_Update dstdomain bg.v4.pr.dl.ws.microsoft.com

apologies for my late reply, I have been beginning to have a hell of a time with crap per the pandemic and mental drain, to answer your first chunk of the question, I have yet to fully figure out if windows updates are properly working with squid even with my implementation, so to be totally honest, I AM aware that THAT goal is a bit of a hail marry goal, given even all of my tech friends state that it USED to be possible, but with changes from Microsoft, your best bet to getting that to work is, and has been for years, to run a WSUS server per the changes MS made in delivery, so that goal is a long term goal, just because I'm lazy and don't want to run more systems than I have to, and I am lazy and stubborn, and determined to either get it to work, or curse Microsoft more than I do on any given daily basis (and trust me, that is already a lot)

second off, part of the reason this has taken me so long to reply to is that I turned off squid locally given the havoc it was having in me just being able to use apps on my phone, given I HAVE the certificate installed, and squid was STILL breaking sites and apps (reddit, twitter, for two specific examples)

third: yes you will need to set up a location, but squid in pfsense should automatically do that for you if you configure it properly in the beginning steps, I have followed numerous guides online to get to the point I'm at now, so my best suggestion is to just follow this guide as a baseline, and go from there.

again, as for windows updates, I was not ever really able to tell one way or another if it did actually cache them or not, my internet is fast enough even when limiting it, that it would not be obvious one way or another on speed, as to if it was working or not, and given my previous knowledge coming into this, I'm just assuming it's not working for my own sanity, but keeping it there as a reminder to find out for sure at a later date.

https://techexpert.tips/pfsense/squid-installation-pfsense/

follow this guys guide, and as for the allowed networks in the acl section, your going to want to set your local network's address, so if you have a default home network (AND HAVE NOT CUSTOMIZED IT LIKE ME) most home networks default to 192.168.1.X subnets, so for the network you'd add, if that is correct for you, would be 192.168.1.0/24 in that section.

mmd123 commented 4 years ago

maybe a good time to ask in squid mailing list http://www.squid-cache.org/Support/mailing-lists.html#squid-users

welp...I do not remember if I ever reached out to the squid mailing list, but I definitely did just now...so now we wait.

mmd123 commented 3 years ago

I can finally respond to this with actual useful information... no, windows updates cannot be cached, I finally have squid FULLY working WITH ssl interception, and windows updates breaks, microsoft is apparently using certificate pinning for their update servers, and you cannot use squid to cache windows updates sadly, not for lack of trying, but they have it that it just will not work, flat out. so I have removed the windows updates section from my current squid configuration, but never got around to updating this repo list. I will do that part now.