Riot CDN is now served over HTTPS

[ti-mo]

As reported by @fsalazar1977, Riot is now serving its content over SSL. (at least the client requests https:// resources) This cripples our ability to cache game content and breaks the client's update mechanism until fixed.

Looking for possible downgrade mechanisms to HTTP. Since Akamai's certs are used for l3cdn.riotgames.com, I assume cert validation is disabled, issuing self-signed to the client is rumored to work. (to be confirmed)

@RiotGames, please don't do this, run artifact integrity checks on the client. :)

[ti-mo]

TLS seems to be properly implemented, cert validity is enforced and there are no endpoint URLs going over plain HTTP that would allow them to be re-written. Schema cannot be reconfigured on the client as far as I can tell, and SSL cannot be disabled anywhere.

One thing, though: RADS/system/system contains the DownloadURL=l3cdn.riotgames.com directive, opening the possibility of specifying a domain we do own a valid certificate for, and translating that to l3cdn requests. If anyone can come up with any other suggestions, feel free to post.

Just contacted @RiotGames support with an explanation. Maybe, just maybe, they'll escalate so we can work on a solution together.

[v3n]

Hi guys,

Try adding the following line to {League install dir}/RADS/system/system.cfg:

UseTLS=false

Cheers! Jonathan Howard // Riot Venia

[ti-mo]

Hey @v3n, thanks for sharing. This is fine as a short-term workaround for smaller events, but I'm afraid this won't do at scale. It also leaves clients 'unprotected' if they forget to remove this after the event, which kind of defeats the point. We have no way of pushing this variable to the client automatically.

I've gotten a response to my Riot ticket (#20689596) stating this has been escalated to operations internally, so I'm hoping for the best! Feel free to give this a little bump. :)

Thanks for reaching out, let me know when you know more!

[v3n]

@ti-mo,

Thanks for the feedback. We're currently investigating potential ways to allow LAN parties/PC cafes to patch as a long-term solution. Please do note that this config flag will not leave clients unprotected, as it will soon revert to using TLS after every patch.

Out of curiosity, would a command line parameter work better than a config option or do you expect similar problems?

Unfortunately, we cannot distribute signed certificates, as that undermines TLS security for Riot as a whole (ex: Microsoft's MD5 collision).

We are open to potential changes as long as they require player override/input, however, the proposed WAN/port 443/TLS failover undermine the security of our TLS implementation as it's only secure as the weakest link.

Are the computers in your LAN event participant-supplied or supplied by you?

Jonathan Howard // Riot Venia // Patcher Team

[ti-mo]

Hey @v3n, gotcha, makes sense to overwrite the config with every patch. I agree with all the points you make, any TLS downgrade/disable vector in the client compromises the whole implementation. I've written my thoughts below in a lengthy (sorry) post, hope it helps!

We run BYOC events ranging from 40 to 1000 participants, so any client-side intervention from our side is either highly inconvenient or not possible at all. The average tech-savvyness of participants is low; we could get away with requiring a config file edit, but a command line switch would be too much to ask, no one would bother. The goal is to do this completely transparently, 100% plug & play. All we really have control over is DNS, so this is what we use to redirect requests to our local servers. Before the participants arrive, we reinstall many popular game clients on our own machines to heat up the cache.

We're always trying to inconvenience the user at little as possible. With our current cache configuration, we don't listen on port 443 at all, so this is what the user sees:

This warning is generic and seems to be triggered when libcurl fails a GET operation from l3cdn. I've tried serving from 443 with a self-signed cert, the error's the same in both cases (serving an invalid cert as well as refusing connections on 443).

Personally, I'd go for a configuration setting to 'disable patching over secure connection', but also adding a shortcut button on the retry screen shown in the screenshot, to make it as painless for the user as possible. We don't want to make this permanent, so something like 'Disable TLS for 3 days', maybe? (in our case) Extra options in the settings menu would be nice, so PC Bangs wouldn't have to click the annoying pop-up every few days, but this currently doesn't apply to us. :+1:

As a long-term solution, I would suggest having your build pipeline pack, sign, checksum and/or encrypt all artifacts you push out to the client. Your artifacts would still be cacheable, the client can detect and minimize errors in transit (I assume this is part of your reason for adopting TLS), you could also implement chunked transfer allowing the client to pause & resume and efficiently repair chunks if needed. (not to mention, this can be cached very efficiently! :) e0951c4 )

To make life easy for event organizers, you could provide an rsync endpoint so they can keep their copy in sync with a 100% integrity, without having to rely on a user to fetch it through the cache first.

[v3n]

@ti-mo, shooting off a quick reply before lunch:

Brainstorming here, you could provide a League shortcut that runs with the command line flag? This runs a risk of players using it during normal operations however, which is not ideal. Another idea would be a Powershell script that patches the config file.

I believe we are wary about placing a disable option in such a player facing way, just due to general lack of tech-savyness.

I'll see about updating that messaging for libcurl failures, definitely agree about the overly generic message.

One big reason for us adopting TLS besides general security is the prevalence of HTTP errors being injected by ISPs and other third parties. For example, ISPs attempt to (poorly) cache our updates and cause broken builds for players (that they could never recover from since there was no way for them to bypass the ISP cache).

Great post, passing your feedback onto the rest of the team, will keep you updated as towards their response.

[v3n]

We are exploring potential options internally for securely downgrading to HTTP.

[ti-mo]

@v3n, thanks for the quick response, really appreciate the openness!

I see the main concern here being broken caching layers, TLS really is the best way to poke through those caches indeed. If those ISPs manipulate DNS like we do, then there's no other way to distinguish between 'us' and 'them' besides checking the address class the l3cdn record resolves to.

How do you feel about only allowing (temporary or permanent) plain HTTP fallback when the endpoint is an RFC1918 address? It would require non-negligible effort for an ISP to implement a 'magic' bogon address in a private range that performs NAT to their caching layer. Either way, the user would still be warned about TLS errors, and if they start troubleshooting themselves, the launcher presents them with the information they need (including mandatory big red button to disable TLS). I don't see this part being really necessary.

The only concern here is the user getting MITM'd, since I've noticed plain .exe files going over the wire in previous iterations of your client. Also cases of cache corruption during some of our events, but we always carry the responsibility for that. You'd want to sign your artifacts precisely for this reason, the only attack vector here being a pre-compromised client that also accepts the attacker's signature.

[v3n]

@ti-mo, absolutely, we try our best to fix things when we break them!

I'm a huge fan of the RFC1918 idea and have raised it to the rest of the team. MITM shouldn't be a huge issue, we normally download packages and fall back to loose files if there is an issue. I believe we also checksum loose files as an additional security precaution.

Is there a specific time frame you are dealing with? (I can't make any promises as towards our schedule, but I've taken a interest in getting this resolved).

[ti-mo]

@v3n, alright, then I believe RFC1918 is our best bet. Need an equivalent for IPv6 though, which should be RFC4193.

I'm running a casual event on the 18th of March, the Multiplay guys are running i57 on the 25th, so the sooner the better! ;)

Can't hold this against you, though, thanks a lot for the cooperation so far, really makes a difference! It's in all of our best interests to get as many people patched, logged on and playing during these events. All I hope for is other platforms not making the move to TLS as well.. They might be a lot harder to reach out to.

[nexusofdoom]

1.Fortigate firewall SSL Inspection do we know if Riot LOL will work with fortigate firewall inspection since it does almost the same as nginx revers proxy with ssl.

2.will the launcher work if i get a domain name Example gameupdates.com and get a real SSL from like godaddy and do a cname from l3cdn.riotgames.com to gameupdates.com

3.what about pushing the cert to the client like Windows Trusted Cert Store. or a options to tell the client to look for the cert in a folder

[ti-mo]

@nexusofdoom

1. Fortigate requires their CA to be installed onto clients, we're not doing this. I would not run packet inspection on CDN connections for the life of me!
2. This would be a huge flaw, a well-written client will run validation against its given endpoint. A stub resolver is generally not aware of any cnames; a cname is not a redirect, merely an intermediary step in a recursive resolving process.
3. Sort of the same as in point 1, this again requires manipulation to the participant's computer. A TLS client is not required to obey the OS trust store (Riot could easily provide their own trust store). Also, this works differently across operating systems. Personally, I would never install some LAN's CA on my personal machine as a visitor.

[nexusofdoom]

so for my setup where all the pc's owned by me would this work?

2.will the launcher work if i get a domain name Example gameupdates.com and get a real SSL from like godaddy and do a cname from l3cdn.riotgames.com to gameupdates.com

then install the CRT in the windows crt store on the windows computer.

do we know what the launcher is looking for since the domain l3cdn.riotgames.com forwards to akimia servers and at that point the current ssl is akimia

[ti-mo]

@nexusofdoom you would need a self-signed cert for l3cdn.riotgames.com, not for any of its cname targets. No guarantees the launcher obeys the Windows trust store, you'll have to test that.

[nexusofdoom]

thanks

[v3n]

We validate the l3cdn cert, so attempting to spoof it with a self-trusted won't work, otherwise we're vulnerable to CA sideloading attacks.

[nexusofdoom]

if that is so then any client of ours that uses Fortigate firewall SSL Inspection or any other firewall inspections will not work also then?

[v3n]

Correct.

The only workaround right now is this:

Try adding the following line to {League install dir}/RADS/system/system.cfg:

UseTLS=false

[nexusofdoom]

would there be a option for riot to make a check box in the launcher to use https Enable/Disable for updates. so the default option is to use https. example you have a check box for Enable/Disable for Allow Peer To Peer Transfer.

[ti-mo]

@nexusofdoom Please read the whole thread :) @v3n Any news so far?

[nexusofdoom]

http://wiki.squid-cache.org/Features/HTTPS

Quick skim of the thread you reference brought squid & HTTPS to mind. Squid can be configured to cache HTTPS stuff.

Maybe some local DNS trickery and squid listening on 443 or nginx talking to it.

Basically it sounds like riot is trying to secure tranmission channel instead of validating the package(think md5sum or gpg signature of the patch). On Mar 9, 2016 3:23 AM, "Timo Beckers" notifications@github.com wrote:

@nexusofdoom https://github.com/nexusofdoom Please read the whole thread :) @v3n https://github.com/v3n Any news so far?

— Reply to this email directly or view it on GitHub https://github.com/ti-mo/ansible-lanparty/issues/8#issuecomment-194200595 .

[v3n]

@nexusofdoom: We actually do validate the packages, our switch to TLS was a move to punch through bad CDNs.

@ti-mo: It's in the pipeline, unfortunately we missed the cut for this patch. Could you give me some idea of the numbers associated with these LANs for some better justification to the team?

Sorry this is affecting you guys adversely, I understand that the 'workaround' is a pretty bad one for players who aren't necessarily technically competent.

[RobertJamesMichael]

@v3n: I am from Bolivia, I am not an expert in the telecommunication aspect but I can tell you something about the problem and how this is affecting a lot of people in my city (and my country). First of all, Bolivia doesn’t have a coast and this mean that all ISP are obligated to get to the Internet Backbones through other countries like Brasil, Chile, Peru and Argentina, this problem makes that the internet connection prices for the final user are very expensive, for example a 3 MB (3mb DOWN / 512kb UP) costs about 90$ per month and this is very expensive for the majority of the people in Bolivia (consider that the minimum salary is 237$). This is the reason why the LANs steel are a place where many people is playing. I have two LANs (80 and 50 pcs), since I installed the solution provided, my pcs began to get the game updates instantly and this helped a lot, since then I spread the solution to many LANs in my city, today (considering only the LANs I personally contacted and helped) may be approximately about 5000 - 6000 computers that are using the solution every day. Consider also that some of the “helped” began to spread the solutions to others (I think this easily reached a 30000 computers just in my city). Since you changed the way the game updates, and you gave the alternative to add the line (useTLS=false) in the configuration (and I did that) many estrange problems began to happened like some computers got FATAL ERROR 0001, other ones began to download the complete game again (or many MB), all the computers are changing the region to Brasil when you start the game, etc. So we had to get the computers updated using old methods like coping the updated game from a “server” again. This two weeks were a fatal headache for me (and the others). I hope this is a good reason for you to help us to get a solution to the problem.

[v3n]

@fsalazar1977: Just want to make sure that you used the correct configuration: UseTLS = false this is case sensitive. Can we also get some copy of logs on the bad machines?

@all: We're actively working on a fix.

[ti-mo]

@v3n i57 (march 25th) currently has over 2000 BYOC seats open, and looks like they'll be sold out. Thanks for the update!

[riconnon]

It might be worth noting that at i57 (and insomnia in general) client machines do not have RFC1918 addresses.

[riconnon]

Looking at a comparable CDN (Steam game content) the way they seem to avoid unwanted caching is by using hashes of the data for the filenames. This way it's very unlikely that a URI would ever be the same twice (and thus caching is never a problem)

[ti-mo]

@irconan Good to know about the addresses, thanks!

Steam is very cache-friendly because they pre-chunk assets, hash the filenames and store them in a hashtree on-disk. This has many benefits, both from a caching and an integrity/validation point of view. It almost guarantees that different versions of the same file (with different contents) never collide, and that chunk contents can be validated against their filenames. This is a sound implementation, it has never caused us any issues.

[v3n]

@irconan: Can you clarify? So HTTP fallback for RFC1918 addresses will not work for i57? (note that the fallback occurs when l3cdn.* is resolved, not if the client machine has a private address). As long as l3cdn resolves to a RFC1918 address the override will work.

@ti-mo: Other than just straight bad data, ISP caches were also giving us unexpected HTTP error codes, like 404s and 500s, hence the move to SSL.

[liv3d]

@v3n i57 has a large public address pool and every user/client gets a real IP. I'll have a talk to the network team when I'm in the office tomorrow about seeing if we can resolve l3cdn to a RFC1918 address and do some funky routing.

[v3n]

@liv3d Thanks. Are these IP4 or IP6 addresses?

[cerealcable]

RFC1918 space is specifically IPv4. On Mar 14, 2016 3:57 PM, "Jonathan Howard" notifications@github.com wrote:

@liv3d https://github.com/liv3d Thanks. Are these IP4 or IP6 addresses?

— Reply to this email directly or view it on GitHub https://github.com/ti-mo/ansible-lanparty/issues/8#issuecomment-196518123 .

[liv3d]

@v3n both, I can get those tomorrow or drop me an email to liv3d [@] multiplay.co.uk and I can put you in touch with the right people.

[stevenh]

@v3n I see why you're trying HTTPS but I'll be honest it seems like the wrong approach. You mentioned that some ISP's are causing issues for clients with broken caches, it would be good to get more information on the exact issues you're seeing?

Having put together one of first big LANcache implementations for the Insomnia Gaming Festivals we have a very good understanding of what works and what doesn't so I'm sure we can help you put together a solution that will prevent caching issues.

I'm sure you're aware but just to be clear LANcache's are critical to the successful operation of both large and small gaming events and HTTPS by design isn't cacheable and hence could prevent LoL and any other Riot titles being enjoyed at said events.

[v3n]

@cerealcable: Was asking the network space of i57, not RFC1918.

@liv3d: I can still get in touch with you guys if you have any questions or concerns after reading the following response.

@stevenh:

Our purpose is two-fold:

1. Security
2. Cache punching

First, let's talk security. It's true that there are alternative ways to TLS to ensure integrity of the downloaded content (such as checksums, which we already do, however if our manifests are sent in the clear, we're still pwned). We've seen evidence of people performing attacks against League users, which TLS stops hard in their tracks. One thing we're looking at as a long-term solution is to deliver manifests over TLS, which will allow us to safely degrade normal patching traffic to HTTP. However, this is a much larger change to support different routes for manifests vs package files.

Second, cache punching. Our telemetry indicated a large amount of patching errors from bad cache servers in between Riot hosted CDNs and patcher clients. TLS allows us to prevent ISPs from caching those servers. We've noted a statistically significant improvement in number of patcher errors reported by the client since implementing TLS. Additionally, there has been a marked patch speed improvement from players having a direct route to Riot-owned CDN's. Overall, we consider these two big successes.

However, it's true that this makes caches impossible. I've only just recently joined Riot, but one of my jobs before that was in university IT maintaining a gaming lab, so I feel your pain (and wish I had known about LANCache while I was there!).

On the good news front, I've just submitted a change that will gracefully degrade to a HTTP connection if l3cdn.riotgames.com resolves to an RFC1918 address. This change should hit PBE tomorrow and go live in the 6.6 patch, so do let me know if you encounter any issues.

Cheers! Jonathan Howard // Riot Venia // Patcher Team

[stevenh]

@v3n Thanks for the response, are you saying that there are malicious proxies out there which are actively tampering with manifests and the downloads to attack League users?

If so that's a new kind of low and I'm shocked, broken proxies yes; I had an open ticket with squid for 8 years iirc before they fixed it, and it wasn't a minor one either total broke caching :smile:

RFC1918 might not be simple for us to as our LAN is too big for NAT, at least cost effect NAT, so we use real IPv4 addresses; so we'll have to see if the network team can engineer a dirty hack to get the cache box to use an RFC1918 address for its riot bind.

@liv3d can you engage with the team to see if that's possible?

While we have your attention @v3n could you confirm the DNS spoof addresses, currently we have two entries: l3cdn.riotgames.com worldwide.l3cdn.riotgames.com

Are there any others?

[v3n]

@stevenh: Yes, exactly that, proxies and even ISP-level DNS attacks. It's dirty, so we take security extremely seriously over here. It's a pretty good attack vector if you think about it, most people don't expect their games to deliver malware.

I can confirm that those two should be overridden. That's all off the top of my head, but I'll check when I get into the office tomorrow and let you know if there are any additional ones.

[ti-mo]

@stevenh I assume you have routers at the event locally (the visitors' default gateways), so you could just attach an RFC1918 network to it, run the cache on it and route the traffic, no NAT required. This network would obviously not be reachable from the outside, but for your visitors, it would do just fine.

[v3n]

Anyone mind verifying the fix works on the current PBE build?

Cheers! Jonathan

[stevenh]

@v3n looking good, previous tests results where for the standard client which obviously didn't work.

PBE seems to just work :+1:

[v3n]

@stevenh Awesome, thanks for verifying it for us.

Cheers! Jonathan Howard // Riot Venia

[ti-mo]

Thanks for the contributions and all the hard work, everyone! Looking forward to seeing this on the live servers. Closing this issue.

[nexusofdoom]

once i remove the UseTLS=false it does not work.. how to make the client use http over https on a 10.0.4.1 network

thanks

where do we get this build "PBE build?"

[stevenh]

@nexusofdoom you need to use the beta client and your dns for l3cdn.riotgames.com needs to resolve to a RFC1918 address

[stevenh]

@v3n do you know when the RFC1918 changes will be merged to the mainline branch?

We have insomnia 57 this weekend (25th - 28th March 2015) featuring a League tournament so it would be great if you could get it released by then.

[riconnon]

@stevenh Even if they merged it today, what are the chances that users would have updated to a version containing it before the event? Way I see it you basically have to run with no LoL content caching for i57 otherwise there'll be loads of people on 6.5 that can't update to 6.6 at the event.

[stevenh]

If that where the full client yes, but it should just need the new patcher, which we could provide as a local download?

[riconnon]

@stevenh how would you provide the information on how to user the locally mirrored patcher to the end-users? I could see that being a recipe for a lot of work for helpdesk

[stevenh]

Local intranet, forums and tournament info pages.

They will soon find out when the updates instantly fail ;-)

[riconnon]

@stevenh Alternatively... can we make l3cdn.riotgames.com a separate IP alias on the lancache instance and simply DNAT to the direct Riot service from port 443 but service port 80 as usual. This way the first stage of the client update should go direct via HTTPS then the second stage will fallback to HTTP once the launcher is updated. Needs testing, but would make it a zero config option again.