ccpgames / sso-issues

Please file issues with the CCP SSO (login.eveonline.com) here.
17 stars 1 forks source link

JWKS server temporarily returned a `Content-Encoding` not in `Accept-Encoding` (brotli) #81

Closed soratidus999 closed 11 months ago

soratidus999 commented 11 months ago

https://gitlab.com/allianceauth/django-esi/-/issues/32

Bug

Between 1800 and 2100 UTC 2023-10-24, https://login.eveonline.com/oauth/jwks was returning brotli compressed json unexpectedly.

Unfortunately i was asleep, and dont have any live request headers from this timeframe, but this is what we've pieced together

Reproduction Steps

        response = requests.get(https://login.eveonline.com/oauth/jwks)
        ## response.request.headers._store['accept-encoding'] = ('Accept-Encoding', 'gzip, deflate')
        response.raise_for_status()
        data = response.json()

Actual Behaviour

response.headers._store['content-encoding'] = ('Content-Encoding', 'br')

Json returned is brotli compressed, I dont have the brotli library, json is not decompressed, JSONDecodeError

Expected Behaviour

response.headers._store['content-encoding'] = ('Content-Encoding', 'gzip')

gzip compressed json is returned, happiness.

Happy to handle brotli compression if this is desired, but this is not a default part of our stack, it was unexpected and ultimately the Accepts-Encoding header should be respected.

Considering this was a temporary set of unexpected behaviour resulting in an outage, I dont think this was intentional, but this is logged for documentation and troubleshooting purposes.

soratidus999 commented 11 months ago

Django-ESI 5.1.0 now ships with the brotli lib as a dependency to stop this happening again

erlendur-ccp commented 11 months ago

A note on the brotli-encoded responses from https://login.eveonline.com/oauth/jwks between 18 and 21 o'clock UTC last night: Nobody at CCP pressed any buttons and we have an open support case with Cloudflare on this incident. Brotli has been enabled on our account for 3 years but brotli-encoded responses should only be sent back if the accept encoding headers on the request include "br".

lynkfox commented 11 months ago

Cloudflare raised an incident about it - seems to br a cloud flare not ccp issue

https://www.cloudflarestatus.com/incidents/sx7nfzdw6c6h

soratidus999 commented 11 months ago

Appreciate the update, and after looking at that cloudflare incident this is entirely upstream