snok / django-auth-adfs

A Django authentication backend for Microsoft ADFS and AzureAD
http://django-auth-adfs.readthedocs.io/
BSD 2-Clause "Simplified" License
272 stars 101 forks source link

buffering causes internal server error #88

Closed pandawankenobi closed 4 years ago

pandawankenobi commented 4 years ago

We run a an azure webapp from a dockercontainer: Alpine linux, ngix, uwisgi, python 3.6, django 2.x and user your excellent package to handle authentication. First things first. Thanks for all the help you have provided :) Now the issue: Every now and then our ADFS server takes long to respond. When running debug=True we get "Unexpected ADFS response: buffering. " with debug = False the user just get send to a 500 page. I tried making a decorator to retry the view, but it does not seem to work. If the user then tries to to go the page again everything works fine. i have retries set to 24 and timeout = 5.

jobec commented 4 years ago

Retrying the view won't help, because authentication takes place outside the view (before it's called actually). Also, the 500 error without debug is normal. Because with debug turned on, Django will send the traceback and extra info to the user.

Do you authenticate other things against your ADFS server? Because it sounds like it's having performance issues.

What you could do, is turn on urllib debugging and see what that gives. But it might give quite some extra logging output.

import requests
import logging
from http.client import HTTPConnection  # py3

log = logging.getLogger('urllib3')
log.setLevel(logging.DEBUG)

# logging from urllib3 to console
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
log.addHandler(ch)

# print statements from `http.client.HTTPConnection` to console/stdout
HTTPConnection.debuglevel = 1
pandawankenobi commented 4 years ago

i think the performance issue is right - however i can't change the setup of the server.

Am I wrong to think that this is line 32-34 of backend.py, handling a non-200 response? What i would like it to do is to simply retry if the response is "buffering". That's why i tried the to decorate the decorator as in:


def retry_on_exception_decorator(view):
    def wrapper(*args, **kwargs):
        allowed_retries = 10
        total_retries = 0
        while allowed_retries > total_retries:
            try:
                print('retry decorator active - not retrying')
                return view(*args, **kwargs)
            except:
                total_retries = total_retries + 1
                print('retrying view due to timeout')
                print
    return wrapper

@retry_on_exception_decorator
@permission_required("model.permission", login_url="django_auth_adfs:login")
def the_view(request):

I'll try to enable logging. Should i just do it in settings.py?

jobec commented 4 years ago

Are your users perhaps a member of a lot of groups?

Anyway, I think enabling that logging on a production system might cause other issues. Do you perhaps have the tracebacks? Because, although we receive such a weird responds, it shouldn't cause an internal server error, but rather a page stating permission is denied.

Try enabling this first (the LOGGING part): https://django-auth-adfs.readthedocs.io/en/latest/troubleshooting.html#turn-on-django-debug-logging

JonasKs commented 4 years ago

Can confirm, we got permission denied due to an incomplete token when people were in too many groups, not a 500 as described above. How ever, if that’s the case it can be fixed by changing your WSGI to allow a greater size in the headers, such as this in Gunicorn

pandawankenobi commented 4 years ago

how much would be considered many? 20 is pretty common in my organization.

pandawankenobi commented 4 years ago

as it turns out i have logging enabled (not verbose, if that is nessecary i will enable it).

2019-11-01T23:25:41.208406732Z [pid: 13|app: 0|req: 206/236] 172.16.0.1 () {72 vars in 1506 bytes} [Sat Nov  2 00:25:41 2019] GET /bestil/ => generated 0 bytes in 2 msecs (HTTP/1.1 302) 5 headers in 163 bytes (1 switches on core 0)
2019-11-01T23:25:41.210232637Z 172.16.0.1 - - [01/Nov/2019:23:25:41 +0000] "GET /bestil/ HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36 OPR/64.0.3417.73" "5.186.115.73:1656"
2019-11-01T23:25:41.272752628Z [pid: 14|app: 0|req: 31/237] 172.16.0.1 () {72 vars in 1659 bytes} [Sat Nov  2 00:25:41 2019] GET /oauth2/login?next=/bestil/ => generated 0 bytes in 2 msecs (HTTP/1.1 302) 4 headers in 394 bytes (1 switches on core 0)
2019-11-01T23:25:41.274083332Z 172.16.0.1 - - [01/Nov/2019:23:25:41 +0000] "GET /oauth2/login?next=/bestil/ HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36 OPR/64.0.3417.73" "5.186.115.73:1656"

2019-11-01T23:25:50.105502099Z Received authorization code: censored_code
2019-11-01T23:25:50.107018504Z Getting access token at: https://adfs.censoredorg.dk/adfs/oauth2/token/

2019-11-01T23:26:05.271838808Z Starting new HTTPS connection (1): sentry.io:443
2019-11-01T23:26:05.308712421Z Internal Server Error: /oauth2/callback
2019-11-01T23:26:05.308733321Z Traceback (most recent call last):
2019-11-01T23:26:05.308749121Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
2019-11-01T23:26:05.308753621Z     six.raise_from(e, None)
2019-11-01T23:26:05.308765421Z   File "<string>", line 2, in raise_from
2019-11-01T23:26:05.308769921Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
2019-11-01T23:26:05.308773821Z     httplib_response = conn.getresponse()
2019-11-01T23:26:05.308777421Z   File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/stdlib.py", line 48, in getresponse
2019-11-01T23:26:05.308781321Z     rv = real_getresponse(self, *args, **kwargs)
2019-11-01T23:26:05.308785221Z   File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
2019-11-01T23:26:05.308788921Z     response.begin()
2019-11-01T23:26:05.308792521Z   File "/usr/lib/python3.6/http/client.py", line 297, in begin
2019-11-01T23:26:05.308796221Z     version, status, reason = self._read_status()
2019-11-01T23:26:05.308800021Z   File "/usr/lib/python3.6/http/client.py", line 258, in _read_status
2019-11-01T23:26:05.308803721Z     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
2019-11-01T23:26:05.308807521Z   File "/usr/lib/python3.6/socket.py", line 586, in readinto
2019-11-01T23:26:05.308811221Z     return self._sock.recv_into(b)
2019-11-01T23:26:05.308814921Z   File "/usr/lib/python3.6/ssl.py", line 1009, in recv_into
2019-11-01T23:26:05.308818621Z     return self.read(nbytes, buffer)
2019-11-01T23:26:05.308822221Z   File "/usr/lib/python3.6/ssl.py", line 871, in read
2019-11-01T23:26:05.308825921Z     return self._sslobj.read(len, buffer)
2019-11-01T23:26:05.308829521Z   File "/usr/lib/python3.6/ssl.py", line 631, in read
2019-11-01T23:26:05.308833221Z     v = self._sslobj.read(len, buffer)
2019-11-01T23:26:05.308836921Z socket.timeout: The read operation timed out
2019-11-01T23:26:05.308840521Z 
2019-11-01T23:26:05.308844121Z During handling of the above exception, another exception occurred:
2019-11-01T23:26:05.308847821Z 
2019-11-01T23:26:05.308851321Z Traceback (most recent call last):
2019-11-01T23:26:05.308855021Z   File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2019-11-01T23:26:05.308858821Z     timeout=timeout
2019-11-01T23:26:05.308862421Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
2019-11-01T23:26:05.308866121Z     _stacktrace=sys.exc_info()[2])
2019-11-01T23:26:05.308869721Z   File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 367, in increment
2019-11-01T23:26:05.308873421Z     raise six.reraise(type(error), error, _stacktrace)
2019-11-01T23:26:05.308877121Z   File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
2019-11-01T23:26:05.308882321Z     raise value
2019-11-01T23:26:05.308886021Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2019-11-01T23:26:05.308892821Z     chunked=chunked)
2019-11-01T23:26:05.308897321Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 386, in _make_request
2019-11-01T23:26:05.308901321Z     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
2019-11-01T23:26:05.308905021Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 306, in _raise_timeout
2019-11-01T23:26:05.308908921Z     raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
2019-11-01T23:26:05.308912821Z urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='adfs.censoredorg.dk', port=443): Read timed out. (read timeout=15)
2019-11-01T23:26:05.308916621Z 
2019-11-01T23:26:05.308920021Z During handling of the above exception, another exception occurred:
2019-11-01T23:26:05.308923821Z 
2019-11-01T23:26:05.308927321Z Traceback (most recent call last):
2019-11-01T23:26:05.308930921Z   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
2019-11-01T23:26:05.308934721Z     response = get_response(request)
2019-11-01T23:26:05.308938521Z   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
2019-11-01T23:26:05.308942321Z     response = self.process_exception_by_middleware(e, request)
2019-11-01T23:26:05.308946021Z   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 124, in _get_response
2019-11-01T23:26:05.308949821Z     response = wrapped_callback(request, *callback_args, **callback_kwargs)
2019-11-01T23:26:05.308953621Z   File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 68, in view
2019-11-01T23:26:05.308957421Z     return self.dispatch(request, *args, **kwargs)
2019-11-01T23:26:05.308961121Z   File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 88, in dispatch
2019-11-01T23:26:05.308964921Z     return handler(request, *args, **kwargs)
2019-11-01T23:26:05.308968621Z   File "/usr/local/lib/python3.6/site-packages/django_auth_adfs/views.py", line 34, in get
2019-11-01T23:26:05.308972421Z     user = authenticate(request, authorization_code=code)
2019-11-01T23:26:05.308976021Z   File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/__init__.py", line 73, in authenticate
2019-11-01T23:26:05.308979821Z     user = backend.authenticate(request, **credentials)
2019-11-01T23:26:05.308983621Z   File "/usr/local/lib/python3.6/site-packages/django_auth_adfs/backend.py", line 264, in authenticate
2019-11-01T23:26:05.308987422Z     adfs_response = self.exchange_auth_code(authorization_code, request)
2019-11-01T23:26:05.308991122Z   File "/usr/local/lib/python3.6/site-packages/django_auth_adfs/backend.py", line 28, in exchange_auth_code
2019-11-01T23:26:05.308995322Z     response = provider_config.session.post(provider_config.token_endpoint, data, timeout=settings.TIMEOUT)
2019-11-01T23:26:05.309001722Z   File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 581, in post
2019-11-01T23:26:05.309005622Z     return self.request('POST', url, data=data, json=json, **kwargs)
2019-11-01T23:26:05.309009322Z   File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
2019-11-01T23:26:05.309013122Z     resp = self.send(prep, **send_kwargs)
2019-11-01T23:26:05.309016722Z   File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
2019-11-01T23:26:05.309020522Z     r = adapter.send(request, **kwargs)
2019-11-01T23:26:05.309024122Z   File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 529, in send
2019-11-01T23:26:05.309027822Z     raise ReadTimeout(e, request=request)
2019-11-01T23:26:05.313916837Z requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='adfs.censoredorg.dk', port=443): Read timed out. (read timeout=15)
2019-11-01T23:26:05.316111243Z 172.16.0.1 - - [01/Nov/2019:23:26:05 +0000] "GET /oauth2/callback?code=censored_code&state=censored_state= HTTP/1.1" 500 27 "https://adfs.censoredorg.dk/adfs/oauth2/authorize/?response_type=code&client_id=a7d1fc23-f356-40dc-93f8-6065fc99ef19&resource=a7d1fc23-f356-40dc-93f8-6065fc99ef19&redirect_uri=https%3A%2F%2Fcensored_url.azurewebsites.net%2Foauth2%2Fcallback&state=censored_state%3D&scope=openid" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36 OPR/64.0.3417.73" "5.186.115.73:1656"
2019-11-01T23:26:05.317096046Z [pid: 14|app: 0|req: 32/238] 172.16.0.1 () {80 vars in 3668 bytes} [Sat Nov  2 00:25:49 2019] GET /oauth2/callback?code=censored_code&state=censored_state= => generated 27 bytes in 16158 msecs (HTTP/1.1 500) 4 headers in 126 bytes (1 switches on core 1)
2019-11-01T23:26:05.744857752Z Not Found: /favicon.ico
2019-11-01T23:26:05.746819358Z 172.16.0.1 - - [01/Nov/2019:23:26:05 +0000] "GET /favicon.ico HTTP/1.1" 404 85 "https://censored_url.azurewebsites.net/oauth2/callback?code=censored_code&state=censored_state=" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36 OPR/64.0.3417.73" "5.186.115.73:1656"
2019-11-01T23:26:05.747652061Z [pid: 13|app: 0|req: 207/239] 172.16.0.1 () {74 vars in 2007 bytes} [Sat Nov  2 00:26:05 2019] GET /favicon.ico => generated 85 bytes in 3 msecs (HTTP/1.1 404) 3 headers in 100 bytes (1 switches on core 0)
2019-11-01T23:26:06.028444318Z https://sentry.io:443 "POST /api/censored/store/ HTTP/1.1" 200 41
JonasKs commented 4 years ago

20 is not considered many. You can take the JWT token and calculate the amount of bytes it is. If it's greater than 8000 bytes, you may face some issues. It would how ever happen every time for those users, and not sporadically.

pandawankenobi commented 4 years ago

@JonasKs ok then that is not the issue. The user gets access when retrying.

pandawankenobi commented 4 years ago

@jobec a bit of extra info i upped timeout to 15 just to be sure. no difference it fails after the set timeout parameter, but it does not retry. So it fails in 15 seconds and not, as i expected, in 15 * retries. the behaiviour i had hoped for was that django-auth-adfs would retry if it got a buffering message. It seems to me as if it only retries if no response is given.


asn1crypto | 0.24.0
certifi | 2019.9.11
cffi | 1.11.5
chardet | 3.0.4
cryptography | 2.1.4
django | 2.1.3
django-appconf | 1.0.3
django-auth-adfs | 1.1.2
django-crispy-forms | 1.7.2
django-select2 | 6.3.1
idna | 2.7
pip | 18.1. 
psycopg2 | 2.7.7
pycparser | 2.19
pyjwt | 1.7.1
pytz | 2018.5
requests | 2.20.1
sentry-sdk | 0.7.3
setuptools | 40.6.3
six | 1.12.0
urllib3 | 1.24.1
wheel | 0.32.3
jobec commented 4 years ago

The traceback you gave indicates a totally different problem. Your ADFS server simply didn’t sent anything for 15 seconds. Which is a very long time.

It’s different in that your first stated issue indicated “something” still was sent back.

If the server (or the network in between) misbehaves, that will be handled by the retry mechanism in the requests package. Because the HTTP request never properly finished. On the other hand, if the ADFS server sends back something and completes the HTTP request (not necessarily a nice 200 response), it won’t be retried.

So, what happened in your traceback can’t be fixed, because the retry should have already happened. If the ADFS server sent back a “buffered” response, then I’m very interested in what exactly was the full response. We now only log the response body, not the HTTP status code. Maybe I need to include more details If it really sends back just “buffered” then that might allow the code to retry on that specific situation.

Still, this all very much feels like a misbehaving ADFS server, or the connection towards it. And there isn’t much you can do about it. You can’t keep on retrying and you need to give up at some point. Is that ADFS server close to your server running Django? Or is django an intranet application while ADFS is some cloud hosted service?

Try doing some packet captures and see how the ADFS response times are when the issue occurs. And if you manage to get a traceback of that “buffered” response, post it here also.

jobec commented 4 years ago

One other thing I notice in that traceback, is that sentry_sdk package. Are you sure it isn’t sentry.io that isn’t responding? It looks like that sdk hooks itself inside the http library of python, and maybe, just maybe, it’s that which is causing the timeouts.

Still, doesn’t explain that weird “buffered” response

JonasKs commented 4 years ago

I think you are right on this one:

Starting new HTTPS connection (1): sentry.io:443

The line in question.

pandawankenobi commented 4 years ago

i doubt it is sentry. but i can disable it to check. I run other projects with sentry, and i have never had an issue. also it makes sense for it to run, as it is an error-loggingtool.

I reproduced the error with debug= true and network analysis enabled.

image


Environment:

Request Method: GET
Request URL: https://my-site.azurewebsites.net/oauth2/callback?code=7mfkUeLYjUW2Lr_U7cc4lA.pXUVY9hf1whPASXE7TIiZ0mQ3cQ.TpMNjhVz7o60elue5lFqtUP9dFrcxQcMY5pf1wbURNAlKMLrgwPpF3Z1XbXfUdXNBXxpEbVkx_wu5iyFwADkBtfxf5b4P12YIYiDRHIz9_ziJa65NsMBcUqonjH0wfTHqRQZ1WiARK8clSxqAmpsODT2Z5C93rm0o_nk0gQh73YOkcojD1lT5aoZDYxzId04d4y2Ay7GxIXLGY3IJVkMBwsRNI5kGpwBbUwIF3bT2xPUCC2t89yBSgFvdeWsYte_YZsuGTYRsX3xuqT_AylDwy0Ep5F6WBVfyyAqvb9gCeWPcAOOxs8ch3iYsQq0hZBY3Xlitz5AoIfPP3ZkXLSOBQ&state=L3Zpa2FyL2Jlc3RpbC8=

Django Version: 2.1.3
Python Version: 3.6.5
Installed Applications:
['django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'vikar.apps.VikarConfig',
 'crispy_forms',
 'django.contrib.admin',
 'django_select2',
 'django_auth_adfs']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']

Traceback:

File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py" in _make_request
  377.                 httplib_response = conn.getresponse(buffering=True)

File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/stdlib.py" in getresponse
  48.         rv = real_getresponse(self, *args, **kwargs)

During handling of the above exception (getresponse() got an unexpected keyword argument 'buffering'), another exception occurred:

File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py" in _make_request
  384.                     six.raise_from(e, None)

File "<string>" in raise_from

File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py" in _make_request
  380.                     httplib_response = conn.getresponse()

File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/stdlib.py" in getresponse
  48.         rv = real_getresponse(self, *args, **kwargs)

File "/usr/lib/python3.6/http/client.py" in getresponse
  1331.                 response.begin()

File "/usr/lib/python3.6/http/client.py" in begin
  297.             version, status, reason = self._read_status()

File "/usr/lib/python3.6/http/client.py" in _read_status
  258.         line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")

File "/usr/lib/python3.6/socket.py" in readinto
  586.                 return self._sock.recv_into(b)

File "/usr/lib/python3.6/ssl.py" in recv_into
  1009.             return self.read(nbytes, buffer)

File "/usr/lib/python3.6/ssl.py" in read
  871.             return self._sslobj.read(len, buffer)

File "/usr/lib/python3.6/ssl.py" in read
  631.             v = self._sslobj.read(len, buffer)

During handling of the above exception (The read operation timed out), another exception occurred:

File "/usr/local/lib/python3.6/site-packages/requests/adapters.py" in send
  449.                     timeout=timeout

File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py" in urlopen
  638.                                         _stacktrace=sys.exc_info()[2])

File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py" in increment
  367.                 raise six.reraise(type(error), error, _stacktrace)

File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py" in reraise
  686.         raise value

File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py" in urlopen
  600.                                                   chunked=chunked)

File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py" in _make_request
  386.             self._raise_timeout(err=e, url=url, timeout_value=read_timeout)

File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py" in _raise_timeout
  306.             raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)

During handling of the above exception (HTTPSConnectionPool(host='adfs.where-i-work.dk', port=443): Read timed out. (read timeout=15)), another exception occurred:

File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  34.             response = get_response(request)

File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  126.                 response = self.process_exception_by_middleware(e, request)

File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  124.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py" in view
  68.             return self.dispatch(request, *args, **kwargs)

File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py" in dispatch
  88.         return handler(request, *args, **kwargs)

File "/usr/local/lib/python3.6/site-packages/django_auth_adfs/views.py" in get
  34.         user = authenticate(request, authorization_code=code)

File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/__init__.py" in authenticate
  73.             user = backend.authenticate(request, **credentials)

File "/usr/local/lib/python3.6/site-packages/django_auth_adfs/backend.py" in authenticate
  264.         adfs_response = self.exchange_auth_code(authorization_code, request)

File "/usr/local/lib/python3.6/site-packages/django_auth_adfs/backend.py" in exchange_auth_code
  28.         response = provider_config.session.post(provider_config.token_endpoint, data, timeout=settings.TIMEOUT)

File "/usr/local/lib/python3.6/site-packages/requests/sessions.py" in post
  581.         return self.request('POST', url, data=data, json=json, **kwargs)

File "/usr/local/lib/python3.6/site-packages/requests/sessions.py" in request
  533.         resp = self.send(prep, **send_kwargs)

File "/usr/local/lib/python3.6/site-packages/requests/sessions.py" in send
  646.         r = adapter.send(request, **kwargs)

File "/usr/local/lib/python3.6/site-packages/requests/adapters.py" in send
  529.                 raise ReadTimeout(e, request=request)

Exception Type: ReadTimeout at /oauth2/callback
Exception Value: HTTPSConnectionPool(host='adfs.where-i-work.dk', port=443): Read timed out. (read timeout=15)
jobec commented 4 years ago

Can you enable that urllib3 debugging as mentioned in the second message and provide the corresponding console output/trace? That should show the retries and some more details about what's going on.

pandawankenobi commented 4 years ago

in settings.py?

pandawankenobi commented 4 years ago

ok i think i managed to get it. the error takes long to reproduce, this time i encountered it by refreshing the site while being logged in.:

2019-11-02T23:27:15.676125720Z retry decorator active - not retrying
2019-11-02T23:27:16.646217389Z 172.16.0.1 - - [02/Nov/2019:23:27:16 +0000] "GET /vikar/bestil/ HTTP/1.1" 302 0 "https://adfs.my-job.dk/adfs/oauth2/authorize/?response_type=code&client_id=a7d1fc23-f356-40dc-93f8-6065fc99ef19&resource=a7d1fc23-f356-40dc-93f8-6065fc99ef19&redirect_uri=https%3A%2F%2Fmy-site.azurewebsites.net%2Foauth2%2Fcallback&state=L3Zpa2FyL2Jlc3RpbC8%3D&scope=openid&client-request-id=5c74e60c-c78d-4657-8e9b-0080010000bf&RedirectToIdentityProvider=AD+AUTHORITY" "Mozilla/5.0 (iPad; CPU OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1" "5.186.115.73:1591"
2019-11-02T23:27:16.648408295Z [pid: 13|app: 0|req: 58/94] 172.16.0.1 () {70 vars in 1890 bytes} [Sun Nov  3 00:27:15 2019] GET /vikar/bestil/ => generated 0 bytes in 973 msecs (HTTP/1.1 302) 5 headers in 240 bytes (2 switches on core 0)
2019-11-02T23:27:16.789968214Z 172.16.0.1 - - [02/Nov/2019:23:27:16 +0000] "GET /oauth2/login?next=/vikar/bestil/ HTTP/1.1" 302 0 "https://adfs.my-job.dk/adfs/oauth2/authorize/?response_type=code&client_id=a7d1fc23-f356-40dc-93f8-6065fc99ef19&resource=a7d1fc23-f356-40dc-93f8-6065fc99ef19&redirect_uri=https%3A%2F%2Fmy-site.azurewebsites.net%2Foauth2%2Fcallback&state=L3Zpa2FyL2Jlc3RpbC8%3D&scope=openid&client-request-id=5c74e60c-c78d-4657-8e9b-0080010000bf&RedirectToIdentityProvider=AD+AUTHORITY" "Mozilla/5.0 (iPad; CPU OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1" "5.186.115.73:1591"
2019-11-02T23:27:16.791820319Z [pid: 13|app: 0|req: 59/95] 172.16.0.1 () {70 vars in 1921 bytes} [Sun Nov  3 00:27:16 2019] GET /oauth2/login?next=/vikar/bestil/ => generated 0 bytes in 2 msecs (HTTP/1.1 302) 4 headers in 402 bytes (1 switches on core 1)
2019-11-02T23:27:18.220565744Z Received authorization code: 7mfkUeLYjUW2Lr_U7cc4lA.xzRFM-xf1whmAZHdhVkKzZ53M3k.iMNjf1xCSRWyY-YPAYdht-XZ6PcpfYjHgjDmNasUwmToFP5Xzx9skkdpXjxn3dxiG8wQW2_sTC7ScdpP7R0OYPv7wnaGmS4DQreos3NzKsrqYtbcxzOFzFFne3gtgyxhc5WWZX8r-JU-a3Tl_L3msVCmOjNiCxd9LLYEu572PpxOY9x_kvDB4dC032sK9ChQQSWSbolZFjWb2yvkRW_1tTzJwWiX7CFvJ5O4_BRoF8cHANSlTBp0cvcXF_e1RSonflFViEEI7WSixgEed08yN8J2ZXP6aBKngGrwLjVMfwXjdIbaFt7bWmNNtUKWXzo-vySmZHoM0quy0yMMcdM5sA
2019-11-02T23:27:18.222404649Z Getting access token at: https://adfs.my-job.dk/adfs/oauth2/token/
2019-11-02T23:27:18.225371158Z send: b'POST /adfs/oauth2/token/ HTTP/1.1\r\nHost: adfs.my-job.dk\r\nUser-Agent: python-requests/2.20.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 549\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n'
2019-11-02T23:27:18.226233660Z send: b'grant_type=authorization_code&client_id=a7d1fc23-f356-40dc-93f8-6065fc99ef19&redirect_uri=https%3A%2F%2Fmy-site.azurewebsites.net%2Foauth2%2Fcallback&code=7mfkUeLYjUW2Lr_U7cc4lA.xzRFM-xf1whmAZHdhVkKzZ53M3k.iMNjf1xCSRWyY-YPAYdht-XZ6PcpfYjHgjDmNasUwmToFP5Xzx9skkdpXjxn3dxiG8wQW2_sTC7ScdpP7R0OYPv7wnaGmS4DQreos3NzKsrqYtbcxzOFzFFne3gtgyxhc5WWZX8r-JU-a3Tl_L3msVCmOjNiCxd9LLYEu572PpxOY9x_kvDB4dC032sK9ChQQSWSbolZFjWb2yvkRW_1tTzJwWiX7CFvJ5O4_BRoF8cHANSlTBp0cvcXF_e1RSonflFViEEI7WSixgEed08yN8J2ZXP6aBKngGrwLjVMfwXjdIbaFt7bWmNNtUKWXzo-vySmZHoM0quy0yMMcdM5sA'

2019-11-02T23:27:22.781586130Z 172.16.0.1 - - [02/Nov/2019:23:27:22 +0000] "GET / HTTP/1.1" 200 4214 "-" "AlwaysOn" "[::1]:18010"
2019-11-02T23:27:22.783743236Z [pid: 13|app: 0|req: 60/96] 172.16.0.1 () {58 vars in 858 bytes} [Sun Nov  3 00:27:22 2019] GET / => generated 4214 bytes in 4 msecs (HTTP/1.1 200) 4 headers in 124 bytes (1 switches on core 1)

2019-11-02T23:27:33.394698912Z Starting new HTTPS connection (1): sentry.io:443
2019-11-02T23:27:33.396177917Z Starting new HTTPS connection (1): sentry.io:443
2019-11-02T23:27:33.512181460Z Internal Server Error: /oauth2/callback
2019-11-02T23:27:33.512217160Z Traceback (most recent call last):
2019-11-02T23:27:33.512223160Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
2019-11-02T23:27:33.512238660Z     six.raise_from(e, None)
2019-11-02T23:27:33.512243460Z   File "<string>", line 2, in raise_from
2019-11-02T23:27:33.512248160Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
2019-11-02T23:27:33.512252560Z     httplib_response = conn.getresponse()
2019-11-02T23:27:33.512256560Z   File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/stdlib.py", line 48, in getresponse
2019-11-02T23:27:33.512260860Z     rv = real_getresponse(self, *args, **kwargs)
2019-11-02T23:27:33.512264960Z   File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
2019-11-02T23:27:33.512269260Z     response.begin()
2019-11-02T23:27:33.512273260Z   File "/usr/lib/python3.6/http/client.py", line 297, in begin
2019-11-02T23:27:33.512277460Z     version, status, reason = self._read_status()
2019-11-02T23:27:33.512281560Z   File "/usr/lib/python3.6/http/client.py", line 258, in _read_status
2019-11-02T23:27:33.512285860Z     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
2019-11-02T23:27:33.512289960Z   File "/usr/lib/python3.6/socket.py", line 586, in readinto
2019-11-02T23:27:33.512294060Z     return self._sock.recv_into(b)
2019-11-02T23:27:33.512298160Z   File "/usr/lib/python3.6/ssl.py", line 1009, in recv_into
2019-11-02T23:27:33.512302460Z     return self.read(nbytes, buffer)
2019-11-02T23:27:33.512306560Z   File "/usr/lib/python3.6/ssl.py", line 871, in read
2019-11-02T23:27:33.512310660Z     return self._sslobj.read(len, buffer)
2019-11-02T23:27:33.512314760Z   File "/usr/lib/python3.6/ssl.py", line 631, in read
2019-11-02T23:27:33.512318960Z     v = self._sslobj.read(len, buffer)
2019-11-02T23:27:33.512323060Z socket.timeout: The read operation timed out
2019-11-02T23:27:33.512327160Z 
2019-11-02T23:27:33.512331060Z During handling of the above exception, another exception occurred:
2019-11-02T23:27:33.512335260Z 
2019-11-02T23:27:33.512339160Z Traceback (most recent call last):
2019-11-02T23:27:33.512343360Z   File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2019-11-02T23:27:33.512347660Z     timeout=timeout
2019-11-02T23:27:33.512351660Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
2019-11-02T23:27:33.512355860Z     _stacktrace=sys.exc_info()[2])
2019-11-02T23:27:33.512359960Z   File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 367, in increment
2019-11-02T23:27:33.512364160Z     raise six.reraise(type(error), error, _stacktrace)
2019-11-02T23:27:33.512368260Z   File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
2019-11-02T23:27:33.512372560Z     raise value
2019-11-02T23:27:33.512376560Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2019-11-02T23:27:33.512383160Z     chunked=chunked)
2019-11-02T23:27:33.512388460Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 386, in _make_request
2019-11-02T23:27:33.512393060Z     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
2019-11-02T23:27:33.512397260Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 306, in _raise_timeout
2019-11-02T23:27:33.512401460Z     raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
2019-11-02T23:27:33.512405760Z urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='adfs.my-job.dk', port=443): Read timed out. (read timeout=15)
2019-11-02T23:27:33.512409960Z 
2019-11-02T23:27:33.512413860Z During handling of the above exception, another exception occurred:
2019-11-02T23:27:33.512417960Z 
2019-11-02T23:27:33.512421860Z Traceback (most recent call last):
2019-11-02T23:27:33.512425860Z   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
2019-11-02T23:27:33.512430060Z     response = get_response(request)
2019-11-02T23:27:33.512434060Z   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
2019-11-02T23:27:33.512438260Z     response = self.process_exception_by_middleware(e, request)
2019-11-02T23:27:33.512442360Z   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 124, in _get_response
2019-11-02T23:27:33.512446561Z     response = wrapped_callback(request, *callback_args, **callback_kwargs)
2019-11-02T23:27:33.512450661Z   File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 68, in view
2019-11-02T23:27:33.512454861Z     return self.dispatch(request, *args, **kwargs)
2019-11-02T23:27:33.512458961Z   File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 88, in dispatch
2019-11-02T23:27:33.512463161Z     return handler(request, *args, **kwargs)
2019-11-02T23:27:33.512467161Z   File "/usr/local/lib/python3.6/site-packages/django_auth_adfs/views.py", line 34, in get
2019-11-02T23:27:33.512471361Z     user = authenticate(request, authorization_code=code)
2019-11-02T23:27:33.512475861Z   File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/__init__.py", line 73, in authenticate
2019-11-02T23:27:33.512480261Z     user = backend.authenticate(request, **credentials)
2019-11-02T23:27:33.512484261Z   File "/usr/local/lib/python3.6/site-packages/django_auth_adfs/backend.py", line 264, in authenticate
2019-11-02T23:27:33.512488461Z     adfs_response = self.exchange_auth_code(authorization_code, request)
2019-11-02T23:27:33.512492661Z   File "/usr/local/lib/python3.6/site-packages/django_auth_adfs/backend.py", line 28, in exchange_auth_code
2019-11-02T23:27:33.512497361Z     response = provider_config.session.post(provider_config.token_endpoint, data, timeout=settings.TIMEOUT)
2019-11-02T23:27:33.512504261Z   File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 581, in post
2019-11-02T23:27:33.512508761Z     return self.request('POST', url, data=data, json=json, **kwargs)
2019-11-02T23:27:33.512512861Z   File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
2019-11-02T23:27:33.512517161Z     resp = self.send(prep, **send_kwargs)
2019-11-02T23:27:33.512521161Z   File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
2019-11-02T23:27:33.512525561Z     r = adapter.send(request, **kwargs)
2019-11-02T23:27:33.512529561Z   File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 529, in send
2019-11-02T23:27:33.512533761Z     raise ReadTimeout(e, request=request)
2019-11-02T23:27:33.517739476Z requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='adfs.my-job.dk', port=443): Read timed out. (read timeout=15)
2019-11-02T23:27:33.518596879Z Internal Server Error: /oauth2/callback
2019-11-02T23:27:33.518613879Z Traceback (most recent call last):
2019-11-02T23:27:33.518619379Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
2019-11-02T23:27:33.518624279Z     six.raise_from(e, None)
2019-11-02T23:27:33.518628779Z   File "<string>", line 2, in raise_from
2019-11-02T23:27:33.518644479Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
2019-11-02T23:27:33.518651979Z     httplib_response = conn.getresponse()
2019-11-02T23:27:33.518657179Z   File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/stdlib.py", line 48, in getresponse
2019-11-02T23:27:33.518661979Z     rv = real_getresponse(self, *args, **kwargs)
2019-11-02T23:27:33.518666779Z   File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
2019-11-02T23:27:33.518671479Z     response.begin()
2019-11-02T23:27:33.518675979Z   File "/usr/lib/python3.6/http/client.py", line 297, in begin
2019-11-02T23:27:33.518680579Z     version, status, reason = self._read_status()
2019-11-02T23:27:33.518685179Z   File "/usr/lib/python3.6/http/client.py", line 258, in _read_status
2019-11-02T23:27:33.518689979Z     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
2019-11-02T23:27:33.518694679Z   File "/usr/lib/python3.6/socket.py", line 586, in readinto
2019-11-02T23:27:33.518699379Z     return self._sock.recv_into(b)
2019-11-02T23:27:33.518703879Z   File "/usr/lib/python3.6/ssl.py", line 1009, in recv_into
2019-11-02T23:27:33.518708579Z     return self.read(nbytes, buffer)
2019-11-02T23:27:33.518713079Z   File "/usr/lib/python3.6/ssl.py", line 871, in read
2019-11-02T23:27:33.518717679Z     return self._sslobj.read(len, buffer)
2019-11-02T23:27:33.518729279Z   File "/usr/lib/python3.6/ssl.py", line 631, in read
2019-11-02T23:27:33.518734379Z     v = self._sslobj.read(len, buffer)
2019-11-02T23:27:33.518738979Z socket.timeout: The read operation timed out
2019-11-02T23:27:33.518743579Z 
2019-11-02T23:27:33.518748079Z During handling of the above exception, another exception occurred:
2019-11-02T23:27:33.518752679Z 
2019-11-02T23:27:33.518757079Z Traceback (most recent call last):
2019-11-02T23:27:33.518761679Z   File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2019-11-02T23:27:33.518766479Z     timeout=timeout
2019-11-02T23:27:33.518770879Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
2019-11-02T23:27:33.518775679Z     _stacktrace=sys.exc_info()[2])
2019-11-02T23:27:33.518780179Z   File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 367, in increment
2019-11-02T23:27:33.518784979Z     raise six.reraise(type(error), error, _stacktrace)
2019-11-02T23:27:33.518789479Z   File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
2019-11-02T23:27:33.518794279Z     raise value
2019-11-02T23:27:33.518798779Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2019-11-02T23:27:33.518803579Z     chunked=chunked)
2019-11-02T23:27:33.518808879Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 386, in _make_request
2019-11-02T23:27:33.518813979Z     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
2019-11-02T23:27:33.518818679Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 306, in _raise_timeout
2019-11-02T23:27:33.518823379Z     raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
2019-11-02T23:27:33.518828279Z urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='adfs.my-job.dk', port=443): Read timed out. (read timeout=15)
2019-11-02T23:27:33.518832979Z 
2019-11-02T23:27:33.518837279Z During handling of the above exception, another exception occurred:
2019-11-02T23:27:33.518841979Z 
2019-11-02T23:27:33.518846379Z Traceback (most recent call last):
2019-11-02T23:27:33.518850979Z   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
2019-11-02T23:27:33.518855679Z     response = get_response(request)
2019-11-02T23:27:33.518860279Z   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
2019-11-02T23:27:33.518864979Z     response = self.process_exception_by_middleware(e, request)
2019-11-02T23:27:33.518869580Z   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 124, in _get_response
2019-11-02T23:27:33.518874480Z     response = wrapped_callback(request, *callback_args, **callback_kwargs)
2019-11-02T23:27:33.518881580Z   File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 68, in view
2019-11-02T23:27:33.518886580Z     return self.dispatch(request, *args, **kwargs)
2019-11-02T23:27:33.518891180Z   File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 88, in dispatch
2019-11-02T23:27:33.518895980Z     return handler(request, *args, **kwargs)
2019-11-02T23:27:33.518900580Z   File "/usr/local/lib/python3.6/site-packages/django_auth_adfs/views.py", line 34, in get
2019-11-02T23:27:33.518905280Z     user = authenticate(request, authorization_code=code)
2019-11-02T23:27:33.518909780Z   File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/__init__.py", line 73, in authenticate
2019-11-02T23:27:33.518914680Z     user = backend.authenticate(request, **credentials)
2019-11-02T23:27:33.518919280Z   File "/usr/local/lib/python3.6/site-packages/django_auth_adfs/backend.py", line 264, in authenticate
2019-11-02T23:27:33.518924080Z     adfs_response = self.exchange_auth_code(authorization_code, request)
2019-11-02T23:27:33.518928680Z   File "/usr/local/lib/python3.6/site-packages/django_auth_adfs/backend.py", line 28, in exchange_auth_code
2019-11-02T23:27:33.518934080Z     response = provider_config.session.post(provider_config.token_endpoint, data, timeout=settings.TIMEOUT)
2019-11-02T23:27:33.518939080Z   File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 581, in post
2019-11-02T23:27:33.518943780Z     return self.request('POST', url, data=data, json=json, **kwargs)
2019-11-02T23:27:33.518948480Z   File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
2019-11-02T23:27:33.518953180Z     resp = self.send(prep, **send_kwargs)
2019-11-02T23:27:33.518957780Z   File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
2019-11-02T23:27:33.518962480Z     r = adapter.send(request, **kwargs)
2019-11-02T23:27:33.518967180Z   File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 529, in send
2019-11-02T23:27:33.518972080Z     raise ReadTimeout(e, request=request)
2019-11-02T23:27:33.525582999Z requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='adfs.my-job.dk', port=443): Read timed out. (read timeout=15)
2019-11-02T23:27:33.533689723Z 172.16.0.1 - - [02/Nov/2019:23:27:33 +0000] "GET /oauth2/callback?code=7mfkUeLYjUW2Lr_U7cc4lA.xzRFM-xf1whmAZHdhVkKzZ53M3k.iMNjf1xCSRWyY-YPAYdht-XZ6PcpfYjHgjDmNasUwmToFP5Xzx9skkdpXjxn3dxiG8wQW2_sTC7ScdpP7R0OYPv7wnaGmS4DQreos3NzKsrqYtbcxzOFzFFne3gtgyxhc5WWZX8r-JU-a3Tl_L3msVCmOjNiCxd9LLYEu572PpxOY9x_kvDB4dC032sK9ChQQSWSbolZFjWb2yvkRW_1tTzJwWiX7CFvJ5O4_BRoF8cHANSlTBp0cvcXF_e1RSonflFViEEI7WSixgEed08yN8J2ZXP6aBKngGrwLjVMfwXjdIbaFt7bWmNNtUKWXzo-vySmZHoM0quy0yMMcdM5sA&state=L3Zpa2FyL2Jlc3RpbC8= HTTP/1.1" 500 219442 "https://adfs.my-job.dk/adfs/oauth2/authorize/?response_type=code&client_id=a7d1fc23-f356-40dc-93f8-6065fc99ef19&resource=a7d1fc23-f356-40dc-93f8-6065fc99ef19&redirect_uri=https%3A%2F%2Fmy-site.azurewebsites.net%2Foauth2%2Fcallback&state=L3Zpa2FyL2Jlc3RpbC8%3D&scope=openid&client-request-id=5c74e60c-c78d-4657-8e9b-0080010000bf&RedirectToIdentityProvider=AD+AUTHORITY" "Mozilla/5.0 (iPad; CPU OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1" "5.186.115.73:1591"
2019-11-02T23:27:33.535484529Z [pid: 14|app: 0|req: 37/97] 172.16.0.1 () {70 vars in 3557 bytes} [Sun Nov  3 00:27:17 2019] GET /oauth2/callback?code=7mfkUeLYjUW2Lr_U7cc4lA.xzRFM-xf1whmAZHdhVkKzZ53M3k.iMNjf1xCSRWyY-YPAYdht-XZ6PcpfYjHgjDmNasUwmToFP5Xzx9skkdpXjxn3dxiG8wQW2_sTC7ScdpP7R0OYPv7wnaGmS4DQreos3NzKsrqYtbcxzOFzFFne3gtgyxhc5WWZX8r-JU-a3Tl_L3msVCmOjNiCxd9LLYEu572PpxOY9x_kvDB4dC032sK9ChQQSWSbolZFjWb2yvkRW_1tTzJwWiX7CFvJ5O4_BRoF8cHANSlTBp0cvcXF_e1RSonflFViEEI7WSixgEed08yN8J2ZXP6aBKngGrwLjVMfwXjdIbaFt7bWmNNtUKWXzo-vySmZHoM0quy0yMMcdM5sA&state=L3Zpa2FyL2Jlc3RpbC8= => generated 219442 bytes in 16103 msecs (HTTP/1.1 500) 4 headers in 130 bytes (1 switches on core 0)
2019-11-02T23:27:33.747949057Z send: b'POST /api/1401487/store/ HTTP/1.1\r\nHost: sentry.io\r\nAccept-Encoding: identity\r\nContent-Length: 8635\r\nX-Sentry-Auth: Sentry sentry_key=54c91b17a76e45599d152f53f9f8ce9c, sentry_version=7, sentry_client=sentry.python/0.7.3\r\nContent-Type: application/json\r\nContent-Encoding: gzip\r\n\r\n'
2019-11-02T23:27:33.749163560Z send: b'\x1f\x8b\x08\x00\xe5\x10\xbe]\x02\xff\xed=\t{\xdaH\xb2\x7f\xa5\xd7\xf92\xc0,\xc8\x9c\x06{\x87\x99\xe7q\xec\xc4\x13_k\xec8\x87\xf3\x14!5X\xb1\x90\x18\xb50&\xd9\xfc\xf7WU\xdd\x12\x92\x10\x87\x93Lv\x92\x87w\xbe,\xea\xa3\xfa\xaa\xae\xab\xab\xab?n8\xfc\x8e;\x1b;l\x83\xfb\xbe\xe7o\x14\xe1\xc7\xbd\xc9\x87\x81\xed\xb9\x90\xfaq\xe3\xcepF\\\xc0\xcf7\x1f7\x06\x9e5r8\x16\xf6\xf9\x9f\x90\x1a\x08-*,\xb0j0\x19R\xf697\xac\x0b{\xc0\xbdQ\x80\xc9\x04\x03\xd3\x9f]\\\x9cu\xf6<\xd7\xe5&\xd69\xf3<\'\x7f\xe3\x89\xa0\x9d3\xac\x9e\xd0\x0c\xc3\xbf\x19\x89[o0\x18\xb9\\\xb3nsE6\xf4\xfc\xa0]\xaf\xd7\n;\x0c\x81\xb2\x00\xa0Z\x0c\xe0j,\xef\x87\t\xf0\xd9\xae4\n\xd8\xd4\x80\x9b7\x86k\x8b\x01u>\xec\x8f\xf5\xdep\xfb\x1e\xe6C\xa6\xe5p\x0b\x12{\x86#\xf8\'H\x12\x81a\xde\x06\xbear\xaa\xd3\xf3\x8dA8\xe0\x9e\xedp\x17>\xa706M\xcf\xe7\x9b\x12\x8a/6\xa3\xe1k\xc3\t\x827\xbaB\x1f\x1a\xc1\rV\xd8\x1c\t\x7f\xd3\xf1L\xc3\xd9t\xec\xee\xe6p\x12\xdcxnM\xdb\xda\x14v\xc0KCh\xd4\xe8s\xb1\xb9\x12\xdc\xde\xc85\xd5\x92l\xd80}\xb4P\xd3\xe5\x9004\x84\xa1\x850\xa6K\x83e\x1d\xdb\xe5\xae\x07eku\xf8\x1a\xfa\\7=7\xe0\xf7\x01\x0et\x83\xc1\x9fi\xb8\xcc\xe7\xce\x84y.\xeb\xf3 \xb0\xdd>3 E\x0caq9\xb3]\x11\xe0|{=\x06\x05\xa7\xfdC\xe0X\xfdz\x03\xff\x17~\xfd\xcf\xd87\x86"\x0fp\xf4\x10B!\xcc\xb3x\x8f\xd1\x18\xf2\n\x8b\n;a\x16\xfe\x05\xfedg\xe3-$\xa8\x0e\xea\xd8u\x1c$\x8b\xfdE\xddj\xb3x\x1b\x11D\x048\x04\xc4\x9a\x19%\xfe\xc9\xce\xb3\xfdp\x0c\xcc\x10\x98\x96\xe8E\xaa\x91\xf0\xa7\xde\xf3|=\x1a|\xd8\\\x11\xab\x17\xe2\xd5}\x1e\x8c|7\xaa\x16f\xa9\xe4h\x05i\x9cw\x86/\x08\xf3\x00Hj\xf3\xe4c\xbf\xf7q\x7f\xe6\xaf\xff\xe2=t\xbdQ,\x14i(jl\xd8\xa3_\xae:O\x0f\xcf\xe5\xf7\x0e{\xba\x7f\xc1r\x9b\x9e1\nn\xaa\x9b\x80\xddN\x17P\xf97\xd3\xb3x\xbb9\xe8\xdd^\xf2\xa3W\xef/\xaf\xaaG\xbe~\xd94\xcd\xba\xb3\xab\xdd\x7f8?8.\xdd\xf7*\xe3\x9b\xc1\xee\xebg\xd6\xcd\x8b\xdb\xe7\x1f^7j\xc7\xb5[\xcd>>y\xdf\xab\xdc\xefu\xce\xaf&\xafJ\xaf\xcev_Y7A\xe9\xe5\xeb\xad3s\xd8{\xf5\xfeY\xff\xfd\x93\xc1\x89!.\xc7\x83\x0b\xef\xe0\xac\xf1\xf2\xc3\xfd\xb6\xb8\xbd\xb5\x86/\xdf\xdf\xbb5\xeb\xde~\xda\x1a\xff\xfb\xaa\xaa\x8b\x8b\xbdf\xc7\xb4\x86g\xcd\xf3\xf2\xe9\xab\xb3\xbb\xe6\xd85\x9e\x0e:\xf5\'\xff\xf6\xb9\'j\'\x1f\x9e\x0b\xff\xcfWA\xd7\xbc\xffpz\xf0\xe1\xe0\xc0\xe5\xb5~\xd0\x9f\xdc\xdf\x98\x8d\xab\xab\xd7/[~\xe9\x8f\xcb\x92Q\xbbp\xf4\xa3\xda@\xbc\xd8\x1b\x9c\xbe?\xb1\xf7\xee\xad\xed\xa3\xa3W\xfb\xa3F\xb3z6\xbc?}\xb5}\xaf\xdf\xde=\xf9\xbdn\xed\x95kU\xf1|{\xef\xe6\xdf\xff\xee\\u\xba\x9e\xf3\xfa\xe0\xfdU\xb7:\xb9\xbb=\xbf\xd2+\xc1\xc5\x87?\xc6W\xf6\xcb\xe6\xde\xc1\xdd\x1f\x8d\xd3\xba\xfe\xfb\xb9w\xd02\x9f\xed\x9et\x9c\x8b\xdf\x87e\xf3\xce|y\xa0\xf3\xcay\xc7s{\xce\xc1\x0b{\x7f\xff\xb0y\xd5\xb1\xef\xfb\xfb\xdc*\xb7&\'\xad?\xaa\xaf_\x9em\x19\xbf?w\xfbO\xfd\xf1\xd1\xfb\x17\xc7\xbd\xf1\xcb\xf7\xd6a\xd78\x08\x9a\xdd\xab\xc1\xc9Ip\xf9\xfc\xea\xe5\x07\xaft7\xe9\x0c^?\xf3\x8e\xcb\x7f\x8e&\xe5\xc9\xf1\xb1i\x1d7\xc4\xeeO@\xd4\x02\xde>\xaa\xbd\x1e\x1a\xd5\x83\xc9Q\xf5\x0f\xc7\xac\x9d\x0f\xbb{\xadv\xeeW\\\xdd\xf8\xae\xa1%\xeez#\xd7b\x03\x0eD\xcab\xbf\x1b\x82?\x93tD\xd3\xe3Eq\xfb\xff\x92Im\xc6\xa2ok\x88\'\xaa\x1e\xf3\xba\xef\x01C\x99\x11\xb0\xf2}\xb3\xd7\xadW\x8cr\xbdU\xed\x96\x7f\xfdu\x03\xa9\xae\xed\xea\xc6p\x08M\x07\xfe\x08\xc9\xf0\n\xc4\xb6\x0b\xbd\xfa\xcat6\x062Nb\x13c^\x81\xd4"\x98\x04\x95\xadT\xb7\xe6\x90Y\xfc\xb3{1\xba*\xd8\x89\xe7\xf2\x19\xe2\x83\x04t\xc8-=\xdc`@\x84\x04wz\xda\xc0\xb8\xe5\xfa\x9d\xcd\xc7\xba\x11x\x03\xdb\xcc\x87\x05\ni\x08DHSi)\xb2\x96ndJ\xd0~\x0e\x93t\xc3\xef\x0b\xf8\x9e&\xdc\x8e1i\xa6\xb9L\xaa\xba\x9c\x92\xa7zDc\x1c\xfa\x9e\xc9\x85\x98\x12Z\xbd;\xd1\x07\xb6\x05\x93\rm\xf3</\xb2\x85\x84>\xde\xb5Gl\xcf\x1b\x0c\x1d\xc3vq\xda\x83\x1b\xcep\xf2\x14%\x062\x88\x93\xcf\xf2\x063\x81bB\xa7I\n*hq\x08\xab\xac\x16\x94\xb1\x05\xb2H\xc35y\xb4$E\x86\x02\x88\xd0\x0e\x14j]\xc0\x17\x10a\xea\xd5\xc1\xef/\xb2\x16\x87V\x16w\x01\xccE\x08F\xd3)E\xd7\x93\xec"\xbdx\xb4\x8dC,f\xa7\xbbH\x9f\xf7T\xde\x0b\x1c\xf3t+n7k\xbcY+\x13%\x98\xce\xab.w\xff,9\xd8\x13~\x0f!\x1cGE\xa3%\xa2\xc9\x8c\x11\x85)4\xcd\x84Z\xdal\xd5Y\xc2\xb0\xdd\xaaX\xf5&\x12\x06\xc4\x95$\x9a\xe1H?\xc5\x93U\xe2\x9b\xb7\xb1\xc4\xcf\x1c9\xac\xaa\xe7\xdcq_\x1f\x18\x81y#Y\xafL9\xc6\x84<Bl\xcbq\xe9\xc8\xedtb\xb08b\xa1\xcd6Rd\xd8\xb5v\xbePd\xb2\xebm\xec\xf8\xc8wh\xf1\xdaS\x9c\x80U\xa3$\xd1~\x93KC\xcf\xc1\xb0(\x0f\xc8\xd6\x9c\x02\x85x\xd7i\xe4\x97\xe7Ga\xc7Yn\xcc\xbb:,\x0eN\xb1\x06\x8d\x8b\x1c\xcb\x13\xc6\xe2?\x05\x96\xfb\xdf\xcd\\8\xf6\x88\xf6c\xd6\x9a\xe3\xff\xe8\x1c\x1f)+-\xed\x17\xb2\xef\xff\x17\xdc\xbb\xbe\x80{\x87\x7f]\x90\x9doI\x8a\x7f \xb3\xf8J\xac\xfd\x81\x9c\xf5Ky\xfd\\uj\x01\xf3/~u^\xff\rX\xfb\x9a\xc9\xae\x99\xec\x9a\xc9\xae\x99\xec\xf7\xc1di\xa7l\xf6\xb9\xcb}\xdb\xfc:Lv.\xc88\x93\xc5B\x99\xbcUn]U{\x96\xb7n\xb5\x96\xb0V\xa0\xc97\x860\x82\xc0\xcf\xe3l\x16Y\x0e\xd8y\xae\xc0`\xca\x98\xeb\x05\xe9\xcc\x1bnX\xb9B&\xab!\xfe\x82\xf9!\xaf\x01@\xe9r\x94\xae\xf6$\xd9\xf9\xe4\xee\xcc*\x85\xe4\x06\x8a\x10m\xcc\xca\x97\x04\tJ(\xa2\xba\xdc\x86I\xc6@\xaak\xd9@\x8c\x90\x1cN\x99r\xc8\x8bWa\xc18\xe74\xf1\xba\xe9\x18\x02\xfb`:\x89NN\x0b\xd8\xae\x1dD=\x9d~\xcc\xf2\xd6\x00\xa4\x11"\x934\xf3\x96g\x8a\xc0G\x9bp\xcf\xf7\x06\x8c\x9a\x89\x97\x1f\r-\xd8\x19\xbad\x97~\xfe\x8eh5t\xa2\xa8r,\xa0\xd9\x85$kM\xed\x95\x15\xa8\xff\xec~i\x95\xcd\x06\xb06\xda{I^\x16gak\x9a\xfb#\xd3\xdc\x18\x12\x87r\x8c#h\xad\xe5f\xc8\xad\x8c_\xb9\xef\x80\xee\x86\x94\xe2shok\x11\xed}\x14\x12\xbfP \xb5\x85\x9b\x0b\xf04\x08%k\x98\x10\xdf\xbb\x03\xa1\xda\xb1E0+CSEMV\xd4\x1co\x0c$\xa0\x00\xd4E\x91\xe0 \x18*\xb1W\xcad3\xd4ZqHy\x9a\x13\xa3\xee\xd9\x80\x8b\x19`\xbd@\x87\x85\x84|+\xa1,qG\xf0\x05\xad-\x82\xb3\x90~+\xda\xad =\x90j\xc7\x0f\xc2\xe64\x9e\x1c\xff,\xdc\xc4\x98\x1c\xaf\xdf\xe7\xbe\x069.\x10\xe8|z\xb8\xb9c\xb9\x9e\'\xc0<w%x\x96\x7f\x0c0\xd8c\x91K\xcf\xf1\xf4\x1bq\xb88k\xe4\r|\xa3\xfd1\x87{r$t\xa4\x94\xb9\x1dV/7\x80\x13\xab\x8a\xf0\xad~}J\xd2{5Y\xb3\x9a\xcf\xecFDN\x1dSx\xbe\x885\xacy\xc3Z\x1e\xff:2\xc6\xaa\xbca\xda\x8a$\xe9_\xca\x18\xe6\xc0\x8bs\x05%\xd9\xa6\x19Bz\xbc\xaby\x02$\x88\x07YM\x80B\t\x01=\x82\xbd}\r\x9a,C\xb0\x9eo\x7f0\xc8\x16\x80{\x82\x8d\r\xc1\x90E\xd8\x16\xb7\xb4\xeb\x8d\x19\xca\x01s%\x89F\xbb^.\xcf\x98r|n\xd9>L\xbb\x1exS)\\\x83-\x88\x94 \x7f\xbdA\xeb|\xbdQ\x88\x0e\xcd\xe7\x11\xe6\x91 \xc2\x8e\x1d\xe4n`\x9bPmJ\x9d\x13\xdd&\xe2\xd5\xc6\x7f\x96\x1f"\x01\x8f#\xc0\xb6 \x15d\xde\x11\x10\x96\xd1l\xa1\x1b\xb0.w\xd9\xe6/\xa0\xd5v\xcco\x00k\x14\xb2\xca=b\xe7jJ\x18L\tr\xe0\xeb\r\xa3\x17@\x1f\x08\xc2\xf5\x06\x1b\xc2zhI\n\x1b\x9bE\x9c\x93\\\xe6\xbe \xfb\x0f\x8c\x9aJ\xac)\xd9\xdf\x84\x92\xe5\xd6\xb6\xa15/\xfa\x0bx\xd1&\x924\x1fX\x0c\xb6\xb6\xa9\x93\xfa\xaf\xeb_\xe1\x1cf>\xd48_\x8a\xd3\xe19\'1\x04H\xc3\x82\t\xee\xd4\xac-\xb3\x13\xb9b\x88\xf6W`\x10\xb8#P\xa6\xcb\xe3\xc4q\xd7\xd2\xe2\xcd\xc6\xa5\xe8\x9fM \x91\x98a8I\x07\x06u~\x81\x07\xf4\xe4g5C\xbb\x1f\xb1\x8b\x1b\xa0\xfe\xaa\x01fy\x9c\xb4#\xc3\xa4z\xd0\x9a\xe0,\x06\x1c\xcf?\xa0G\xa3\x01|\x0b\x8d]\xf8\x13\xa2\xe1.\x0c\x044*\xae\xa5\xc1\xe3 mw\xc4\x1f\xec\x08\xa7\xf8]\xd6\xc0\xf3\x0b\xc6\xbd\xcc7\xee\x8c\xfb\x03[\x08X\xc5\'\xdc\xb5\xb9\xb5l>\x841\x11\xc8\xa8D\xe0\rQ\xef\xf3F>C\x97\xca[\xc1J0r(I\x1d\x157\xde\xc8\x91f\xbc.\xe8\x94J\x19\x81\xf2\x80\xe1\xf053-\xd1\x11[\x06#\xced\xc2\xd1<&\xf8"V\x89\xdb\xe0U\xaf#\xdc\x9f\xb5\x10\x84\xf3\xb9\x0b\x1f\xb8-\xf7\x80\xd4\xfe.\xd3r1\x08\xd9\xdbyA\xe5\xec\x1d\xddS\x16\xac\xd8*Q\xbfg\xe5\x955\xd3\xfe{1\xedOk\xae\xfd#s\xed\xcf\xd0\xf8\xc2\xcd\xff\xf5t\xbe$\xc4\x87p\xd7Y\x9a\x94`\xb1\xd5\xad\xfaB{\xe0!\x1d\xaa\xfb<\x87Z\x0fPw\x00\xc0<\x9f\xd4\xbd"\x1bs\xd0@\x84 \x8a\xeb{N\xa8\xa5\x10\x87\x8b\xf5\x0b\xf5\xc3X\xdb1:>K\xdcB\xaa\x8e\x8dd\xe4\xb6\xdb,\x97\x9b\xa1\xf8\xca\xf8e\xf1\xee\xa8\x0f\x8abz\xe0s\xfaB\xea*nI`?\xddQ\x80\xe3\x9b\xa3\xd3\x82>\xc5A\x99\xb3\x94\xf29{\x80\xb3T%\xc5n\xe8i\xb7\x07\xe0\xb37\xd0S.{\x8a\xad\xe5g\x87\xbc\xc4\xcd1j\xc0$/\x01\xb9@\xedd{o@m\x8ce_o\xbcM\x9c\xda\x88\xa9\x194t6\x88\x17\xcf\xc7?\x8aI\xc8Y\xfe\xee\xc4h\x95j\x9fd\xc1)\x03\xdc\x9a\xb1\xad\xb5\xd1\xf5\n\xfe=\xb5\xd1\xcf\x13_\xff\xe6\xacr\x96\xde~\x0e\xc3\\t~\xf6)\xc5\xdf\x84\xbcC%\xb4\xbd\xa3\xc3\xfd\x93\x0b\xbd\xb3\xbfw\xbe\x7f1\xc3\xbf,#0\xde\xe4L\xc7\x06&\xa5\x0b\x0e*@\x90{K49\xabz\xdan\x9a\xe2~O\xc3{[D\xb6\x15\xcb6\x02\xb4\xda\xb2\x7f\x86\xe6Y\x1f\xbb\xdf\xb3\xfb\x1ae\xeb0\xc8\xa1g\xbbAa\xc9\x81W\xc4\xbf\xd2`\x04\'EQC\xfe\x94_\xdcF\x91\x86[\x8c.\x1cE\x83\xbc8<\xde?\xbd\xbcX\xc5\x9d\xbfZ.C\'\xee\x0c\xc7\xb6\xd4\x00C\x16\x9d,W\xa7r9\xe1\xe1)\x134\x92C\xf1b\xec{0=JA\xcdp\xf6\x889\x04j\xb13.\x94=\x00\xde<\xe1\x83\xab\xabY\xbbO\x0e:\xb0p>\xfa\x88E^\x88\x86\xf2<T\x8b\x10\x81\x7f/<7_\x00\xfe,-\xec\x16\x17\xa6o\x93\x07$0\xe9\x94\xab\x04\xce\x1a\xfd\xe8\xfb\x06\xa0Ix\xad17\xcbG\xa5yWb\x93m\xc9BM\xab\xd23\xab\xb5R\xaf\xd6\xd8*\xd5\xcb\x96Y\xda\xae\xf5Z\xa5\xad\xf2V\xa3gno\xf3^e[\x11|e>\x1e\xf96\xd5\xc4\xc3I\xb1\xb3\xb99\x10\xb0\xa8\xb7\x86\xaf\x19\x1fF>\x1f\xf3.nC\xa1\xb9<Hs\x81\xb5uy\xad\xa8\xae\x97\xf0\xbf\xc2\xd0\xd72\xf5w S\x7f{\xa9+\xbc\xa9\xbf\xa9\xd8\xf4\x17\x9cF\xcf\x03\x15\x97\xb3\x90y\'%\xab(T@X+!R5Z\x95\x052\xd5\xce\xd0\xf0\x8d\x01CV\xb9\xc3\xf2\x1eqG\xc3)P\x02\x99\x9bqvl\xe9\xa1\xd4\xf5\xac\t\xfa\x8c\xe0\xef\x1d\xf2\xfc\xday\xa7(\xdc\xbb\x84aY\x01\xbd\xbe\xfe\x19\xfe\x93z\xe9\x0e;U\xb0\xa7V{\x00\x04\x13\xfe\xee\x9d\xea\xff\xbbw\xe4\x11)\x92\xa0|d\xc5\x91\xc3\x8b\xd0\xceS\x17\xbe\x937\xe3Wq)\x8a;\xa4\xe6sg\xa7\x9d\x8b\x1c9\xb8K\xd1\xa9-\xe5\'\x1c\x7f\x1b\xffy\x88\xab\xd1p\x14(\xb7\xa2)84\xb7\xccs+\xf2e\xc7;0\xc7\x82\x19\xec\xec\xf2":\x98g\xe7\xd4Y\x11\x9bh9\xeew\nW\xb5\xb4\xa0\xaa&\x1dZ\xdea\x97\xe7G\xac\xe7\xf9\xca^4\x9eY\xad\x08\xc6\x1c\x0b\xc2\x86\x12\x1fq\xf6*\r\xda\x038\x17q\x0b\xffZ`Z\x13\xeb/\x17\x98\x00[\x13\xcb\x9a\x19`\x81R\xc3e%\x85d3I\xeag\x08\xa0\xd6\x91?2.\xe94\xeb\xdbf\xef\xefH\xd7c\xba\xd2\xca\xa4\xbd\xb6\xe8\x08\x17I\xb7\x1e9\xa2\x7f\x9cqZT{<\xb7\x13*\x8b3\x1eE9:?\xd4\xc3M(\xa0h*%Q%\xa1\x9f\xc7Z\xd7\xa4\x83z>\xd4E\x97\xeb\xc1\xa1\xc5\x14\x81\x80\xc2\xcb\x87HAc\x10\x97k\xb1\xb1\xf0 qB\x89T\x1a\xfd\x9dbTz%\xd2\x8c\xf2\xfb\x03Hs\xca\x0b?\xeaw\x06mE\x84J\xce)f\\\xf8\xf2\xa4\x1a\xf4\\\xbb7\x89\xa7\x80\xfa\x7fo\xf3\xc8\xce+\x02\x9f\x1b\x18\x8ag\xe3\x00\xa3\xed\x10\xad\xe3~\x10Qj*\xa3\xe6]F\x1b\xfa\x1a\x10qIh\xe7\x9d\xc1\x0f\x03z\xae\x18\x0b{\x83\xec\xf4\xad\xba\x9a\xf5\'\x15\xc9\xc8J\xf3\x92\x04\xfc\xc4\xa8\xc3\x94i\xa7\xc2\x94\x1b\xcf\xbb\x15\xf1\x84\xd40\x16L\xea\xec\x02\x8c\xe4n\x0eA!\x15H\xc06\xa11;\x99\x84\xb7^\xb8/\xd6,q\xcd\x12\xbf.K$1.\x81W_\xc4$\xa7\x97a\xa5\xa8\xfb\xe31N\xa1\x8c\xc8\xabr\xcd\xadzV\xe4\x98\xa4i\xb3\x13\x18~@|\x91\xe5\xe9b\xc6\xbd=\x00\x16\xe6L\n\xa1\x02\x94a\xda\x14T\t\xcd\xb7\xbc\xc7}\x1f\xef4\xc3\x08o\xf3\x19\xd7\xaa\x91\xaf$\xc0,\xe4\x88t\xe4h\x0c\x03\xeeK\x8e\x18s;Z\x95\x1f>b\x17^\x00\xca\x17w\x8c\xa1\xe02 Uj(\xe9\x91&oy\xc8j\x19\x83c%9\xf0\x04\xf7\xd4\xa6\x15(\x16\x16w\x02\x03\x98.\xf4\xce\x12m\x95W\x98=\xc3\x94\x80\x88.7\xab\xcdZ\xb3Zki\xd5j\xbdQ\xaf\xd6\t\x81\xe4$$QW%\n\r\xc3v\xed\xca\x8f\x0c}\xbe\xb6em\xa9\x0b\xd3!\xeb\xf8\x18\xbf>\xfc\xe6\xed\x1c\xe6\xb7\x80\x91,b\xea_\x85\xcf\xc6\xed\x89\x0bX\xed\xb7\xda\xcf\xd1T\x7f\xf9~N\x81Zq?\x87\xb5\x92Rpu{\x81\x14\xac\xdc\xed\xf2z\xa7sD\x8e\x87E\xa6#\xa6\xd0\xefBv\x14\x85d\xac!P\xdf\xa3\xca\xd9\xd7`CW=`\xfb\xe6\r\x1e\x82\xa0\xee\rT\x1bf\xa2\xc6\x00\x11h)\x187|\xe0\xf8\xa8\x94\x1b.\xbb\xab\x00bgFo0l\xc1Y\xd8^,8C;\xee\xa9\x90\xb81\xe4\xcc\xf47\x1d\xd4\xae\xb0b,\x0bj;\x1e\x1doN\xf3\x8b#Ud\xdd\x08\x8b\xc0\xcf\xdeK\x98\x1ae\xba#\xdb\xb1fB\r\x16I\x8aO\x9d\xd7\x987#\xf7\x96\xa2LN\xb7O\xc8\'\xe7\xb1B\xe80!\xd8/ji\xd05V\xc5\xf8\x1bz\x9e\xa3e\xc4\xfd\xcb $\xd5^\xcd2$!I\xed\xe8\xf9bl\x86\xe4\x19ub\x14\xd8\x8e\xa6245\xef\xd9\xf6\xc8FO:1fP\x8e\xcf&\x14\x0f\xa4\x9e\xb3\xc4\xe2\xed\'\xa2\x17\xd3\xcd\x1a\x8eki\xdc\xd2\xfd0\x1c\xea\x0f\x11\xbc\xf4oN$\xeb\xf5\xed\x15\xee"\xc5/\n\xb5\t\xb9\x8aY\x1b\x99\\\xa2\x84\x00t\xd7\x05\x86M\xa35Z\\\x1cZv<\xc3\x92\xad\xbb\xcbJ[\x1c\xb5\x93\x15\x0b\x03\x05\xf1a\x17\xb6U\x84\x9f{]%\x14W\xa2y\xe4\x07\xae\x90$\x98\x06\xd4]H\xe1\xf0oF\xa8\xcb\x16\xec\xb4Yj=\x87:\xc6\xa22 ]*\xb2\x1c\x92\x97\x89\x8e\xa4)W\xd8Y\x93\xbf\xbf%\xf9K\xcbJjX\x9b\xa9\xb9\xfdlZ\xb0\x10^\x9c @Ao\xc8\xdd$M\xc8^\xe9\xa46Tk\xad@\x18\xd0+\xe5\x0c\xd1QJ%\xb9=\xc3\xc5+\x06\n.\x1e\x14\x11\xb2j@\x92\xf9*\x02J\xbe\x03\xba\x03\x0f\x94@F\xf2X\xc8.\n\xd92\x96\xeaA\xe0\x99\x9e\x13v"\x1a\x153\xba\xc0\x08\xb85m?\xc3\x93\x14i\x02]}\xa4_\x9a\xed\x9a>\xc7s\xa8|x/\x9b\xcc\x81\xe4+\xd2F\x99\x0f\xa7\x8a\x88\xca\xea\x84$\xebO\x9f2\x90\xb6\x98P\xd0h\xddv{^\xbe\xf0\xa6\xfav\xb94\x15vW8\x9c\x0f\xf3s\xa8\xces\xc8\x93\xb7AB\x9d\x8e\x86\x11\x9d\x01!\x90\tS\xd7\xd7gI\x92\x8f\x1a&\x9fU\xc7l\xa1\xbb|\xacK:\x14\x92\x8fh\xd3\xa9M\xab\xc3v\xf9\xdc\x1d\xddR\xb72\x1cn\xb8:\xbf\xb7\x83D\x03\xd0\xafd\\\'(&\xb8\x8e\x97]\xa2\xde\x84\xa4%\xfc\x9e\x16V\xe1\xa4o\xc74\x94\x14\xffI\xb4\x93\xe46\xd3,\xdc\xedx\xe6\t\x98\x90\xb4\x06f\x10\xdf\xb0s3\xb3\x84X\xa4\xc7\x08_\x08\xe4\xeb\xd0\xc24\x17N\xf5Ir\xf2T"\xe1\x93\x94\xc5\x00+\xf2\x01\xda\x05\xda\x18\x8fO\xedfud\x89\xe2S\xf4S\x89\x04\xf2S\xdd2\xa6P\\IS\xe8\xc7\x8dK\xe8Oi\xb7\xaf\xe61\'\xc9Z)\x92h\xaaZ\xb5\xacU\x88\x1d\xed\xd2\xc5\xae\xd2\xbe\x0b\x93\x0fXI\xc5\xfb\x1f\xeca\x11\x8d\xf4\x8e\x11\xf0X)\xca\xfcy\xf3gJ\x9an{J\xbe\x05\xdc/\x19\x8e}\xc7\xc3\\\\\xc5\xd2\x11w\xfb\xea\xeeQ\xa3\xbe\x9d\xc8\xba\x88\x0c\xb3\xc3\xa1\xa3\xdc\xd67\xefK\xe3\xf1\xb8\x04\x1bfP\x82\xe5\xe0\xd8+n\xe5"\x1c\x90\xdd\x8b\xec\xba\xedY\x9b\xeeO\x91=\xb7\xbd\x8a-\xf7\xa7\xb8\x1d\xb7M\xa6\xbd\xc7\xb5\xdd\xc7\xd5\x03\xf8o\xbe%\x172%c\x87\x1f\xa15\xf7\xa7\xb5\x1b\xd17\xb0\xe2j\x9a\xb6\x8a\x90\xb5\xd0\xdc\xfaU\x04\xaf\xd5e\x10$\'\x9bD\xfa\xbf\\\xfe\x98\x81\x95|/A\xb1\xd2l\xe9\x83\xc8\x1a\xd5MFG\xd8j\xae y\x84"F\xa9\xcd*i\xe6G\x82\x85\xe4t\x18A\x8b\x14\x00\xdd\xc6\x0b\x14@\xe9\xa5\xcb(\xcf\xb6\xda<\x92\xea"u\xea\xb7\x0c\xf3\x0f\xe9\x8e\xb6`D6\xf1\xb6\x0cJ;\x11|\x15M\x86j\x1b]\x87+\t\xe2AV\x16a\xdf\xc3\x9c\xd0\xef<R\x14\xd5S%\x80\x14\xe3b\xc3*&\x97X\x9f\xe7\xc6O\x90\xe2\x04\x14\x9a;\x97\x91\x0f2Ng\xe4G+\x83\xbd*j\x85(C/3dO\xaa\x8a\xa8\xa0&6;vB\x08"y\xc0\x87<%\xc1e\x8d\x91\x8c\xeb\x98\x03n\xebzc77-\x07\x131r\x83$\xbf\x9f\xf2\xbai\x9aa%O\xf1\x08\x95\x12\\\x18\x99\x1f&Tk\xf8\xa9\'\x94\xfd\x8d_\xe8\x17\x05\x90\x9d\xdd\x8f\x95\xbaU5%\x17&Q\xf1\xab\xaaV\xf2\xa9\x98\x0c{\xc97y\x9eb^d\xcd/!~_E\xd8X\x9d\xf6E\x04\x0c\xf7\xd9\x17S\xbf\x0chI\xd7\x8d\xc8\xb49K\xfd\xc2\xba\x1a\xd4M\xc5i\x9c\xf7\x82\x01z)D\xb4\x01\x04"2\x90\x15Y\xd0\x95\xd3\xb0\x93\xf2\xa7\xa7\xec\xb9\x17\xb6en\x9b\x05J{HW\xd4t=Bs]\x0f\tH\xd0\x9d}\xac\x85\xe8\x96\xac3\xb6\x83\x9bi\xb5|\xd0]\xe2\xd31\xad:\xf7\xa4*a\x18\xc1\x19\xe0\xf7\xdc\xd4\xf3$[\xe9@\x0f\xfb\x8e\x07\x0b\xa8\xb0\x02i\x88\xfaHN\x87t\xdb\xd8\x87\xaa\xa3\x80\xcb;\x86x\xf9}\x1a\x8dVK>\xa3\xa3fBA\x9f\x13\xa28\xe8>\x84\x1aD\xe6\xcc\xff\xda\xde\r\x86\xf1\xe0|\xb3\x06Y-\xdd\xb3\xdc\x8fi\xda(\x97\x97\x1c\xf4J\xaeu\x8cq8\xe3\x07\xa1*"\x1f\n\xe5\xd0\x06\x9b6\x11\xf7\xbdL\xc4\xbb\x93%g\xee\xa0\xea\x14p<\xf45\x95f\xbd\xb8\xb5a\x9eIs\xf1_\xca\\\x89\n\xf8gBB\xcd\xa6\x8d\xff\x14\x99R\xe2\xda\xea\xff\xbf\xcc\xe6\x11\x13\xdf\xa4\xae\xdcV\xff\xbf\xfc\xa0:\xba\x18=\xe69\x9f\xb3\xbe\x87\xb7\xad\x02\x8f)\x05\x9b\x16&\xb6 \xb0\xb9\xdf\xbd\xeb\xd9.\xe8B\x93\x9dw\xef\x8a\x98\xef\xce\xc2\x93\xab\xab\x16\'\x0c-\xe2r<\x08\xf7\x18J\x17hA\x1a\x05)\xe8\x1a;\xc5\xeb\xd9c;\xe9n,A\xda\x01\x1b\xdb\x8e\xc3@B\xf40\x92H\xbc\x93\x90\x87\x12\x14\x0c\x01\n\xdc\x18w PA\xab#\x10\x15K\xaa\xc8,<\x0c\x00\xa6\xad\xed/k\xfb\xcb\xda\xfe\xb2\xb6\xbf\xac\xed/?\x80\xfd\xe5\x9b\x0bJ\tq\xe3\xc1\xe2R\xad\xb5\xb5\xcaI\xd0\xca\xef\x90\x84&\x81\x81w\'\xd9v\x14\xfbKF8\xa7c\x8a\x08\x8cyC\xaf\x8c\xb8\xec\x8cf\x81\xd5\xfe5\x1f\xa6\x17red\xb4\x0ezs1\xc7\xbeE&\x0b\xac\x12\x88\xc4`\x80B\x834\x0ca\x84\x11\xe2\xeahN\xd0\xe6\xc1$k\x0c*):v\rO\xa8"\xea\x9av$\x92\'W\x8a;\x14\xe9!\xbf\xa9kQ\xecX\xab\xb0\xe2\xf3lRP\x94\x8d\x07\xa1\xdb\x8dOGP\xb0hm:\x94\nY=\xa9\x16m2o\xa9\xa4\x15\x0e\x90\xb2\xbcn\x1e\xb1\xdd\xe1p\xdf\xedC\x7f"qH\xc9)\xca_\x89\xe1Itb\xbe(\x90r\x98;\x8d#\xad\x0e\xabu\x99-\\]\x04>\xe8-9\xdcU\x9b\xbf\xe5\n\xa9\xcb\xf4\xd1M\xfa\xc7\xe8\xca\n\xff<\x16\xec\xfa\x1a?\x19\xfd\x87\xbf\xd5\xaf\r\x15\x86Z\x987|\xc0\xc3\x98\xd4\x1ez\x04\xc9x*\xa0\x1a\x15\xd36\xa6\xe9\xd4L}\xf1\xbeLR\xdan\xd5\xba\x15K\xb9\x0eF\x82>\xed\xb1P\xbaYs\xa85\x87\n9\xd4\x0f-\x8d}\x9a#~\x7f\xa14\xad\xf4\x90\xcfa\xee\x0b\xbcZ\xb4\x17\xe8\x81bs+\xc5\xe03\x99\xfbv\xb7Y\xfe\xf5\xff\x97\xb4\x90\xe09\x0f\x17\x17\xca\x8b\xdc\xe8\xd1\x80\x97bj\xf2\xce\x15\xb06\xe5i\x91`jYV\xbcC\x11se0\xcc`\x84J>p(U\xf17v\x85\xfa\xb7\xb4-\x1aq\xb7U<\xb0\xc1\xd0i\xb1{\xc1Is_\xdc\x19\xc5\x8f\xd8\xb6\xaa\xbd\xc21\xce\x8c\xa3\xac\xb4\xef\xc5n\x95]o,\xb6\xd6=\x16\x05\xe4s\xa9IX\xc1\x8d\xbf\xc3\xa50etQ\xac\xc2\x87\xd7`\x13+\xeb\xc5\xfe\xee\xd3\xdd\xc3\x93\xb8\x10\xa5\xb1\xc3\xe8\xa3\x8a\x11\xe5\x902019-11-02T23:27:33.749163560Z \xc9\xa7^\xa1\xf1\x18\x86y\x85\x8db\xd2\xfc\x9a\xf8\xf6Eh\xc1\x08n|oL\xed\xa9\x8e\xb2\xe8\x11\xfb\x0c\xbf5\x9aK\x8cg\xeez\xea9!\xf8\xad\xd17vJ\xef\xe2\x8d\x01\xa0n:%\x899\x16\xe6\xbf`Z\x93\xd6\xdcx\xdeTHXF|\x945$\xc4\xe6\xdc\x05\x99\x94\xf0\xbd\xf6!\xf7eH\xbb\xa8[9i\x87\xfd\xabI\xc9\x8c\xef\xad <\x8e\xfb\xdb\xc6vw4\xdeE=\xffk\x9dd\xbf[\xcdh\x95@\xfe\xf3^\x83\x9ec \xc6\xf6\xf0h5rw\xcf\x8cR\xff\x9d\xab[\xab\x1a\x91\x17\xe8_\xcb\x02)?X!+~u\xfd\xeb\xeb\xa9[kef\xad\xcc\xac\x95\x99\xb52\xf3\xbd*3\xbf\xc8\'\x0c\x7f\x9d\xe1\xe7Pg3\x9e\x99p\x9f\x88\x08\xff\x03=(\xaa\x19\\Y\xca\xfc\x97\xb0\\v\x80\x0e\xdf\xe8\x02=\xf6m\n\xdaH\xee\x00\x14t\xc8\x1f\xb9\x02\x8fy#\xd1\x18\x85\xd5\x9a\xd2\x17\xb2\x98V&\'\xc2\xe7\x91\x87\x13\xdf\xee\xdf\x00\x1b2\x0b\xacZ\xae\x94K\xf0O\x83\xfd\xce\xdd\xf7\xc6\x00\xd9+\xc7\xeb\x05\x1e\x1dI>\xa2\x7fb\x91\xff\xd1\xdd\x00\xc3?w\'\x8ch-\xb7\x8a\xc0\xab9]\xcc\x05\x06\xed\xf7\xd1\xe9\xc3\x83\xbeM\xd8\x90\xa0\xc0\xda\x04\xc0\xb7)\x00%\x8cF\xca9\x8f\xa4\xcf7\xc0\x12^/\xa0\xb7\x89q4\xa0\xffx\xa6\x8d\x0fC\xe2\xd3\x92\x14qI\xcaz\x14\xdd\x80\xe5\xe5\x8b7\x1dU\xe5z\xa3@mY\xdcp$P\x15\xf4),\xc0\xd0\xe5\x03\x91\x1e\xe4\x15XFZ\xbc"\x142\x9d\x11\xee\xf1(\xdb\xb1\x07\xb6j\x88\x0e]qrD\x92\xaf\xe1JO\xa5\xef\xe8\xbaH\x98\xb0\xb2\x84\xfd\x03[\xe2\xcb\x0b\xe4\xcd,9\x13\xd3P\xcc\x08\xf1Yk\xd2\xcbG\xac;\xea\xf58=*\n8\x82\x9b=:\x05\x17Y\xf2\xde*bj\x04\xb2\x8dg\xbf\xb3\x97=\xd2\xef|\xc4\xbaU\xcbjs\xe5\x97\xd6\x1f$G/\xbd\xbf\xb5>\xc7\xf8\x8cs\x8c\xb5h\xba\x16M\xd7\xa2\xe9Z4\xfd\x1eES\xdc\xd8\x9b\x922\xcc\x15\x00\x12\xac\x7f\xb6B\xea=H?\x16\x8ar\xca\xdf\xb1\x9a&\xab%\x98z\xa5V[\x14\x883\xf3^p\xda\x9f0\xf2\xaa"\xd7\xce\xbc<\x1d\x05R\x1dz\x15\xb6\x95\xd7\xa1\xbc\x8e\x90\xb6B\xcc\x13\x1cV\xbb\xb5\x10\xde\x06\xe8r|_q\x85;\t\x92\xcfLW0\xfb\xd9\xaf\xc8\x04c:\xde<#XdS\x99\xbd\x80>\xed\x17:\xe3\xe9\x04\x84\xfd\xa3\xcd\xf4\xcb\x93\xe7\'\xa7W\'\xd9\xd6\x1e\x9dB\xfd\xc2\xac\xea{\x1d\xfd\xf0\xc9\xd1~\x9a\xbbM}\xdf\x7f\x89\xad\'\xe1d\x18\xed-k\x8b\x19\xf5\xad\xad\xd62\xec\xfe\x8c\xed\xf2\xcd\xb1\x9b\x16y5\xbc\xaen/\xbb\xc6\xf3\x08\x1d!\xd1\xd8\xe5\xa0\xd80\x91\xe1\x82\xb8E\xb2=yw\xc6\x9c3\x17\xbc\x00\x93\xb0\xa7\x91^0r\x81\xf4\xe1\xa9\x06>fl0\x17h~\xa5\\\xce\x845\xbe\x81\xc9b(\xb1.Gwe\x92\x0b\x1d\xed\xc8\x15O\x90\xcf\x83\xb2\x15\xa2\xf0#\xf3V\xd8\x07\xf8P\x01\x95E\xb4\xdc;=\xb98<\xb9\xdc\xcf\xdc\x063o\xb1\xc9\xa1\x8a[{(\xfd\x91\x89\xe3M\x85\xceycM\x8f7k\xc7!L5\x9e\xdeP\xc3\x11\xe1L\xe4\xf5\xe3\xdd\x97G\x87\'\xfb\xec\x9f\xacRH\x871T\x18\xfd\x19\x1b\xe2\x9b\xe3o|\x8dVD\xe3Fka,M\xa4P\xc8\x90u\x87\xf70\xc8WH`\xc2U\xeaN@0d\x94I\x8e\xc0\x06\x85M6G\xbe\x8f\xe7sT5\x19\x1e\x13\x00:$-\xcc\x00\x93\x00\xdd\xd1\xa0\x8b\xc1\x08zq\xd0\xb6\x9b\xb9\xe2\x04-F\x00g\xbb\x87\xe4G\xfa+\xcb\x12\xb0H\x14\xf7\xba\x97\x80\x97<\xbd\x8d\xa196\xb0\xe4\\\x14\xbf\x11\xa5\xd4\xd3\xf2\xf3\xd1\nO\xf1l\xe1\x95Z\xad\xc6v\xa9\xa2^}\x9a\xbb\x85`\xfb\xc04\xe5\x11L\x81\xfd\xcaBHs\xce\x0e\x8f\xa0\xd8\x85\xe7\x1dyn_\xbdi<\x12\xd4\xb3\xd4\xe3R\xf4z\x08t\x92\x9c\x91\x1c~\xc7\x1d\x80=\xfb\xfa\xc4\x10\xf4\\|\x1c\xd9\xe7Cg\xb2\x83\x0eI\xf0\xcb\x97\x9dI\xc3\xc3[3\x98\xb1\xf3-\xb7\x8d<\xf0[m\xc7$\xca&\xaf/!\x1d\x0e\xbc\xe4F\x99\x9e%\xc6B\x89o-\xdb#\xbay\xc3\xcd[<\xa4\xa5\x8b\x91\x99\x93\xaeGz\xa2I\xfb\xc3\x9a\xb3\x98\xa7\x9d\xf0\xda\x8c)\xc3M\xd0\xae\x92\xe4/\xb4\x03\xa9\xd9K-\xef\x02\xea\xf7\x00\x81g\x1a\x98J\xc7\xc9\x00t6\xeft\x9c\xa8|we\xf1G\ru\xbe\xd833\x19\xb0\x87B\xa7\xfa\x95\xe4 \xd5\x90\xf2\x92\x98g\xc4\xc0\x8b\xb2\x1aF\xab{S~\x9b}\x16\x9f\xc0Zy\xe1i\xc0\x07\x9e?I\\sj\xb5\xb6\xea)\tG\xe1\x954\x19\x1c\x9ef\xe3s\x95\xaf\x86\xcf\xc2Y\x11\x99\x85\x93\x89\xc9j\x85R\xa8,\x92\xf65`\x9c\x8b\xa2!Ex\n\xf5`0Y\xe1\xe1@U\xec\x13C/\xcf\x8ft\xc6^\xa0ES\xa2p\xb6\xbd\xe6z\x03E\x96\x0f\xdc\xf7\x14@D\xf2\xd8\x9b\xadhW\x10\x92\x9d\x84\x98W@\xab5\xfaF\xb2\xc7\xd90\x95\x80K\n\x82\xae\x17\x96\xbb\xb2$\x82\xe2\x1bV\xde%vST\xa6\xc3%G\xb0\xd9J\x8b\x84(\xd1b\xdau\xe9\xc6!\xc1\x16Y\xd8\x0c\r\xbc\x90f>X\x8b,XQ%\xc0\x91v\xa5\x8c\xd7$\xa8F\xbb\x9c\xf4\x18\x8a\x11\x9f=dpV>%\xbeP-\x1c|\x19\xab\xc9\xd6\xf1\xb3U\xd9\xa68|\xb2_\x0fA{\xc0\xc0N\xe7H\xa2={Cl\xd5z\xcbzV\xbb\x04z^\xcf\x18\xd8\xce\xa4\xbdkY\xc0_\xc5\x01}i\xbb\x07:\xb0\xae\x8b"#\xc3U\xb5\x8c1\xb9\x86\x18\xd3\xa6\xbd\xf5M6\x87a-\xde\x17\xadfe\xc9\x85\xb9\xaci\x9ee\x83\x0bw\xcf\xcc\xdeP>=h;%\x90\xe8\xbd5r\xc7>\xcc\x04|\xc0\x14\x87\xa8\x94"\xf2+Q\xf2$\x15\xa7\x1eI4w\xf0\xb9\xc5\x95p\\\xd2\xd7\xd06\x8a$\xf6>\x8b$\xdcG\xe4\xb5\xdd\xc6\xd2\xfa\xfe\xf9\xf9\xe9\xb9\xbe\x7fz0\rN F\xc3!\xe2\x83\xee\x1b\xfd>\xb7t\xee\xf5\xc4<r-\xfb\xb6\xf4B\x7fl\x90\xe5L\x07\x12\xda\xa2I\xe2\xbe"\xaa\xc3\x14\xc5w\xc8\x8f\x8c\xf9[\xb5e\x98\x7f\xd8c99q\xf4\xcc\x99z\x85\xcd*\x86\xa2>\xbd\xce\x8aA@\xe5\xb2\xc9\x00\n\xb4,\xf4\x08G(\xcc\xc7AJ\xb9\x1e\xebk\xd9\xaf\x99,\xc1\x86\xa5zl\xa4\xb1~\x1e\xe2g\x11\xf7\xb90\xb3\x1e+\xbdK\x93u<\x88\xe5\x8a\xa6c\xb4\xf4\xc2_\x83\x99\xa7R\x02\xc9\xb4\xa7\x98\x95\xd6\x1cW:\n\x00\x0c\xaa@\x14\x93}\xcblX[5n\xd5{\xe5J\xdd\xd8\xb6\xb6k0\x86r\xab\xbc]\xad4\x9bM^\r\r\xb2\xa0f\x0c\xe8\xc6u\xb5\\\x01\x8d\xa6R*W/\xaa\xb5\x9djs\xa7V{M\xbc\x05\'\xc9\xf4\x01\x07\x94O\x1c\xde\xf54\xfa\xb4`\x9d\xfd\xa3\xfd\xbd\x0bXtz\xa8\x08\x9fw\xbd\xde\x00Jg[\xa8m\xa4S\xd1\xa9u\xec\xf9\x99y\xc0\xf01\xb4F\xdfv\xb3rm\xa1\x03\xf1\xe1\xbeL\x98\xcd\xc7\x1f\xb8\xdb\xb2\xf2z\xb6\x0f\xa0\xe7\xe5R\xbb\xf32\xf9\xc0\xb0\x9dy\xfd\t\x8c^oN\x9e\x01{\xf7.\x13"\xbe$\xa1\xbf\xf7\x00\xdda\x16\xd8\xc1\xf9\xe9q\xb2\x08\xbbz\xb6\x7f\xbe\xbf`|\xec\xb0\xc3N.\x8f\x8ed\x8c\x91\x80\xf7\x01\xcdp!\xfe\x1cq\x19\xa0f\xf1\x92VZ\xaf\xe3\xfe\x95\x80\x11\xc6\xc8\t6>\xbd\x8d\xe8\x8bD\xe5\xf1\r\xe7de/k\xb5\xaaVSfw4\x05\xd2y\x99V\xadk\x14\x83\x05}\x1a(\xa1R\xd5\xca\x12\x99\x83\xd10\xf0<z\xa1~\xa3^\xd6\xb6dm\x01\x88\xe9OJ\xc2\xba\x95P\x9b29<\xe4\xa0\xce\xd29\x07m\xeaI\xf0Au\xbf\xa55d\xca\xfb\xb1<\xaa\x83\x9a\xaa\x8c9\x84\x8d\'\xf7[U\xablS\xa2\x98\x98\xde\xb0_\x95iM\xadI\x896\xcd\x05\x80\xa2\x8a\xb6\xe5\x1a*\x9f\xae\xefJ\xc7P\tDvJ&\x95`_\xc2\x06$X0\x08YYe\x99\xbe-\x86\x13:\xf9\x10a\xaf\xaa\xb1|\\\xbd\x12\x1eO\xa8\xc9If\x0e\x87\xf8\xf8\xa5\xcc*\xcb&M\x7f\x02\xb3\xd6\x07\xa9\xe1f\x12\xf6\xa5.o\x0b\x1b\xbe\xc5i\xe85(,\xd3z=[\x01\xae\xc8\xe9\xc1\xe0\x95\xb6L\xc4%\xd7\xb6!G^\xf0u+\x12\xb4\x9cuX\xb62\x91\x0f\xa0\x96\xbe|\xed\x01\x83\xfa\x01\xeb\xbf#\xe29\x1a\x8b\xbe\xbd\xf16\x19\xbc\xfbcx\xe2\xf2\xf0\xa7\x1a\xb0\x0f\x84\x99\xbat\x9eA(\xebS\xc6\xef\xe4\xd9\xc0\xe4Y\xdaS\xf9\xb8,w\xef\x08%:\xfb\xe7/\xf6\xcf\xf5\x93\xdd\xe3\xfd\xd0\xc1H%\x9d\x9d\x9e_\x10{+\x97%\xae\xc5\x8f:\xd3\xc7\x89\xd9\xa7\x8f\x98\xfaL\xddG\x9f\x8flX\xeaei\x8flR\xa5C\xda\xe3\r\xad\xd2\xda\x02\xdcoh\xcdZ"\xfb\xcc\x0b#\xf3oWf\xcfCI\x06L\x9e\x9c\xa2<\xb1y\x13\x0c\x9cb\xe2\xc0\x13S\xfey\x9fN\x1d8\xff\xfa\xb3]\xd6\xb6\x8b?o\xfeL\xbfZs\x0ek\xbb\xa0:&\xcekc\xe5\x8e\x808\x8c\x14[\xb5\x8c\x92E\x9b\xe7\xd8\xb8/\x1dx\xfe\x18\x88\x80\xa4%Dd\xcf\xf1\x11\x03I\xf9V\x7fU#\xf44\xe0\x9b\xbfE\x07q$\xcf~\x9e\xd3\x81\xf0F\xbe\xc9\xbf\xb5\x8b\xc2\\l}\\{\xf2\x93\x00\xda\xcf\xdb\x18l\xc5\xb6\xd4\x80\xc2S\xf4\x12\x0c\xaca6\xeb|\xabl\x96\xccf\xcb*\xd5\xb7\x1a\xcdR\x8bowKe\x10\x8a\xca\x15\xc0\xd6r\xb7\xf7S\x18\x03\xec\xc2;\xb4\xa0\xbe\x1dL\xce\xd4\x8b\xc5\xed\xdd\'\xff\xdc\xbd\xbcxvz~x\xf1\nW!yl\x7f\xec}\xb0\x1d\xc7\xd8lhe\x96\xb7\xcf\x0c\xeb_l\xef\xec\x92\x9dvX\xa5\xa2\x97\xa5g\xcc\xb1ab\xc2\xcb\x02:T;\xfc\x8aw\x9f\xdb\xc1\xe6V\xb9\x8e\x9c\xa7\xc5\xf2\xcf\x9f]\x1c\x1f\x15e\xd9\xa7\xa0\xa4z\x05\xf6B\x9e\xe6l\x02\xa5/\xb3c\xaf\x0bj\xc5f\xa5\xb1[kVY\xc7\xe8\x19\xbe-\xabKd\xbf2\xc6\xa2t\xe9\xaa\x83\xf8\xd2\xa5$\xdb\xeb\x87[\xbfg\n<\x97\xbe\xed\x84\xb4\xecei\xd7\xf7KG^\xbftH\x94\xba\xd9\xa8U[\x9c\x97K@%\x9a\xa5z\xbd\xd2(m\xd7\xb7\xadR\xabW3\x1a\xf5\xeeV\xb9\xb7E\xf5\x9e\xd8\xa2?\xb2A\xf9-\xadJk;\xe8N\xf8\x84\x0f\x1do2\xa0NQ{\xc7\xc2{\x81\xb5\xb0\xcc\x95!\xa0\x00\t\x95\x045T\x81\x97A>\xf5\xed>\x86\xc1Y\xe3\xec\x8f\x80\xb3/C\x9e\x05\xb8u #\x01.\xc0\xdc\x8e\x901\x0c\xcb\xf5\xd6\x7f\xaa\x8d\xad\xff\xec\xb5/;E\xd6i\x03.\xe1\xbb\xec\x01\x9ee\x1f\xb5\x810\x0f<\xd7*\xb2\xd3\xf6\xb1m\xfa\x1ez\x0b\xb3=\xcf\x1fz\xd2\xb9\x162.c9\x87\x17E\xb6w\x92H`\x17G\x1d\xb6\xb7\xcb\x1a\xff\x81\x8c\x9f\xe7a\xe2\xb4\xe7\x14\xaf;\xe2\xb1\xaa\xbf\xc3!\xbe\xdfn\x9b<\x9dMNO T\xbb\xc2\x88\xe4\x8a,qX>\xff\xae\x87\x1b\xa3\x07\xcc\xbf\xcc{\xddJ\xafBF\x1e\xa9 }\xdc\x88lG\xa47i\xd2F\xa4\xe2\xd2\x0b\x05>\xd2\xa2B\x0f_\xa9\x9e\x87U\x87\x93\xa1\xbd\x13\xd3\xbb\xb2*\x93\xe2g\x83\xfc\xd5\x97\xb3H 6H\x1f@\xf5!\xa0\xa8J\x14\xe4\xc8\x02\xfd;\xa6/\x91\x0e\x81r\x0b>GDF\r\xaf\xdfG)\'\xa6H\xd2\xcd:\x0b\x94FR9o\x94\xc3\x84T/\x86 \xfd\xa0\xfa$;\x1a\x0e\x0e}\x7f\xa4Z\x12\xb9\xa4\xcag\xff\xd0\xde*%\xc8J\xf8\xd4R\xe6\x8d\xbd0\x7fj\x93\x91>\x97\x1f7\xe8\x1fiyi4*\xa4\xe0`\xe3o\xdel\xfc\x83\x9c\xbe\xb1y\xf4\xcfo\xe0\xe1J\xa3R}\xfb\xf6\xd3\'Z\xd3\xfa\xd7\x06\xd8H\x02\\\xec,\xfa\xb9\xed\xa8\xb6\xaaKf\xab\xfcM\xfa\x12u\xe4\x1b\x0c\x1a\xff\xfe\x0f\xd58\xf2!\xb6\xba\x00\x00'
2019-11-02T23:27:33.913937548Z reply: 'HTTP/1.1 200 OK\r\n'
2019-11-02T23:27:33.915723453Z header: Server header: Date header: Content-Type header: Content-Length header: Connection header: Access-Control-Allow-Headers header: X-Content-Type-Options header: Content-Language header: Access-Control-Expose-Headers header: Expires header: Vary header: Last-Modified header: X-XSS-Protection header: Allow header: Cache-Control header: Access-Control-Allow-Origin header: Access-Control-Allow-Methods header: X-Frame-Options header: X-Served-By header: Strict-Transport-Security https://sentry.io:443 "POST /api/1401487/store/ HTTP/1.1" 200 41
2019-11-02T23:27:33.916578156Z https://sentry.io:443 "POST /api/1401487/store/ HTTP/1.1" 200 41
pandawankenobi commented 4 years ago

got another this morning dircetly after pressing enter when entering adfs password (the normal place to encounter it):

2019-11-04T09:32:42.500782042Z Received authorization code: 7mfkUeLYjUW2Lr_U7cc4lA.ivtJ8Alh1wh5AZOpXSh8-f7jnGQ.gc5hede0KcJ5VPrZjhVMqQTDgUDl4ifGc3-bTZugrjAmQ63LU5b9K6RKAq0LHXpggZWEZNJPflE_g8_2hjR3b4kifmmjYiyOeX6VbE0r5-a-0dhbu-C82eB4y4fatggErMaRITzdjFh8kiPDPmJYtByS1Mh79GrKPADcsL1ncuaNnqPS_KUrBHL29N-f5biLWQtMZDluEEIJq39ssPa9dnWqL_0oOiGQdHC5HHbzZjLvfBGxJdcUIAuoFdB_I_87YnQoOKFMM7xPfCyDdaUyR3srtsV2vCUfppvqXnGyJ2n1OH2LDdwi4CD0Czf4QeqMy5Mn8Vc2ib2pb8EdV-IvfQ
2019-11-04T09:32:42.501877145Z Getting access token at: https://adfs.my-job.dk/adfs/oauth2/token/
2019-11-04T09:32:42.504654953Z send: b'POST /adfs/oauth2/token/ HTTP/1.1\r\nHost: adfs.my-job.dk\r\nUser-Agent: python-requests/2.20.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 549\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n'
2019-11-04T09:32:42.505319455Z send: b'grant_type=authorization_code&client_id=a7d1fc23-f356-40dc-93f8-6065fc99ef19&redirect_uri=https%3A%2F%2Fmy-site.azurewebsites.net%2Foauth2%2Fcallback&code=7mfkUeLYjUW2Lr_U7cc4lA.ivtJ8Alh1wh5AZOpXSh8-f7jnGQ.gc5hede0KcJ5VPrZjhVMqQTDgUDl4ifGc3-bTZugrjAmQ63LU5b9K6RKAq0LHXpggZWEZNJPflE_g8_2hjR3b4kifmmjYiyOeX6VbE0r5-a-0dhbu-C82eB4y4fatggErMaRITzdjFh8kiPDPmJYtByS1Mh79GrKPADcsL1ncuaNnqPS_KUrBHL29N-f5biLWQtMZDluEEIJq39ssPa9dnWqL_0oOiGQdHC5HHbzZjLvfBGxJdcUIAuoFdB_I_87YnQoOKFMM7xPfCyDdaUyR3srtsV2vCUfppvqXnGyJ2n1OH2LDdwi4CD0Czf4QeqMy5Mn8Vc2ib2pb8EdV-IvfQ'

2019-11-04T09:32:57.553198844Z Resetting dropped connection: sentry.io
2019-11-04T09:32:57.565467981Z Resetting dropped connection: sentry.io
2019-11-04T09:32:57.666804888Z Internal Server Error: /oauth2/callback
2019-11-04T09:32:57.666840088Z Traceback (most recent call last):
2019-11-04T09:32:57.666846888Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
2019-11-04T09:32:57.666852288Z     six.raise_from(e, None)
2019-11-04T09:32:57.666856988Z   File "<string>", line 2, in raise_from
2019-11-04T09:32:57.666862488Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
2019-11-04T09:32:57.666867788Z     httplib_response = conn.getresponse()
2019-11-04T09:32:57.666872588Z   File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/stdlib.py", line 48, in getresponse
2019-11-04T09:32:57.666877488Z     rv = real_getresponse(self, *args, **kwargs)
2019-11-04T09:32:57.666882188Z   File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
2019-11-04T09:32:57.666886888Z     response.begin()
2019-11-04T09:32:57.666891388Z   File "/usr/lib/python3.6/http/client.py", line 297, in begin
2019-11-04T09:32:57.666896088Z     version, status, reason = self._read_status()
2019-11-04T09:32:57.666900788Z   File "/usr/lib/python3.6/http/client.py", line 258, in _read_status
2019-11-04T09:32:57.666905388Z     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
2019-11-04T09:32:57.666910088Z   File "/usr/lib/python3.6/socket.py", line 586, in readinto
2019-11-04T09:32:57.666914688Z     return self._sock.recv_into(b)
2019-11-04T09:32:57.666919288Z   File "/usr/lib/python3.6/ssl.py", line 1009, in recv_into
2019-11-04T09:32:57.666923988Z     return self.read(nbytes, buffer)
2019-11-04T09:32:57.666928488Z   File "/usr/lib/python3.6/ssl.py", line 871, in read
2019-11-04T09:32:57.666933088Z     return self._sslobj.read(len, buffer)
2019-11-04T09:32:57.666937688Z   File "/usr/lib/python3.6/ssl.py", line 631, in read
2019-11-04T09:32:57.666942289Z     v = self._sslobj.read(len, buffer)
2019-11-04T09:32:57.666946789Z socket.timeout: The read operation timed out
2019-11-04T09:32:57.666951389Z 
2019-11-04T09:32:57.666955789Z During handling of the above exception, another exception occurred:
2019-11-04T09:32:57.666960489Z 
2019-11-04T09:32:57.666964989Z Traceback (most recent call last):
2019-11-04T09:32:57.666969489Z   File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2019-11-04T09:32:57.666974289Z     timeout=timeout
2019-11-04T09:32:57.666978689Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
2019-11-04T09:32:57.666992389Z     _stacktrace=sys.exc_info()[2])
2019-11-04T09:32:57.666997589Z   File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 367, in increment
2019-11-04T09:32:57.667002389Z     raise six.reraise(type(error), error, _stacktrace)
2019-11-04T09:32:57.667007089Z   File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
2019-11-04T09:32:57.667011789Z     raise value
2019-11-04T09:32:57.667016289Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2019-11-04T09:32:57.667108089Z     chunked=chunked)
2019-11-04T09:32:57.667119589Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 386, in _make_request
2019-11-04T09:32:57.667124789Z     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
2019-11-04T09:32:57.667129489Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 306, in _raise_timeout
2019-11-04T09:32:57.667134189Z     raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
2019-11-04T09:32:57.667138989Z urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='adfs.my-job.dk', port=443): Read timed out. (read timeout=15)
2019-11-04T09:32:57.667143589Z 
2019-11-04T09:32:57.667147789Z During handling of the above exception, another exception occurred:
2019-11-04T09:32:57.667152389Z 
2019-11-04T09:32:57.667185789Z Traceback (most recent call last):
2019-11-04T09:32:57.667195289Z   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
2019-11-04T09:32:57.667200189Z     response = get_response(request)
2019-11-04T09:32:57.667204689Z   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
2019-11-04T09:32:57.667209389Z     response = self.process_exception_by_middleware(e, request)
2019-11-04T09:32:57.667213989Z   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 124, in _get_response
2019-11-04T09:32:57.667218789Z     response = wrapped_callback(request, *callback_args, **callback_kwargs)
2019-11-04T09:32:57.667223389Z   File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 68, in view
2019-11-04T09:32:57.667228089Z     return self.dispatch(request, *args, **kwargs)
2019-11-04T09:32:57.667232689Z   File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 88, in dispatch
2019-11-04T09:32:57.667237389Z     return handler(request, *args, **kwargs)
2019-11-04T09:32:57.667241889Z   File "/usr/local/lib/python3.6/site-packages/django_auth_adfs/views.py", line 34, in get
2019-11-04T09:32:57.667246489Z     user = authenticate(request, authorization_code=code)
2019-11-04T09:32:57.667250989Z   File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/__init__.py", line 73, in authenticate
2019-11-04T09:32:57.667261389Z     user = backend.authenticate(request, **credentials)
2019-11-04T09:32:57.667266489Z   File "/usr/local/lib/python3.6/site-packages/django_auth_adfs/backend.py", line 264, in authenticate
2019-11-04T09:32:57.667271190Z     adfs_response = self.exchange_auth_code(authorization_code, request)
2019-11-04T09:32:57.667275990Z   File "/usr/local/lib/python3.6/site-packages/django_auth_adfs/backend.py", line 28, in exchange_auth_code
2019-11-04T09:32:57.667282090Z     response = provider_config.session.post(provider_config.token_endpoint, data, timeout=settings.TIMEOUT)
2019-11-04T09:32:57.667287090Z   File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 581, in post
2019-11-04T09:32:57.667291790Z     return self.request('POST', url, data=data, json=json, **kwargs)
2019-11-04T09:32:57.667296390Z   File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
2019-11-04T09:32:57.667301090Z     resp = self.send(prep, **send_kwargs)
2019-11-04T09:32:57.667305590Z   File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
2019-11-04T09:32:57.667310690Z     r = adapter.send(request, **kwargs)
2019-11-04T09:32:57.667315390Z   File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 529, in send
2019-11-04T09:32:57.667320090Z     raise ReadTimeout(e, request=request)
2019-11-04T09:32:57.669868597Z requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='adfs.my-job.dk', port=443): Read timed out. (read timeout=15)
2019-11-04T09:32:57.670351399Z Internal Server Error: /oauth2/callback
2019-11-04T09:32:57.670366399Z Traceback (most recent call last):
2019-11-04T09:32:57.670371299Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
2019-11-04T09:32:57.670375499Z     six.raise_from(e, None)
2019-11-04T09:32:57.670379399Z   File "<string>", line 2, in raise_from
2019-11-04T09:32:57.670383599Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
2019-11-04T09:32:57.670387799Z     httplib_response = conn.getresponse()
2019-11-04T09:32:57.670402799Z   File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/stdlib.py", line 48, in getresponse
2019-11-04T09:32:57.670409099Z     rv = real_getresponse(self, *args, **kwargs)
2019-11-04T09:32:57.670413099Z   File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
2019-11-04T09:32:57.670417199Z     response.begin()
2019-11-04T09:32:57.670420999Z   File "/usr/lib/python3.6/http/client.py", line 297, in begin
2019-11-04T09:32:57.670425099Z     version, status, reason = self._read_status()
2019-11-04T09:32:57.670429099Z   File "/usr/lib/python3.6/http/client.py", line 258, in _read_status
2019-11-04T09:32:57.670433199Z     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
2019-11-04T09:32:57.670443399Z   File "/usr/lib/python3.6/socket.py", line 586, in readinto
2019-11-04T09:32:57.670447999Z     return self._sock.recv_into(b)
2019-11-04T09:32:57.670451999Z   File "/usr/lib/python3.6/ssl.py", line 1009, in recv_into
2019-11-04T09:32:57.670456299Z     return self.read(nbytes, buffer)
2019-11-04T09:32:57.670460499Z   File "/usr/lib/python3.6/ssl.py", line 871, in read
2019-11-04T09:32:57.670464599Z     return self._sslobj.read(len, buffer)
2019-11-04T09:32:57.670468599Z   File "/usr/lib/python3.6/ssl.py", line 631, in read
2019-11-04T09:32:57.670472599Z     v = self._sslobj.read(len, buffer)
2019-11-04T09:32:57.670476599Z socket.timeout: The read operation timed out
2019-11-04T09:32:57.670480599Z 
2019-11-04T09:32:57.670484399Z During handling of the above exception, another exception occurred:
2019-11-04T09:32:57.670488499Z 
2019-11-04T09:32:57.670492199Z Traceback (most recent call last):
2019-11-04T09:32:57.670496199Z   File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2019-11-04T09:32:57.670500299Z     timeout=timeout
2019-11-04T09:32:57.670504099Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
2019-11-04T09:32:57.670508199Z     _stacktrace=sys.exc_info()[2])
2019-11-04T09:32:57.670512199Z   File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 367, in increment
2019-11-04T09:32:57.670516299Z     raise six.reraise(type(error), error, _stacktrace)
2019-11-04T09:32:57.670520299Z   File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
2019-11-04T09:32:57.670524399Z     raise value
2019-11-04T09:32:57.670528299Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2019-11-04T09:32:57.670532399Z     chunked=chunked)
2019-11-04T09:32:57.670537199Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 386, in _make_request
2019-11-04T09:32:57.670541599Z     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
2019-11-04T09:32:57.670545599Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 306, in _raise_timeout
2019-11-04T09:32:57.670549799Z     raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
2019-11-04T09:32:57.670553899Z urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='adfs.my-job.dk', port=443): Read timed out. (read timeout=15)
2019-11-04T09:32:57.670557899Z 
2019-11-04T09:32:57.670561799Z During handling of the above exception, another exception occurred:
2019-11-04T09:32:57.670565799Z 
2019-11-04T09:32:57.670569599Z Traceback (most recent call last):
2019-11-04T09:32:57.670575900Z   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
2019-11-04T09:32:57.670580400Z     response = get_response(request)
2019-11-04T09:32:57.670584300Z   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
2019-11-04T09:32:57.670588500Z     response = self.process_exception_by_middleware(e, request)
2019-11-04T09:32:57.670592500Z   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 124, in _get_response
2019-11-04T09:32:57.670596600Z     response = wrapped_callback(request, *callback_args, **callback_kwargs)
2019-11-04T09:32:57.670600600Z   File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 68, in view
2019-11-04T09:32:57.670604800Z     return self.dispatch(request, *args, **kwargs)
2019-11-04T09:32:57.670608800Z   File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 88, in dispatch
2019-11-04T09:32:57.670612900Z     return handler(request, *args, **kwargs)
2019-11-04T09:32:57.670616900Z   File "/usr/local/lib/python3.6/site-packages/django_auth_adfs/views.py", line 34, in get
2019-11-04T09:32:57.670621000Z     user = authenticate(request, authorization_code=code)
2019-11-04T09:32:57.670625000Z   File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/__init__.py", line 73, in authenticate
2019-11-04T09:32:57.670629200Z     user = backend.authenticate(request, **credentials)
2019-11-04T09:32:57.670633200Z   File "/usr/local/lib/python3.6/site-packages/django_auth_adfs/backend.py", line 264, in authenticate
2019-11-04T09:32:57.670637300Z     adfs_response = self.exchange_auth_code(authorization_code, request)
2019-11-04T09:32:57.670641400Z   File "/usr/local/lib/python3.6/site-packages/django_auth_adfs/backend.py", line 28, in exchange_auth_code
2019-11-04T09:32:57.670646000Z     response = provider_config.session.post(provider_config.token_endpoint, data, timeout=settings.TIMEOUT)
2019-11-04T09:32:57.670650300Z   File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 581, in post
2019-11-04T09:32:57.670654500Z     return self.request('POST', url, data=data, json=json, **kwargs)
2019-11-04T09:32:57.670658500Z   File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
2019-11-04T09:32:57.670662600Z     resp = self.send(prep, **send_kwargs)
2019-11-04T09:32:57.670666700Z   File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
2019-11-04T09:32:57.670670800Z     r = adapter.send(request, **kwargs)
2019-11-04T09:32:57.670674700Z   File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 529, in send
2019-11-04T09:32:57.670678900Z     raise ReadTimeout(e, request=request)
2019-11-04T09:32:57.671610903Z requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='adfs.my-job.dk', port=443): Read timed out. (read timeout=15)
2019-11-04T09:32:57.674826412Z 172.16.0.1 - - [04/Nov/2019:09:32:57 +0000] "GET /oauth2/callback?code=7mfkUeLYjUW2Lr_U7cc4lA.ivtJ8Alh1wh5AZOpXSh8-f7jnGQ.gc5hede0KcJ5VPrZjhVMqQTDgUDl4ifGc3-bTZugrjAmQ63LU5b9K6RKAq0LHXpggZWEZNJPflE_g8_2hjR3b4kifmmjYiyOeX6VbE0r5-a-0dhbu-C82eB4y4fatggErMaRITzdjFh8kiPDPmJYtByS1Mh79GrKPADcsL1ncuaNnqPS_KUrBHL29N-f5biLWQtMZDluEEIJq39ssPa9dnWqL_0oOiGQdHC5HHbzZjLvfBGxJdcUIAuoFdB_I_87YnQoOKFMM7xPfCyDdaUyR3srtsV2vCUfppvqXnGyJ2n1OH2LDdwi4CD0Czf4QeqMy5Mn8Vc2ib2pb8EdV-IvfQ&state=L3Zpa2FyL2Jlc3RpbC8= HTTP/1.1" 500 220212 "https://adfs.my-job.dk/adfs/oauth2/authorize/?response_type=code&client_id=a7d1fc23-f356-40dc-93f8-6065fc99ef19&resource=a7d1fc23-f356-40dc-93f8-6065fc99ef19&redirect_uri=https%3A%2F%2Fmy-site.azurewebsites.net%2Foauth2%2Fcallback&state=L3Zpa2FyL2Jlc3RpbC8%3D&scope=openid" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36" "5.186.115.73:1604"
2019-11-04T09:32:57.675307914Z [pid: 22|app: 0|req: 607/1356] 172.16.0.1 () {82 vars in 3712 bytes} [Mon Nov  4 10:32:41 2019] GET /oauth2/callback?code=7mfkUeLYjUW2Lr_U7cc4lA.ivtJ8Alh1wh5AZOpXSh8-f7jnGQ.gc5hede0KcJ5VPrZjhVMqQTDgUDl4ifGc3-bTZugrjAmQ63LU5b9K6RKAq0LHXpggZWEZNJPflE_g8_2hjR3b4kifmmjYiyOeX6VbE0r5-a-0dhbu-C82eB4y4fatggErMaRITzdjFh8kiPDPmJYtByS1Mh79GrKPADcsL1ncuaNnqPS_KUrBHL29N-f5biLWQtMZDluEEIJq39ssPa9dnWqL_0oOiGQdHC5HHbzZjLvfBGxJdcUIAuoFdB_I_87YnQoOKFMM7xPfCyDdaUyR3srtsV2vCUfppvqXnGyJ2n1OH2LDdwi4CD0Czf4QeqMy5Mn8Vc2ib2pb8EdV-IvfQ&state=L3Zpa2FyL2Jlc3RpbC8= => generated 220212 bytes in 16454 msecs (HTTP/1.1 500) 4 headers in 130 bytes (1 switches on core 0)
2019-11-04T09:32:57.922647463Z send: b'POST /api/1401487/store/ HTTP/1.1\r\nHost: sentry.io\r\nAccept-Encoding: identity\r\nContent-Length: 8689\r\nX-Sentry-Auth: Sentry sentry_key=54c91b17a76e45599d152f53f9f8ce9c, sentry_version=7, sentry_client=sentry.python/0.7.3\r\nContent-Type: application/json\r\nContent-Encoding: gzip\r\n\r\n'
2019-11-04T09:32:57.923701066Z send: b'\x1f\x8b\x08\x00I\xf0\xbf]\x02\xff\xed=\t{\xd3\xc8\x92\x7f\xa57|\x8c\xedy\xb6\xe23q2\xe3\x99\r!@\x86\x84dr\x00\x0f\xc2\nYj\xdb"\xb2$\xd4r\x82\xe1\xf1\xdf\xb7\xaa\xba%\xeb\xf2\x11`\xd8\x19\xd6y\xf3\xf1\xac>\xaa\xaf\xea\xba\xba\xba\xfa\xd3\x86\xc3o\xb8\xb3\xb1\xcb6x\x10x\xc1F\x15~|0\xb9\x1f\xda\x9e\x0b\xa9\x9f6n\x0cg\xc2\x05\xfc|\xfdic\xecY\x13\x87c\xe1\x80\xbf\x87\xd4Phqa\x81U\xc3\xa9O\xd9g\xdc\xb0.\xec1\xf7&!&\x13\x0cL\x7frqqz\xbe\xef\xb9.7\xb1\xce\xa9\xe79\xe5\x91\'\xc2^\xc9\xb0\x06B3\x8c`4\x11\xd7\xdex<q\xb9f]\x97\xaa\xcc\xf7\x82\xb0\xd7n\xb7*\xbb\x0c\x81\xb2\x10\xa0Z\x0c\xe0j\xac\x1cD\t\xf0\xd9kt*\xd8\xd4\x98\x9b#\xc3\xb5\xc5\x98:\x1f\xf5\xc7zg\xb8C\x0f\xf3!\xd3r\xb8\x05\x89\x03\xc3\x11\xfc3$\x89\xd00\xaf\xc3\xc009\xd5\x19\x04\xc68\x1a\xf0\xc0v\xb8\x0b\x9f3\x18\x9b\xa6\x17\xf0M\t%\x10\x9b\xf1\xf05\x7f\x8a\xe0\x8d\xbe\xd0}#\x1ca\x85\xcd\x89\x086\x1d\xcf4\x9cM\xc7\xeeo\xfa\xd3p\xe4\xb9-mkS\xd8!\xaf\xf9\xd0\xa81\xe4bs%\xb8\x83\x89k\xaa%\xd9\xb0a\xfah\xa1f\xcb!ah\x08C\x8b`\xcc\x96\x06\xcb:\xb6\xcb]\x0f\xca\xb6\xda\xf0\xe5\x07\\7=7\xe4\x1fB\x1c\xe8\x06\x83?\xd3pY\xc0\x9d)\xf3\\6\xe4ah\xbbCf@\x8a\xf0aq9\xb3]\x11\xe2|{\x03\x06\x05g\xfdC\xe0X\xfdj\x03\xff\x17}\xfd\xf7m`\xf8\xa2\x0cp\xf4\x08B%\xca\xb3\xf8\x80\xd1\x18\xca\n\x8b*\xbbQ\x16\xfe\x85\xc1tw\xe3\r$\xa8\x0e\xea\xd8u\x1c$K\xfc\xc5\xdd\xea\xb1d\x1b1D\x04\xe8\x03b\xe5F\x89\x7f\xb2\xf3\xec \x1a\x033\x04\xa6\xa5z\x91i$\xfa\xa9\x0f\xbc@\x8f\x07\x1f5W\xc5\xea\x95d\xf5\x80\x87\x93\xc0\x8d\xabEY*9^A\x1a\xe7\x8d\x11\x08\xc2<\x00\x92\xd9<\xe5\xc4\xef\x03\xdc\x9f\xe5\xab\xbfx\x0f]mT+U\x1a\x8a\x1a\x1b\xf6\xe8\xd7\x17\xe7\x8f\x0f\xcf\xe4\xf7.{|p\xc1J\x9b\x9e1\tG\xcdM\xc0n\xa7\x0f\xa8\xfc\xbb\xe9Y\xbc\xb7=\x1e\\_\xf2\xa3\x7f\xbf\xbb|\xd1<\n\xf4\xcbm\xd3l;{\x9a}\x13\xfe\xd1\xddsF\x8d\xdbQg\xef\xd5\x89\xff\xf2|\xd4\xad\r\xb6\xdf\xb9\x8f\xff\xd4\x86fg\xc4-^\x7fj\xfe\xd1y~\x1a\xbcz7z~\xfc\xfe\xcf\x8b\x87\xc3\xcb\x87N\xdb\x1e<6[\xb5\xfe\xc5\xab\xc90x\xb77\xfes\xabut\xd9\xe9\xef<\xdd:{\xba\xf7\xbe~\xf4\xe4\xa5?\x1c\xbezq\xf0\xea\xd9\x1f\xa7\x03\xe7@\x1fv\xf5\xe6\xe8\xddY\xab\xdf\xbe\xb6\x07\xe3\xf1\xbb\x7f\xdb\xd3\x13\xfer\xeby\xff\xa0\x1etjF\xadn\x8d\xfa\x93\xda~\xb7\xc9\x1f\xb4\xa7\xed\x81\x11\x0e\x87\x07\xc1\xb1qvx\xf1\xd1z\xf7h\xd4\xbd\xb6O\x1f\x9e\x8e\xff\xf8w\xf8`z\xde8\x1em\xef<\x0e\x9e\x9e\xee=4\xc5Q\xc35\'\xc63\xf7\xfd\xe9\xb9\xfe\xf42x\xf0\xe4\xa8\xb9\xf3\xac6\xe8\xf4\xed\xa3\x17\x7f\x86\xc7\xaf\x1e:\x93\x83\x83\xc3?\xde\xb7v\x8485v,\xf7\xc5\xfb#\xbd\xee\x9d\xd8\x8f\xff\xb4\x9e\xecw\x9e<\xe9\x7f|\xf5\xee\xe8f\xf0\xe0\xf1\x87?,\xf3\xf2po\xe2=\xb2\x1e\xe8\x87zw\xfb\xdf\xee\x9f\xde\xc9\xd3G\xc7\xc7\xdb\x1fN\x07\xfb\xd3\x87\x96q9=k\x89 \x14\xcf\x9b7\xfb\x97\x03\xdf\xbfy\xff\xd2}<\xfd\xa3\xe96N\x9e4\x8f\x1eZ\xb7v{\xffa}\xff\xe3\xa0\xfd\'\x7f\x7f<\xed\x1c\xbb\xdd\xe7f\xd3\xee7\xfd~\xf7\xc0z^;\xbc\x19\xfc\xf9\x13\x10\xb5\x90\xf7\x8eZ\xaf|\xa3\xf9hz\xd4\xfc\xc31[g~\x7f\xbf\xdb+\xfd\x86\xab\x9b\xdc5\xb4\xc4}o\xe2Zl\xcc\x81HY\xec\x81!\xf8\x13IG4=Y\x14\xb7\xff\xaf\x85\xd4\xe6V\x0cm\r\xf1D\xd5c^\xff\x1d`(3BV\xff\xb0=\xe8\xb7\x1bF\xbd\xddm\xf6\xeb\xbf\xfd\xb6\x81T\xd7vu\xc3\xf7\xa1\xe90\x98 \x19^\x81\xd8\xf6\xa1W\xdf\x98\xce&@&Ilj\xcc+\x90Z\x04\x93\xa2\xb2\x8d\xe6\xd6\x1c2\x8b\x7f\xf6 AW\x05{\xe6\xb9<G|\x90\x80\xfa\xdc\xd2\xa3\r\x06DHpg\xa0\x8d\x8dk\xae\xdf\xd8\xfcV7Bol\x9b\xe5\xa8@%\x0b\x81\x08i&-C\xd6\xb2\x8d\xcc\x08\xda\xcfQ\x92n\x04C\x01\xdf\xb3\x84\xeb[L\xca5WHU\x97S\xf2L\x8fh\x8c~\xe0\x99\\\x88\x19\xa1\xd5\xfbS}l[0\xd9\xd06/\xf3*[H\xe8\x93]\xbb\xc7\xf6\xbd\xb1\xef\x18\xb6\x8b\xd3\x1e\x8e8\xc3\xc9S\x94\x18\xc8 N>+\x1b\xcc\x04\x8a\t\x9d&)\xa8\xa2%!\xac\xb2ZP\xc6\x16\xc8"\r\xd7\xe4\xf1\x92T\x19\n B{\xa4P\xeb\x02\xbe\x80\x08S\xaf\x1e=x^\xb48\xb4\xb2\xb8\x0b`."0\x9aN)\xba\x9ef\x17\xd9\xc5\xa3m\x1ca1;\xd9C\xfa\xbc\xaf\xf2\x9e\xe3\x98g[qg\xbb\xdd\xb7\xea]\xa2\x04\xb3y\xd5\xe5\xee\xcf\x93\x83}\x11\x0c\x10\xc2q\\4^"\x9a\xcc\x04Q\x98A\xd3L\xa8\xa5\xe5\xab\xe6\t\xc3N\xb7a\xb5\xb7\x910 \xae\xa4\xd1\x0cG\xfa9\x99\xac\x12_\xbfI$~\xe1\xc8aU=\xe7\x86\x07\xfa\xd8\x08\xcd\x91d\xbd2\xe5\x18\x13\xca\x08\xb1\'\xc7\xa5#\xb7\xd3\x89\xc1\xe2\x88\x85\x96o\xa4\xca\xb0k\xbdr\xa5\xcad\xd7{\xd8\xf1I\xe0\xd0\xe2\xf5f8\x01\xabFI\xa2\xf7\xba\x94\x85^\x82aQ\x1e\x90\xad9\x05*\xc9\xae\xd3\xc8/\xcf\x8e\xa2\x8e\xb3\xd2-\xef\xeb\xb088\xc5\x1a4.J\xacL\x18\x8b\xffTX\xe9\x7f6K\xd1\xd8c\xda\x8fYk\x8e\xff\xa3s|\xa4\xac\xb4\xb4_\xc9\xbe\xff_p\xef\xf6\x02\xee\x1d\xfd\xf5Av\xbe&)\xfe\x8e\xcc\xe2\x1b\xb1\xf6;r\xd6\xaf\xe5\xf5s\xd5\xa9\x05\xcc\xbf\xfa\xcdy\xfdw`\xedk&\xbbf\xb2k&\xbbf\xb2\xff\x0c&K;es\xc8]\x1e\xd8\xe6\xb7a\xb2sA&\x99,\x16*\xe4\xadr\xeb\xaa\xday\xde\xba\xd5]\xc2Z\x81&\x8f\x0ca\x84aP\xc6\xd9\xac\xb2\x12\xb0\xf3R\x85\xc1\x941\xd7\x0b\xb3\x99#nX\xa5J!\xab!\xfe\x82\xf9\x11\xaf\x01@\xd9r\x94\xae\xf6$\xd9\xf9\xe4\xee,*\x85\xe4\x06\x8a\x10m,\xca\x97\x04\tJ(\xa2\xba\xdc\x86I\xc6@\xaak\xd9@\x8c\x90\x1c\xce\x98r\xc4\x8bWa\xc18\xe74\xf1\xba\xe9\x18\x02\xfb`:\xa9N\xce\n\xd8\xae\x1d\xc6=\x9d}\xe4yk\x08\xd2\x08\x91I\x9ay\xcb3E\x18\xa0Mx\x10xcF\xcd$\xcbO|\x0bv\x86.\xd9eP\xbe!Z\r\x9d\xa8\xaa\x1c\x0bhv%\xcdZ3{e\x05\xea\x9f\xdf/\xdb\xfdz\xc3\xda\x92\x9c&\xcd\xcb\x92,lMs\x7fd\x9a\x9b@\xe2H\x8eq\x04\xad\xb5\xdc\x0c\xa5\x95\xf1\xab\xf4\x0f\xa0\xbb\x11\xa5\xf8\x12\xda\xdb]D{\xefE\xc4/\x12Hm\xe1\x96B<\rB\xc9\x1a&$\xf0n@\xa8vl\x11\xe6eh\xaa\xa8\xc9\x8a\x9a\xe3\xdd\x02\t\xa8\x00uQ$8\x0c}%\xf6J\x99,G\xad\x15\x87\x94\xa79\t\xea^\x0c\xb8Z\x00\xd6\x0buXH\xc8\xb7R\xca\x12w\x04_\xd0\xda"8\x0b\xe9\xb7\xa2\xdd\n\xd2\x1d\xa9v\xf2 lN\xe3\xe9\xf1\xe7\xe1\xa6\xc6\xe4x\xc3!\x0f4\xc8q\x81@\x97\xb3\xc3-\x1d\xcb\xf5|\x06\xccsO\x82g\xe5\xfb\x00\x83\xdd\x17\xa5\xec\x1c\xcf\xbe\x11\x87\xaby#o\x18\x18\xbdO%\xdc\x93\x13\xa1#\xa5,\xed\xb2v\xbd\x03\x9cXU\x84o\xf5\xebs\x9a\xde\xab\xc9\xcak>\xf9\x8d\x88\x9c:\xa1\xf0|\x15kX\xf3\x86\xb5<\xfemd\x8cUy\xc3\xac\x15I\xd2\xbf\x961\xcc\x81\x97\xe4\nJ\xb2\xcd2\x84\xecxW\xf3\x04H\x11\x0f\xb2\x9a\x00\x85\x12\x02z\x04{\xfb\n4Y\x86`\xbd\xc0\xfeh\x90-\x00\xf7\x04\xbb5\x04C\x16a[\xdc\xd2\xae6r\x94\x03\xe6J\x12\x8d^\xbb^\xcf\x99r\x02n\xd9\x01L\xbb\x1ez3)\\\x83-\x88\x94\xa0|\xb5A\xeb|\xb5Q\x89\x0f\xcd\xe7\x11\xe6\x89 \xc2\x8e\x1d\xe4nh\x9bPmF\x9dS\xdd&\xe2\xd5\xc3\x7f\x96\x1f"\x01\x8f#\xc0\xb6 \x15d\xde\x11\x10\x96\xd1l\xa1\x1b\xb0.7\xc5\xe6/\xa0\xd5v\xc2o\x00kT\x8a\xca\xddcgjJ\x18L\tr\xe0\xab\rc\x10B\x1f\x08\xc2\xd5\x06\xf3a=\xb44\x85M\xcc"\xceI\xa9p_\x90\xfd\x07FM%\xd6\x94\xecoB\xc9Jk\xdb\xd0\x9a\x17\xfd\x05\xbch\x13IZ\x00,\x06[\xdb\xd4I\xfd\xd7\xf5op\x0e3\x1fj\x92/%\xe9\xf0\x9c\x93\x18\x02\xa4a\xc1\x14w\xdan-\xb3\x13\xb9\xc2G\xfb+0\x08\xdc\x11(\xd3\x95q\xe2\xb8ki\xc9f\x93R\xf4\xcf&\x90H\xcc0\x9c\xb4\x03\x83:\xbf\xc0\x03z\xf2\xb3\xca\xd1\xee{\xecb\x04\xd4_5\xc0,\x8f\x93vd\x98T\x0fZ\x13\x9c%\x80\xe3\xf9\x07\xf4h2\x86o\xa1\xb1\x8b`J4\xdc\x85\x81\x80F\xc5\xb5,x\x1c\xa4\xedN\xf8\x9d\x1d\xe1\x14\xbf+\x1axy\xc1\xb8\x97\xf9\xc6\x9d\xf2`l\x0b\x01\xab\xf8\x90\xbb6\xb7\x96\xcd\x870\xa6\x02\x19\x95\x08=\x1f\xf5>o\x120t\xa9\xbc\x16\xac\x06#\x87\x92\xd4Q1\xf2&\x8e4\xe3\xf5A\xa7T\xca\x08\x94\x07\x0c\x87\xaf\xdc\xb4\xc4Gl\x05\x8c\xb8\x90\t\xc7\xf3\x98\xe2\x8bX%i\x83W\xbd\x8eq?o!\x88\xe6s\x0f>p[\xee\x03\xa9} \xd3J\t\x08\xc5\xdbyA\xe5\xc2\x1d\xdd\xdeQg%\x89U\xa2~\xe7\xe5\x955\xd3\xfe{1\xed\xcfk\xae\xfd#s\xed/\xd0\xf8\xa2\xcd\xff\xedt\xbe4\xc4\xbbp\xd7<MJ\xb1\xd8\xe6V{\xa1=\xf0\x90\x0e\xd5\x03^B\xad\x07\xa8;\x00`^@\xea^\x95\xddr\xd0@\x84 \x8a\x1bxN\xa4\xa5\x10\x87K\xf4\x0b\xf5\xc3D\xdb\t:\x9e\'n\x11U\xc7F\nr{=V*\xe5(\xbe2~Y\xbc?\x19\x82\xa2\x98\x1d\xf8\x9c\xbe\x90\xba\x8a[\x12\xd8O\x7f\x12\xe2\xf8\xe6\xe8\xb4\xa0OqP\xe6,\xa5|\xe6\x0fp\x96\xaa\xa4\xd8\r=\xeb\xf6\x00|v\x04=\xe5\xb2\xa7\xd8Z9?\xe4%n\x8eq\x03&y\t\xc8\x05\xea\xa5\xdb{\rjc"\xfbj\xe3M\xea\xd4F\xcc\xcc\xa0\x91\xb3A\xb2x9\xf9QMC.\xf2w\'F\xabT\xfb4\x0b\xce\x18\xe0\xd6\x8cm\xad\x8d\xaeW\xf0\xef\xa9\x8d~\x99\xf8\xfa7g\x95yz\xfb%\x0cs\xd1\xf9\xd9\xe7\x0c\x7f\x13\xf2\x0e\x95\xd0\xf6\x8f\x0e\x0f\x9e]\xe8\xe7\x07\xfbg\x07\x179\xfee\x19\xa1\xf1\xbad:60)]pP\x01\xc2\xd2\x1b\xa2\xc9E\xd5\xb3v\xd3\x0c\xf7{\x1c\xdd\xdb"\xb2\xadX\xb6\x11\xa2\xd5\x96\xfd+2\xcf\x06\xd8\xfd\x81=\xd4([\x87A\xfa\x9e\xed\x86\x95%\x07^1\xff\xca\x82\x11\x9c\x14E\r\xf9Syq\x1bU\x1an5\xbep\x14\x0f\xf2\xe2\xf0\xf8\xe0\xe4\xf2b\x15w\xfef\xbd\x0e\x9d\xb81\x1c\xdbR\x03\x8cXt\xba\\\x9b\xca\x95\x84\x87\xa7L\xd0H\t\xc5\x8b\xdb\xc0\x83\xe9Q\nj\x81\xb3G\xc2!PK\x9cq\xa1\xec\x01\xf0\xe6\t\x1f\\]\xcd\xda{\xf8\xe8\x1c\x16.@\x1f\xb1\xd8\x0b\xd1P\x9e\x87j\x11b\xf0\xef\x84\xe7\x96+\xc0\x9f\xa5\x85\xdd\xe2\xc2\x0cl\xf2\x80\x04&\x9dq\x95\xc0Y\xa3\x1f\xc3\xc0\x004\x89\xae5\x96\xf2|T\x9aw%6\xd9\x96,\xb4m5\x06f\xb3U\x1b\xb4:[\xb5v\xdd2k;\xadA\xb7\xb6U\xdf\xea\x0c\xcc\x9d\x1d>h\xec(\x82\xaf\xcc\xc7\x93\xc0\xa6\x9ax8)v77\xc7\x02\x16\xf5\xda\x084\xe3\xe3$\xe0\xb7\xbc\x8f\xdbPh.\x0f\xb3\\`m]^+\xaa\xeb%\xfc?a\xe8k\x99\xfa\x1f S\x7f\x7f\xa9+\xba\xa9\xbf\xa9\xd8\xf4W\x9cF\xcf\x03\x95\x94\xb3\x90y\xa7%\xab8T@T+%Ru\xba\x8d\x052\xd5\xaeo\x04\xc6\x98!\xab\xdcee\x8f\xb8\xa3\xe1T(\x81\xcc\xcd8;\xb6\xf4P\xea{\xd6\x14}F\xf0\xf7.y~\xed\xbeU\x14\xeem\xca\xb0\xac\x80^]\xfd\x0c\xffI\xbdt\x97\x9d(\xd83\xab=\x00\x82\t\x7f\xfbV\xf5\xff\xed[\xf2\x88\x14iP\x01\xb2\xe2\xd8\xe1Ehg\x99\x0b\xdf\xe9\x9b\xf1\xab\xb8\x14%\x1dR\xcb\xa5\xd3\x93\xf3\x8b\x129\xb8K\xd1\xa9\'\xe5\'\x1c\x7f\x0f\xff\xb9\x8b\xab\x91?\t\x95[\xd1\x0c\x1c\x9a[\xe6\xb9\x15\x05\xb2\xe3\xe70\xc7\x82\x19\xec\xf4\xf2">\x98gg\xd4Y\x91\x98h9\xee\xb7\nW\xb5\xac\xa0\xaa&\x1dZ\xdee\x97gGl\xe0\x05\xca^t\x9b[\xad\x18\xc6\x1c\x0b\xc2\x86\x12\x1fq\xf6\x1a\x1d\xda\x038\x17I\x0b\xffZ`Z\x13\xeb\xaf\x17\x98\x00[S\xcbZ\x18`\x81R\xa3e%\x85d3M\xeas\x04P;\x97?\n.\xe9lo5\x9b\xdb\x7fG\xba\x9e\xd0\x95V&\xed\xadEG\xb8H\xba\xf5\xd8\x11\xfdS\xceiQ\xed\xf1\xd2n\xa4,\xe6<\x8aJt~\xa8G\x9bP@\xd1LJ\xaaJJ?O\xb4\xaeI\x07\xf5r\xa4\x8b.\xd7\x83#\x8b)\x02\x01\x85\x97\xfbHA\x13\x10\x97k\xb1\x89\xf0 IB\x89T\x1a\xfd\x9d\x12Tz%\xd2\x8c\xf2\xfb\x1dHs\xc6\x0b?\xeew\x01mE\x84J\xcf)f\\\x04\xf2\xa4\x1a\xf4\\{0M\xa6\x80\xfa\xff\xc1\xe6\xb1\x9dW\x84\x0170\x14\xcf\xc6#\x8c\xb6C\xb4\x8e\x07aL\xa9\xa9\x8c\x9aw\x19m\xe8[@\xc4%\xa1\x9dw\n?\x0c\xe8\xb9b,\xec5\xb2\xd37\xeaj\xd6{*R\x90\x95\xe5%)\xf8\xa9QG)\xb3NE)#\xcf\xbb\x16\xc9\x84\xcc0\x16Lj~\x01&r7G\xa0\x90\n\xa4`\x9b\xd0\x98\x9dN\xc2[/<\x10k\x96\xb8f\x89\xdf\x96%\x92\x18\x97\xc2\xab\xafb\x92\xb3\xcb\xb0R\xd4\xfd\xf1\x18\xa7PF\xe4U\xb9\xe6V\xbb(rL\xda\xb4y\x1e\x1aAH|\x91\x95\xe9b\xc6\x07{\x0c,\xcc\x99V"\x05\xa8\xc0\xb4)\xa8\x12\x9ao\xf9\x80\x07\x01\xdei\x86\x11^\x97\x0b\xaeU#_I\x81Y\xc8\x11\xe9\xc8\xd1\xf0C\x1eH\x8e\x98p;Z\x95\x1f\xdec\x17^\x08\xca\x17w\x0c_p\x19\x90*3\x94\xecH\xd3\xb7<d\xb5\x82\xc1\xb1\x9a\x1cx\x8a{j\xb3\n\x14\x0b\x8b;\xa1\x01L\x17zg\x89\x9e\xca\xab\xe4\xcf0% \xa2\xcb\xdb\xcdnggg\xab\xa9u\xea\xad\x9d\xad\xae\xa4\xd3r\x12\xd2\xa8\xab\x12\x85\x86a\xbb\xf6\xe4G\x1e}\xbb\xadm\xa3\xbf]\xff-\xc9:>%\xaf\x0f\xbf~3\x87\xf9-`$\x8b\x98\xfa7\xe1\xb3I{\xe2\x02V\xfb\xbd\xf6s<\xd5_\xbf\x9f3\xa0V\xdc\xcfQ\xad\xb4\x14\xdc\xdcY \x05+w\xbb\xb2~~~D\x8e\x87U\xa6#\xa6\xd0\xefJq\x14\x85t\xac!P\xdf\xe3\xca\xc5\xd7`#W=`\xfb\xe6\x08\x0fAP\xf7\x06\xaa\r3\xd1b\x80\x08\xb4\x14\x8c\x1b\x01p|T\xca\r\x97\xdd4\xb4f\xb30z\x83a\x0b\xce\xa2\xf6\x12\xc1\x19zIO\x85\xd4\x8d!\'\xd7\xdflP\xbb\xca\x8a\xb1,\xa8\xeddt\xbc9\xcd/\x8eTQt#,\x06\x9f\xbf\x9703\xca\xf4\'\xb6c\xe5B\rVI\x8a\xcf\x9c\xd7\x98\xa3\x89{MQ&g\xdb\'\xe2\x93\xf3X!t\x98\x10\xecW\xb54\xe8\x1a\xabb\xfc\xf9\x9e\xe7h\x05q\xff\n\x08I\xd3\xe0}.\tIfG\xcf\x17c\x0b$\xcf\xb8\x13\x93\xd0v4\x95\xa1\xa9y/\xb6G\x1aM\xd9l\x01\xe5\xf8bBqG\xea\x99\'\x16o>\x13\xbd\x98m\xd6h\\K\xe3\x96\x1eD\xe1P\x7f\x88\xe0\xa5\x7fs"\xd9n\xef\xacp\x17)yQ\xa8G\xc8U-\xda\xc8\xe4\x12%\x04\xa0\xbb.0l\x1a\xad\xd1\xe2\xe2\xd0\xb2\xe3\x19\x96l\xdd]V\xda\xe2\xa8\x9d\xacX\x18(H\x00\xbb\xb0\xa7"\xfc|\xd0UBu%\x9aG~\xe0\nI\xc2Y@\xdd\x85\x14\x0e\xffrB]\xb1`\xa7\xe5\xa9\xf5\x1c\xea\x98\x88\xca\x80t\xa9\xcaJH^\xa6:\x92\xa6RewM\xfe\xfe\x96\xe4/++\xa9amf\xe6\xf6\x8bi\xc1BxI\x82\x00\x05=\x9f\xbbi\x9aP\xbc\xd2im\xa8\xd5]\x810\xa0W\xca)\xa2\xa3\x94JJ\xfb\x86\x8bW\x0c\x14\\<("d\xd5\x80$\xf3U\x04\x94\xf29\xe8\x0e<T\x02\x19\xc9c\x11\xbb\xa8\x14\xcbX\xaa\x07\xa1gzN\xd4\x89xT\xcc\xe8\x03#\xe0\xd6\xac\xfd\x02OR\xa4\tt\xf5\x91~i\xb6k\x06\x1c\xcf\xa1\xca\xd1\xbdl2\x07\x92\xafH\x0fe>\x9c*"*\xab\x13\x92\xa2?}\xc6@zbJA\xa3u\xdb\x1dx\xe5\xca\xeb\xe6\x9b\xe5\xd2T\xd4]\xe1p\xee\x97\xe7P\x9d\xa7\x90\'o\x83D:\x1d\r#>\x03B S\xa6\xae\xaf\xe7IR\x80\x1a&\xcf\xabc\xb6\xd0]~\xabK:\x14\x91\x8fx\xd3\xa9M\xab\xc3v\xf9\xc2\x1d\xbd\xb3\xadne8\xdcpu\xfe\xc1\x0eS\r@\xbf\xd2q\x9d\xa0\x98\xe0:^v\x89{\x13\x91\x96\xe8{VX\x85\x93\xbe\xbe\xa5\xa1d\xf8O\xaa\x9d4\xb7\x99e\xe1n\xc73O\xc0\x84\xb45\xb0\x80\xf8F\x9d\xcb\xcd\x12b\x91\x9e |\x11\x90oC\x0b\xb3\\8\xd3\'\xc9\xc93\x89\x84OR\x16\x03\xac(\x87h\x17\xe8a<>\xb5\x9b\xd5\x91%\x8aO\xf1O%\x12\xc8Ou\xcb\x98Bq\xa5M\xa1\x9f6.\xa1?\xb5\xbd\xa1\x9a\xc7\x92$k\xb5X\xa2ij\xcd\xba\xd6 v\xb4G\x17\xbbj\x07.L>`%\x15\x1f~\xb4\xfd*\x1a\xe9\x1d#\xe4\x89R\x94\xf9\xf3\xe6\xcf\x944\xdb\xf6\x94|\r\xb8_3\x1c\xfb\x86G\xb9\xb8\x8a\xb5#\xee\x0e\xd5\xdd\xa3N{\'\x95u\x11\x1bf}\xdfQn\xeb\x9b\x1fj\xb7\xb7\xb75\xd80\xe3\x1a,\x07\xc7^q\xab\x14\xe3\x80\xec^l\xd7\xed\xe5m\xba?\xc5\xf6\xdc\xde*\xb6\xdc\x9f\x92v\xdc\x1e\x99\xf6\xee\xb7\xf6\xee7\x1f\xc1\x7f\xf3-\xb9\x90)\x19;\xfc\x88\xac\xb9?\xad\xdd\x88\xbe\x83\x15W\xd3\xb4U\x84\xac\x85\xe6\xd6o"x\xad.\x83 9\xd9$\xd2\xff\xf5\xf2G\x0eV\xfa\xbd\x04\xc5J\x8b\xa5\x0f"kT7\x1d\x1dak{\x05\xc9#\x121j=\xd6\xc82?\x12,$\xa7\xc3\x08Z\xa4\x00\xe86^\xa0\x00J/]Fy\xb1\xd5\xe6\x9eT\x17\xa9S\xbf\x17\x98\x7fHw\xb4\x05#\xb2\x89\xb7eP\xda\x89\xe1\xabh2T\xdb\xe8;\\I\x10w\xb2\xb2\x08\xfb\x03\xcc\t\xfd.#EQ=U\x02H5)6\xacbrI\xf4yn\xfc\x04)N@\xa1\xb9s\x19\xfb \xe3t\xc6~\xb42\xd8\xab\xa2V\x882\xf42C\xf1\xa4\xaa\x88\njb\x8bc\'D \xd2\x07|\xc8SR\\\xd6\x98\xc8\xb8\x8e%\xe0\xb6\xaew\xeb\x96f\xe5`"&n\x98\xe6\xf73^7K3\xac\xf4)\x1e\xa1R\x8a\x0b#\xf3\xc3\x84f\x0b?\xf5\x94\xb2\xbf\xf1+\xfd\xa2\x00\xb2y.ll\xd5\xfb\xa6\x94_HT\xfc\xa6\xaa\x95|*\xa6\xc0^\xf2]\x9e\xa7\x98\x17Y\xf3k\x88\xdf7\x116V\xa7}1\x01\xc3}\xf6\xd5\xd4\xaf\x00Z\xdau#6m\xe6\xa9_TW\x83\xba\x998\x8d\xf3^0@/\x85\x986\x80@D\x06\xb2*\x0b\xfbr\x1av3\xfe\xf4\x94=\xf7\xc2\xb6\xcc\xed\xb1Pi\x0f\xd9\x8a\x9a\xae\xc7h\xae\xeb\x11\x01\t\xfb\xf9\xc7Z\x88n\xc9:\xb7v8\x9aU+\x87\xfd%>\x1d\xb3\xaasO\xaaR\x86\x11\x9c\x01\xfe\x81\x9bz\x99d+\x1d\xe8\xe1\xd0\xf1`\x01\x15V \rQ\x1f\xe9\xe9\x90n\x1b\x07Pu\x12ry\xc7\x10/\xbf\xcf\xa2\xd1j\xe9gt\xd4L(\xe8sB\x14\x87\xfd\xbbP\x83\xd8\x9c\xf9\x7f\xb6wC?\x19\x9c/o\x90\xd5\xb2=+\xfd\x98\xa6\x8dz}\xc9A\xaf\xe4Z\xc7\x18\x873y\x10\xaa"\xf2\xa1P\x0em\xb0Y\x13I\xdf\xcbT\xbc;Y2w\x07U\xa7\x80\xe3\x91\xaf\xa94\xeb%\xad\r\xf3L\x9a\x8b\xff2\xe6JT\xc0\xbf\x10\x12j6=\xfc\xa7\xca\x94\x12\xd7S\xff\xffu6\x8f\x84\xf8&u\xe5\x9e\xfa\xff\xe5\x07\xd5\xf1\xc5\xe8[^\n8\x1bzx\xdb*\xf4\x98R\xb0ia\x12\x0b\x02\x9b\xfb\xed\xdb\x81\xed\x82.4\xdd}\xfb\xb6\x8a\xf9n\x1e\x9e\\]\xb58Qh\x11\x97\xe3A\xb8\xc7P\xba@\x0b\xd2$\xcc@\xd7\xd8\t^\xcf\xbe\xb5\xd3\xee\xc6\x12\xa4\x1d\xb2[\xdbq\x18H\x88\x1eF\x12Iv\x12\xf2P\x82\x82!@\x81\x91q\x03\x02\x15\xb4:\x01Q\xb1\xa6\x8a\xe4\xe1a\x000mm\x7fY\xdb_\xd6\xf6\x97\xb5\xfdem\x7f\xf9\x01\xec/\xdf]PJ\x89\x1bw\x16\x97Z\xdd\xadUN\x82V~\x87$2\t\x8c\xbd\x1b\xc9\xb6\xe3\xd8_2\xc29\x1dS\xc4`\xcc\x11\xbd2\xe2\xb2S\x9a\x05\xd6\xfae>L/\xe2\xca\xc8h\x1d\xf4\xe6b\x8e}\x8dL\x16X%\x10\x89\xf1\x18\x85\x06i\x18\xc2\x08#\xc4\xd5\xd1\x9c\xa0\xcd\x83I\xd6\x18TRt\xec\x1a\x9eP\xc5\xd45\xebH$O\xae\x14w\xa8\xd2C~3\xd7\xa2\xc4\xb1Ve\xc5\xe7\xd9\xa4\xa0(\x1b\x0f#\xb7\x9b\x80\x8e\xa0`\xd1zt(\x15\xb1zR-zd\xdeRI+\x1c \x15y\xdd\xdcc{\xbe\x7f\xe0\x0e\xa1?\xb18\xa4\xe4\x14\xe5\xaf\xc4\xf0$:5_\x14H9\xca\x9d\xc5\x91V\x87\xd5\xba\xcc\x16\xae.\xc2\x00\xf4\x96\x12\xee\xaa\xcd\xdfK\x95\xcce\xfa\xf8&\xfd}te\x85\x7f\xee\x0bvu\x85\x9f\x8c\xfe\xc3\xdf\xea\xd7\x86\nC-\xcc\x11\x1f\xf3(&\xb5\x87\x1eA2\x9e\n\xa8F\xd5\xac\x8di653_\xbc\xaf\x94\x94\xcc\xe6V\xdb\x90\x92\xd2L\xd0\xa7=\x16I7k\x0e\xb5\xe6P\x11\x87\xfa\xa1\xa5\xb1\xcfs\xc4\xef\xaf\x93\xa6#=\xe4K\x98\xfb\x02\xaf\x16\xed9z\xa0\xd8\xdc\xca0\xf8\xa2-\xde\xed4U\x88\xc0\xff?\xd2B\x8a\xe7\xdc]\\\xa8/r\xa3G\x03^\x86\xa9\xc9;W\xc0\xda\x94\xa7E\x8a\xa9\x15Y\xf1\x0eE\xc2\x95\xc10\xc3\t*\xf9\xc0\xa1T\xc5\xdf\xd9\x0b\xd4\xbf\xa5m\xd1H\xba\xad\xe2\x81\r\x86NK\xdc\x0bN\x9b\xfb\x92\xce(A\xcc\xb6U\xed\x15\x8eqr\x8e\xb2\xd2\xbe\x97\xb8Uv\xb5\xb1\xd8Zw_T\x90\xcfe&a\x057\xfes.\x85)\xa3\x8fb\x15>\xbc\x06\x9bXY/\x0e\xf6\x1e\xef\x1d>K\nQ\x1a;\x8c?\x9a\x18Q\x0e\x99|\xe6\x15\x1a\x8fa\x98W\xd8(&\xcd\xaf\x89o_D\x16\x8cp\x14x\xb7\xd4\x9e\xea(\x8b\x1f\xb1/\xf0[\xa3\xb9\xc4x\xe6\xae\xa7\x9e\x13\x82\xdf\x1a}c\xa7\xf4>\xde\x18\x00\xea\xa6S\x92\x98ca\xfe\x0b\xa65m\xcdM\xe6\xcd\x84\x84e\xc4GYC"l.]\x90I\t\xdfk\xf7y 2019-11-04T09:32:57.923701066Z C\xda\xc5\xdd*I;\xec_MJr\xbe\xb7\x82\xf08\xe9o\x9b\xd8\xdd\xf1x\x17\xf5\xfc\xafu\x92\xfd\xc7jF\xab\x04\xf2\x9f\xf7\x1a\xf4\x1c\x031\xb6\x87G\xab\xb1\xbb{a\x94\xfa\x7f\xb8\xba\xb5\xaa\x11y\x81\xfe\xb5,\x90\xf2\x9d\x15\xb2\xea7\xd7\xbf\xbe\x9d\xba\xb5Vf\xd6\xca\xccZ\x99Y+3\xffTe\xe6W\xf9\x84\xe1o9~\x0eu6\x93\x99)\xf7\x89\x98\xf0\xdf\xd1\x83\xa2Y\xc0\x95\xa5\xcc\x7f\t\xcbe\x87\xe8\xf0\x8d.\xd0\xb7\x81MA\x1b\xc9\x1d\x80\x82\x0e\x05\x13W\xe01o,\x1a\xa3\xb0\xdaR\xfaB\x11\xd3*\xe4D\xf8<\xb2?\r\xec\xe1\x08\xd8\x90Ya\xcdz\xa3^\x83\x7f:\xec\x01w\xdf\x19cd\xaf\x1c\xaf\x17xt$y\x8f\xfeID\xfeGw\x03\x0c\xff\xdc\x9f2\xa2\xb5\xdc\xaa\x02\xaf\xe6t1\x17\x18t0D\xa7\x0f\x0f\xfa6e>A\x81\xb5\t\x81oS\x00J\x18\x8d\x94s\xeeI\x9fo\x80%\xbcAHo\x13\xe3h@\xff\xf1L\x1b\x1f\x86\xc4\xa7%)\xe2\x92\x94\xf5(\xba\x01+\xcb\x17o\xceU\x95\xab\x8d\n\xb5eq\xc3\x91@U\xd0\xa7\xa8\x00C\x97\x0fDz\x90W`\x19i\xf1\xaaP\xc8t&\xb8\xc7\xe3l\xc7\x1e\xdb\xaa!:t\xc5\xc9\x11i\xbe\x86+=\x93\xbe\xe3\xeb"Q\xc2\xca\x12\xf6\x0fl\x89\xaf/\x907\x8b\xe4LLC1#\xc2gm\x9b^>b\xfd\xc9`\xc0\xe9QQ\xc0\x11\xdc\xec\xf1)\xb8(\x92\xf7V\x11Sc\x90=<\xfb\xcd_\xf6\xc8\xbe\xf3\x91\xe8V\xab\xa8\xcd\x95_Z\xbf\x93\x1c\xbd\xf4\xfe\xd6\xfa\x1c\xe3\x0b\xce1\xd6\xa2\xe9Z4]\x8b\xa6k\xd1\xf4\x9f(\x9a\xe2\xc6\xde\x94\x94a\xae\x00\x90b\xfd\xf9\n\x99\xf7 \x83D(\xca\x19\x7f\xc7j\x9a\xac\x96b\xea\x8dVkQ \xce\xc2{\xc1Y\x7f\xc2\xd8\xab\x8a\\;\xcb\xf2t\x14Hu\xe4U\xd8S^\x87\xf2:B\xd6\n1OpX\xed\xd6Bt\x1b\xa0\xcf\xf1}\xc5\x15\xee$H>3[\xc1\xe2g\xbfb\x13\x8c\xe9x\xf3\x8c`\xb1M%\x7f\x01}\xd6/t\xc6\xd3\t\x08\xfb\xaf\x1e\xd3/\x9f=}v\xf2\xe2Y\xb1\xb5G\xa7P\xbf0\xab\xfa\xfe\xb9~\xf8\xf0\xe8 \xcb\xddf\xbe\xef\xbf&\xd6\x93p2\x8a\xf6V\xc8\xc0\xb6\x1b\xcd\xa5\xd8\xfd\x05\xdb\xe5\xbbc7-\xf2jx\xdd\xdcYv\x8d\xe7\x1e:B\xa2\xb1\xcbA\xb1a*\xc3\x05q\x8bd{\xf2\xeeL8g.x\x01&eO#\xbd`\xe2\x02\xe9\xc3S\r|\xcc\xd8`.\xd0\xfcF\xbd^\x08\xebv\x04\x93\xc5Pb]\x8e\xee\xca$\x179\xda\x91+\x9e \x9f\x07e+D\xe1G\xe6\xad\xb0\x0f\xf0\xa1\x02*\x8bh\xb9\x7f\xf2\xec\xe2\xf0\xd9\xe5A\xe16\xc8\xbd\xc5&\x87*\xaem_\xfa#\x13\xc7\x9b\t\x9d\xf3\xc6\x9a\x1do\xd1\x8eC\x98j<\x03_\xc3\x11\xe1L\x94\xf5\xe3\xbd\x97G\x87\xcf\x0e\xd8\xbfX\xa3\x92\rc\xa80\xfa\x0b6\xc4w\xc7\xdf\xe4\x1a\xad\x88\xc6\x9d\xee\xc2X\x9aH\xa1\x90!\xeb\x0e\x1f`\x90\xaf\x88\xc0D\xab\xd4\x9f\x82`\xc8(\x93\x1c\x81\r\n\x9blN\x82\x00\xcf\xe7\xa8j:<&\x00tHZ\xc8\x01\x93\x00\xdd\xc9\xb8\x8f\xc1\x08\x06I\xd0\xb6[\xb8\xe2\x04-A\x00\xf3\xddC\xf2#\xfd\x95e\tX$\x8a{=H\xc1K\x9f\xde&\xd0\x1c\x1bXr.\x8a\xdf\x88R\xeai\xf9\xf9h\x85\xa7x\xb6\xf0j\xddng\xa7\xd6P\xaf>\xcd\xddB\xb0}`\x9a\xca\x08\xa6\xc2~c\x11\xa49g\x87GP\xec\xc2\xf3\x8e<w\xa8\xde4\x9e\x08\xeaY\xe6q)z=\x04:I\xceH\x0e\xbf\xe1\x0e\xc0\xce\xbf>\xe1\x83\x9e\x8b\x8f#\x07\xdcw\xa6\xbb\xe8\x90\x04\xbf\x02\xd9\x99,<\xbc5\x83\x19\xbb\xdfs\xdb\xc8\x03\xbf\xd5vL\xaal\xfa\xfa\x12\xd2\xe1\xd0Ko\x94\xd9Yb"\x94\xf8\xd6\xb2=\xa2\x9b#n^\xe3!-]\x8c,\x9ct=\xd6\x13M\xda\x1f\xd6\x9c\xc5<9\x8f\xae\xcd\x982\xdc\x04\xed*I\xfe";\x90\x9a\xbd\xcc\xf2.\xa0~w\x10xf\x81\xa9t\x9c\x0c@g\xf3F\xc7\x89*\xf7W\x16\x7f\xd4P\xe7\x8b=\xb9\xc9\x80=\x149\xd5\xaf$\x07\xa9\x86\x94\x97\xc4<#\x06^\x94\xd50Z\xdd\xeb\xfa\x9b\xe2\xb3\xf8\x14\xd6\xca\x0bOc>\xf6\x82i\x123\x07;]++\xbf+\xbc\x92&\x83\xc3\x93"|\xdejwwV\xc3g\xe1\xac\x88\xcc\xc2)\xc4d\xb5B\x19T\x16i\xfb\x1a0\xceE\xd1\x90b<\x85z0\x98\xa2\xf0p\xa0*\x0e\x89\xa1\xd7\xe7G:c\xcf\xd1\xa2)Q\xb8\xd8^s\xb5\x81"\xcbG\x1ex\n "y\xe2\xcdV\xb4+\x08\xc9N"\xcc\xab\xa0\xd5\x1a}#\xd9\xfdb\x98J\xc0%\x05A\xd7+\xcb]YRA\xf1\r\xab\xec\x12\xbb\xa9*\xd3\xe1\x92#\xd8b\xa5EB\x94h1\xeb\xbat\xe3\x90`\xab,j\x86\x06^\xc92\x1f\xacE\x16\xac\xb8\x12\xe0H\xafQ\xc7k\x12T\xa3WO{\x0c%\x88\xcf>28\xab\x9c\x11_\xa8\x16\x0e\xbe\x8e\xd5d\xeb\xf8\xd9m\xecP\x1c>\xd9\xaf\xbb\xa0=`\xe0\xf9\xf9\x91D{\xf6\x9a\xd8\xaa\xf5\x86\r\xac^\r\xf4\xbc\x811\xb6\x9dio\xcf\xb2\x80\xbf\x8aG\xf4\xa5\xed=\xd2\x81u]T\x19\x19\xae\x9au\x8c\xc9\xe5cL\x9b\xde\xd6w\xd9\x1c\x86\xb5x_t\xb7\x1bK.\xcc\x15Ms\x9e\r.\xdc=\xb9\xbd\xa1|z\xd0vJ \xd1{k\xe2\xde\x060\x13\xf0\x01S\x1c\xa1R\x86\xc8\xafD\xc9\xd3T\x9cz$\xd1\xdc\xc1\xe7\x16W\xc2qI_#\xdb(\x92\xd8\x0fE$\xe1CL^{=,\xad\x1f\x9c\x9d\x9d\x9c\xe9\x07\'\x8ff\xc1\t\xc4\xc4\xf7\x11\x1f\xf4\xc0\x18\x0e\xb9\xa5so \xe6\x91k\xd9\xb7\xa5\x17\xfa\x13\x83\xac\x17:\x90\xd0\x16M\x13\xf7\x15Q\x1d\xa6(\xb9C~d\xcc\xdfj-\xc3\xfc\xc3\x01+\xc9\x89\xa3g\xce\xd4+lV5\x12\xf5\xe9uV\x0c\x02*\x97M\x06P\xa0e\xa1G8"a>\tR\xca\xf5X_+~\xcdd\t6,\xd5cc\x8d\xf5\xcb\x10\xbf\x88\xb8\xcf\x85Y\xf4X\xe9M\x96\xac\xe3A,W4\x1d\xa3\xa5W\xfe\x1a\xcc<\x91\x12H\xa1`=\xe7Q\x9f7\x9f?S\x00`P\x05\xe2\x98\xec\x03\xab\xdb\xed\x9b;\xed\x86\xd5\xe5\xed\x81\xd1\xd9\xb1\xfa\xdb\xdc\xacou\x07\xad\xd6v\xd7\xda\x8a\x0c\xb2\xa0f\x8c\xe9\xc6u\xb3\xde\x00\x8d\xa6Q\xab\xb7/\xea;\xbb\xad\xe6ng\xfb\x15\xf1\x16\x9c$3\x00\x1cP>qx\xd7\xd3\x18\xd2\x82\x9d\x1f\x1c\x1d\xec_\xc0\xa2\xd3CE\xf8\xbc\xeb\xd5\x06P:\xdbBm#\x9b\x8aN\xad\xb7^P\x98\x07\x0c\x1fCk\x0cm\xb7(\xd7\x16:\x10\x1f\x1e\xc8\x84|>\xfe\xc0\xddV\x947\xb0\x03\x00=/\x97\xda\x9d\x97\xc9\xc7\x86\xed\xcc\xebOh\x0c\x06s\xf2\x0c\xd8\xbb7\x85\x10\xf1%\t\xfd\x9d\x07\xe8\x0e\xb3\xc0\x1e\x9d\x9d\x1c\xa7\x8b\xb0\x17O\x0e\xce\x0e\x16\x8c\x8f\x1d\x9e\xb3g\x97GG2\xc6H\xc8\x87\x80f\xb8\x10\xef\'\\\x06\xa8Y\xbc\xa4\xed\xe6\xab\xa4\x7f%`\xb51q\xc2\x8d\xcfob\xfa"Q\xf9v\xc49Y\xd9\xebZ\xab\xa9\xb5\x94\xd9\x1dM\x81t^\xa65\xdb\x1a\xc5`A\x9f\x06Jh4\xb5\xbaD\xe6p\xe2\x87\x9eG/\xd4o\xb4\xeb\xda\x96\xac-\x001\x83iMX\xd7\x12\xea\xb6L\x8e\x0e9\xa8\xb3t\xceA\x9bz\x1a~T\xdd\xefj\x1d\x99\xf2\xeeV\x1e\xd5AMU\xc6\xf4a\xe3\xc9\xfd\xd6\xd4\x1a;\x94(\xa6\xa6\xe7\x0f\x9b2m[\xdb\xa6D\x9b\xe6\x02@QE\xdbr\r\x95O\xd7w\xa5c\xa8\x04";%\x93j\xb0/a\x03\x12,\x18\x84\xac\xac\xb2\xcc\xc0\x16\xfe\x94N>D\xd4\xabf"\x1fW\xaf\x86\xc7\x13jr\xd2\x99\xbe\x8f\x8f_\xca\xac\xbal\xd2\x0c\xa60kC\x90\x1aF\xd3\xa8/my[\xd8\x08,NCoAa\x996\x18\xd8\npCN\x0f\x06\xaf\xb4e".\xb9\xb6\x039\xf2\x82\xaf\xdb\x90\xa0\xe5\xac\xc3\xb2\xd5\x89|\x00\xb5\x0c\xe4k\x0f\x18\xd4\x0fX\xff\r\x11\xcf\xc9\xad\x18\xda\x1bo\xd2\xc1\xbb?E\'.w\x7f\xaa\x01\xfb@\x98\xa9K\xe7\x19\x84\xb2>e\xfc\x87<\x1b\x98>K{,\x1f\x97\xe5\xee\r\xa1\xc4\xf9\xc1\xd9\xf3\x833\xfd\xd9\xde\xf1A\xe4`\xa4\x92NO\xce.\x88\xbd\xd5\xeb\x12\xd7\x92G\x9d\xd9\xe3\xc4\xe2\xd3GL}\xa2\xee\xa3\xcfG6,\xf5\xb2\xb6O6\xa9\xda!\xed\xf1\x8e\xd6\xe8n\x01\xeew\xb4\xedV*\xfb\xd4S\x91\xf9\xb7\xea\xed\xfcy(\xc9\x80\x94l\x80rP\xdb\x97O\xd6S\xeb\xc6\x87\x1a\xb0\xba^=}\xae\x8a\xd2\xc6\xe6(\x1c;\xd5\xd4q(\xa6\xfc\xebC6u\xec\xfc\xf2\xbeW\xd7v\xaa\xf6\x18@m\xc2\x10|\xf5\xd3\xf0\xdda\xf5\xe7\xcd\x9f)\xbf\x9b\xaa%\xec!\xf0\x88Z\xf4,\xf1/7\xbd~k\xce\tp\xea\x00\x18\x84\xa3 Q\xee\x08*O\x14\xaf\xb6\x8c\xda\xc3\xa7U\xcbP\xbd\xe1n\xed\xf2\\\xb5\xcc]\xfa\xb1]u\xfb\xf4\x83\x84\x83c\x18\xfb#/\xb8\x05\x02$\xe9\x18M\xc2\x19>\xa0 \xa9\xee\xea/zD^\x0e|\xf3\xf7\xf8\x10\x90d\xe9/sx\x10\xde$0\xf9\xf7v\x8f\x98\xbbS\xee\xb7\x1e\xfe$\x80\xef\xf0\x1e\x06z\xb1IXO\x9f\xe9\x1f{\x1fm\xc716;Z\x9d\x95_\xd8\xae\xe5\xdd\n\xf6\xec\x825\xeaZ\xfd\x17\x06\t[\xed_\xd8\x87\xadv\x05=\xad\x1d\xfe\x82\xf7\x9f\xda\xe1f\xa7\x05\\r\x8b\x95\x9f>\xb98>\xaaJ\xe7\x9a\xc7\xa0\xbbz\x15\xb6?\n\xbc1\xdf\xdc\xee"\x03\xd9\xa9\xb7\xb5\xee6;7\x06F`\xabZ\xd4\x07\x1f\x18\x8a\xc5k\x870\xdd&\x8c\xb0v\x96\xe0\xb6\xc4\x1f\x1e\xca\xee\xd1\xefsn\xd6\x1e\xf1\xd0\x1c\xd5.\x15W\xfd=\x93~\x0e\xd3C\xdb%\xf0\x84\xa8\xe1d\xa5\xf3\x8f\xd5\x9b<\xaeqc\x0f\r\x99\xfb\xb2\xf6\xc2\xb8\x15\xb5KWy\x07\xd4.%/Y\xbf&\xfbOf\x0bs\x89\xeenD`_\xd6\xf6\x82\xa0v\xe4\rk\x87\xc4>\xea[\xe6\x0e\xe0g\x07v\xe8\xd6N\xad\xdd\xdf1k\xfd\xce\xa0Sk[m\xab\xd1\xe7\xadV}@\xf5\x1e\xdab8\xb1A#\xaf\xad\xca\x00\x10+k\x0f\xb9\xefx\xd31u\x8a\xda;\x16\xdes\xac\x85e^\x18\x02\n\x90\xa4KP#\xbd|\x19\xe4\x93\xc0\x1ebl\x9e5\xce\xfe\x088\xfb2bf\x80[\x8fdx\xc2\x05\x98{.d`\xc5z\xbb\xfb\x9ffg\xeb?\xfb\xbd\xcb\xf3*;\xef\x01.\xe1c\xf1!\x1e\xb0\x1f\xf5\xce\xb85\xf6\\\xab\xcaNz\xc76\x92Eo\x80>#\x81\xefI\x8f_\xc8\xb8L\xe4\x1c^T\xd9\xfe\xb3T\x02\xbb8:g\xfb{\xac\xf3\x1f\xc8\xf8y\x1e&\xcezNA\xc4c\xe6\xab\xfa\xeb\xfb\xf8\xa8\xbcm\xf2l6yb\x81\xa4\xef\n#\x16v\x8adt\xf9&\xbd\x1em\x8c\x01\x08\x0fu>\xe87\x06\rbfRk\xfb\xb4\x11\x1b\xb4H\x99\xd3\xa4\xe1J\x05\xcb\x17\n|\xac\xdaEn\xc7\xd2f\x10U\xf5\xa7\xbe\xbd\x9bP\x06\x8b*\x936j\x83P8\x94\xb3H 6HIA\x9d&\xa4PO\x14y\xc9\x9a\xf8<\xa1\xc4\x91b\x83r\x0f\xbe\x91D\x96\x16o8D))\xa1\xdd\xd2u?\x0b4Y\xd2\x83G\xca\x8bC\xea<>HP\xa8\xd3\xc9\x8eF\x83C\x87$\xa9+\xc5~\xb2\xf2-B4\x02K\xb1\xb6\x11\xbd\xffTx\x8d0\xca\x9f\x19\x8a\xa4#\xe8\xa7\r\xfaG\x9a\x83:\x9d\x06i]\xd8\xf8\xeb\xd7\x1b\xffE\x9e\xe8\xd8<^\x1a\xe8\xe0\x89O\xa7\xd1|\xf3\xe6\xf3gZ\xd3\xf6\xb7\x06\xd8I\x03\\\xec\xc1\xfa\xa5\xed\xa8\xb6\x9aKf\xab\xfe]\xfa\x12w\xe4;\x0c\x1a\xff\xfe\x17\xfa\x88\xa3\xa4K\xbb\x00\x00'
2019-11-04T09:32:58.218821160Z reply: 'HTTP/1.1 200 OK\r\n'
2019-11-04T09:32:58.220709866Z header: Server header: Date header: Content-Type header: Content-Length header: Connection header: Access-Control-Allow-Headers header: X-Content-Type-Options header: Content-Language header: Access-Control-Expose-Headers header: Expires header: Vary header: Last-Modified header: X-XSS-Protection header: Allow header: Cache-Control header: Access-Control-Allow-Origin header: Access-Control-Allow-Methods header: X-Frame-Options header: X-Served-By header: Strict-Transport-Security https://sentry.io:443 "POST /api/1401487/store/ HTTP/1.1" 200 41
2019-11-04T09:32:58.221478768Z https://sentry.io:443 "POST /api/1401487/store/ HTTP/1.1" 200 41
jobec commented 4 years ago

It looks like Requests isn't retrying at all...

pandawankenobi commented 4 years ago

True. here are my settings


AUTH_ADFS = {
    "SERVER": "adfs.my-job.dk",
    "CLIENT_ID": "secret",
    "RELYING_PARTY_ID": "secret",
    "AUDIENCE": "microsoft:identityserver:secret",
    "CLAIM_MAPPING": {"first_name": "given_name",
                      "last_name": "family_name",
                      "email": "email"},
    "USERNAME_CLAIM": "winaccountname",
    "GROUPS_CLAIM": "group",
    "MIRROR_GROUPS": False,  # skal sættes til true hvis der skal oprettes nye grupper på baggrund af AD - mistænkt for at sløve
    "TIMEOUT": 15,
    "RETRIES": 24
}
pandawankenobi commented 4 years ago

so i would expect 24 retries

jobec commented 4 years ago

https://github.com/psf/requests/blob/d2590ee46c0641958b6d4792a206bd5171cb247d/requests/adapters.py#L94-L100

 :param max_retries: The maximum number of retries each connection
    should attempt. Note, this applies only to failed DNS lookups, socket
    connections and connection timeouts, never to requests where data has
    made it to the server. By default, Requests does not retry failed
    connections. If you need granular control over the conditions under
    which we retry a request, import urllib3's ``Retry`` class and pass
    that instead.

Interesting... Things never really work as you expect them to.

Ok, so it seems that Requests instructs urllib3 not to retry when the request made it to the server (read=False). This to prevent any side effect of the second request. With a custom Retry this seems to be bypass-able.

Let me see if I can simulate it and get it to retry as expected. I think it could fix your issue.

jobec commented 4 years ago

Could you try with the code in the improved-retry branch?

pip install --force-reinstall git+https://github.com/jobec/django-auth-adfs.git@improved-retry
pandawankenobi commented 4 years ago

i'm using docker so i removed the package from requirements and added


# ADFS with improved retry
RUN apk add --no-cache git
RUN pip install git+https://github.com/jobec/django-auth-adfs.git@improved-retry

i will let it run for a day, and see if it is fixed. I normally get this error ten times a day. fingers crossed :)

pandawankenobi commented 4 years ago

It seems it is not fixed. I got the error again. Here is the sentry error repport. I will post the docker console log i a minute:

EXCEPTION(most recent call last)
timeout: The read operation timed out

  File "urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from

  File "urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "http/client.py", line 1331, in getresponse
    response.begin()
  File "http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
  File "ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
  File "ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)

ReadTimeoutError: HTTPSConnectionPool(host='adfs.my-job.dk', port=443): Read timed out. (read timeout=5)
  File "requests/adapters.py", line 449, in send
    timeout=timeout
  File "urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "urllib3/util/retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "urllib3/packages/six.py", line 686, in reraise
    raise value
  File "urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "urllib3/connectionpool.py", line 386, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "urllib3/connectionpool.py", line 306, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)

ReadTimeout: HTTPSConnectionPool(host='adfs.my-job.dk', port=443): Read timed out. (read timeout=5)
  File "django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "django/core/handlers/base.py", line 126, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "django/core/handlers/base.py", line 124, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "django/views/generic/base.py", line 88, in dispatch
    return handler(request, *args, **kwargs)
  File "django_auth_adfs/views.py", line 34, in get
    user = authenticate(request=request, authorization_code=code)
  File "django/contrib/auth/__init__.py", line 73, in authenticate
    user = backend.authenticate(request, **credentials)
  File "django_auth_adfs/backend.py", line 264, in authenticate
    adfs_response = self.exchange_auth_code(authorization_code, request)
  File "django_auth_adfs/backend.py", line 28, in exchange_auth_code
    response = provider_config.session.post(provider_config.token_endpoint, data, timeout=settings.TIMEOUT)
  File "requests/sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
pandawankenobi commented 4 years ago

dockerlog:


2019-11-05T09:30:39.021988316Z retry decorator active - not retrying
2019-11-05T09:30:39.023829221Z 172.16.0.1 - - [05/Nov/2019:09:30:39 +0000] "GET /vikar/bestil/ HTTP/1.1" 302 0 "https://my-site.azurewebsites.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36" "5.186.115.73:1638"
2019-11-05T09:30:39.025633526Z [pid: 13|app: 0|req: 36/53] 172.16.0.1 () {80 vars in 1726 bytes} [Tue Nov  5 10:30:39 2019] GET /vikar/bestil/ => generated 0 bytes in 5 msecs (HTTP/1.1 302) 5 headers in 169 bytes (1 switches on core 0)
2019-11-05T09:30:39.070764949Z 172.16.0.1 - - [05/Nov/2019:09:30:39 +0000] "GET /oauth2/login?next=/vikar/bestil/ HTTP/1.1" 302 0 "https://my-site.azurewebsites.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36" "5.186.115.73:1638"
2019-11-05T09:30:39.073242356Z [pid: 14|app: 0|req: 18/54] 172.16.0.1 () {80 vars in 1801 bytes} [Tue Nov  5 10:30:39 2019] GET /oauth2/login?next=/vikar/bestil/ => generated 0 bytes in 3 msecs (HTTP/1.1 302) 4 headers in 402 bytes (1 switches on core 0)
2019-11-05T09:30:40.219287602Z Received authorization code: 7mfkUeLYjUW2Lr_U7cc4lA.Y70n0tJh1wiQAUM9XqRQuHTaG80.QPMDLwndKfMTUAfRK75mHKdQvR06WJdEVDgNCtWc5mmIb8-S9Udpuak-y8jE-9uiLgWwiVnI1HCzq09_KHmuKCZokC9KObbyLe-T8KdPY15b_dIr90BNyuO2XXvg21flsraegvEovNaq77MHlJMjWoxndZ96vZ1jl0m3LHYnAKcxzsVHOSWD96s-JNspWEl8JezASPuu7tvQX2_Ernv0iL2Rn18eb6_zFot3yRVgrYkymRPU6mcFRnxSivTHAFvqHUvhuuASAAVVp3iqvCnM-RuJZyxpmBBNKJfIOgwq0eNXkPGq9vxTVRR6q3Z5FWcA-ELSFGFhjDbm1QbihNr_IA
2019-11-05T09:30:40.220932307Z Getting access token at: https://adfs.my-job.dk/adfs/oauth2/token/

2019-11-05T09:30:45.370105041Z Starting new HTTPS connection (1): sentry.io:443
2019-11-05T09:30:45.405147138Z Internal Server Error: /oauth2/callback
2019-11-05T09:30:45.405178338Z Traceback (most recent call last):
2019-11-05T09:30:45.405185138Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
2019-11-05T09:30:45.405189638Z     six.raise_from(e, None)
2019-11-05T09:30:45.405193538Z   File "<string>", line 2, in raise_from
2019-11-05T09:30:45.405198038Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
2019-11-05T09:30:45.405202138Z     httplib_response = conn.getresponse()
2019-11-05T09:30:45.405206138Z   File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/stdlib.py", line 108, in getresponse
2019-11-05T09:30:45.405219738Z     rv = real_getresponse(self, *args, **kwargs)
2019-11-05T09:30:45.405223938Z   File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
2019-11-05T09:30:45.405227738Z     response.begin()
2019-11-05T09:30:45.405231338Z   File "/usr/lib/python3.6/http/client.py", line 297, in begin
2019-11-05T09:30:45.405235138Z     version, status, reason = self._read_status()
2019-11-05T09:30:45.405238838Z   File "/usr/lib/python3.6/http/client.py", line 258, in _read_status
2019-11-05T09:30:45.405242538Z     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
2019-11-05T09:30:45.405246338Z   File "/usr/lib/python3.6/socket.py", line 586, in readinto
2019-11-05T09:30:45.405250038Z     return self._sock.recv_into(b)
2019-11-05T09:30:45.405253638Z   File "/usr/lib/python3.6/ssl.py", line 1009, in recv_into
2019-11-05T09:30:45.405257338Z     return self.read(nbytes, buffer)
2019-11-05T09:30:45.405260938Z   File "/usr/lib/python3.6/ssl.py", line 871, in read
2019-11-05T09:30:45.405264638Z     return self._sslobj.read(len, buffer)
2019-11-05T09:30:45.405268238Z   File "/usr/lib/python3.6/ssl.py", line 631, in read
2019-11-05T09:30:45.405271938Z     v = self._sslobj.read(len, buffer)
2019-11-05T09:30:45.405275538Z socket.timeout: The read operation timed out
2019-11-05T09:30:45.405279238Z 
2019-11-05T09:30:45.405282738Z During handling of the above exception, another exception occurred:
2019-11-05T09:30:45.405286538Z 
2019-11-05T09:30:45.405290038Z Traceback (most recent call last):
2019-11-05T09:30:45.405293738Z   File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2019-11-05T09:30:45.405297538Z     timeout=timeout
2019-11-05T09:30:45.405301138Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
2019-11-05T09:30:45.405304838Z     _stacktrace=sys.exc_info()[2])
2019-11-05T09:30:45.405308438Z   File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 367, in increment
2019-11-05T09:30:45.405312238Z     raise six.reraise(type(error), error, _stacktrace)
2019-11-05T09:30:45.405315938Z   File "/usr/local/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
2019-11-05T09:30:45.405319738Z     raise value
2019-11-05T09:30:45.405323238Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2019-11-05T09:30:45.405327038Z     chunked=chunked)
2019-11-05T09:30:45.405331638Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 386, in _make_request
2019-11-05T09:30:45.405335538Z     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
2019-11-05T09:30:45.405342338Z   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 306, in _raise_timeout
2019-11-05T09:30:45.405346338Z     raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
2019-11-05T09:30:45.405350238Z urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='adfs.my-job.dk', port=443): Read timed out. (read timeout=5)
2019-11-05T09:30:45.405354038Z 
2019-11-05T09:30:45.405357438Z During handling of the above exception, another exception occurred:
2019-11-05T09:30:45.405361138Z 
2019-11-05T09:30:45.405364638Z Traceback (most recent call last):
2019-11-05T09:30:45.405368338Z   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
2019-11-05T09:30:45.405438538Z     response = get_response(request)
2019-11-05T09:30:45.405442338Z   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
2019-11-05T09:30:45.405446238Z     response = self.process_exception_by_middleware(e, request)
2019-11-05T09:30:45.405449938Z   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 124, in _get_response
2019-11-05T09:30:45.405453738Z     response = wrapped_callback(request, *callback_args, **callback_kwargs)
2019-11-05T09:30:45.405457438Z   File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 68, in view
2019-11-05T09:30:45.405461338Z     return self.dispatch(request, *args, **kwargs)
2019-11-05T09:30:45.405465038Z   File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 88, in dispatch
2019-11-05T09:30:45.405468838Z     return handler(request, *args, **kwargs)
2019-11-05T09:30:45.405472538Z   File "/usr/local/lib/python3.6/site-packages/django_auth_adfs/views.py", line 34, in get
2019-11-05T09:30:45.405486039Z     user = authenticate(request=request, authorization_code=code)
2019-11-05T09:30:45.405490639Z   File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/__init__.py", line 73, in authenticate
2019-11-05T09:30:45.405494439Z     user = backend.authenticate(request, **credentials)
2019-11-05T09:30:45.405498239Z   File "/usr/local/lib/python3.6/site-packages/django_auth_adfs/backend.py", line 264, in authenticate
2019-11-05T09:30:45.405502039Z     adfs_response = self.exchange_auth_code(authorization_code, request)
2019-11-05T09:30:45.405505939Z   File "/usr/local/lib/python3.6/site-packages/django_auth_adfs/backend.py", line 28, in exchange_auth_code
2019-11-05T09:30:45.405511539Z     response = provider_config.session.post(provider_config.token_endpoint, data, timeout=settings.TIMEOUT)
2019-11-05T09:30:45.405515539Z   File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 581, in post
2019-11-05T09:30:45.405519339Z     return self.request('POST', url, data=data, json=json, **kwargs)
2019-11-05T09:30:45.405523039Z   File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
2019-11-05T09:30:45.405531639Z     resp = self.send(prep, **send_kwargs)
2019-11-05T09:30:45.405535539Z   File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
2019-11-05T09:30:45.405539339Z     r = adapter.send(request, **kwargs)
2019-11-05T09:30:45.405543039Z   File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 529, in send
2019-11-05T09:30:45.405546839Z     raise ReadTimeout(e, request=request)
2019-11-05T09:30:45.410740553Z requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='adfs.my-job.dk', port=443): Read timed out. (read timeout=5)
2019-11-05T09:30:45.413074659Z 172.16.0.1 - - [05/Nov/2019:09:30:45 +0000] "GET /oauth2/callback?code=7mfkUeLYjUW2Lr_U7cc4lA.Y70n0tJh1wiQAUM9XqRQuHTaG80.QPMDLwndKfMTUAfRK75mHKdQvR06WJdEVDgNCtWc5mmIb8-S9Udpuak-y8jE-9uiLgWwiVnI1HCzq09_KHmuKCZokC9KObbyLe-T8KdPY15b_dIr90BNyuO2XXvg21flsraegvEovNaq77MHlJMjWoxndZ96vZ1jl0m3LHYnAKcxzsVHOSWD96s-JNspWEl8JezASPuu7tvQX2_Ernv0iL2Rn18eb6_zFot3yRVgrYkymRPU6mcFRnxSivTHAFvqHUvhuuASAAVVp3iqvCnM-RuJZyxpmBBNKJfIOgwq0eNXkPGq9vxTVRR6q3Z5FWcA-ELSFGFhjDbm1QbihNr_IA&state=L3Zpa2FyL2Jlc3RpbC8= HTTP/1.1" 500 27 "https://my-site.azurewebsites.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36" "5.186.115.73:1638"
2019-11-05T09:30:45.414489963Z [pid: 14|app: 0|req: 19/55] 172.16.0.1 () {80 vars in 3436 bytes} [Tue Nov  5 10:30:39 2019] GET /oauth2/callback?code=7mfkUeLYjUW2Lr_U7cc4lA.Y70n0tJh1wiQAUM9XqRQuHTaG80.QPMDLwndKfMTUAfRK75mHKdQvR06WJdEVDgNCtWc5mmIb8-S9Udpuak-y8jE-9uiLgWwiVnI1HCzq09_KHmuKCZokC9KObbyLe-T8KdPY15b_dIr90BNyuO2XXvg21flsraegvEovNaq77MHlJMjWoxndZ96vZ1jl0m3LHYnAKcxzsVHOSWD96s-JNspWEl8JezASPuu7tvQX2_Ernv0iL2Rn18eb6_zFot3yRVgrYkymRPU6mcFRnxSivTHAFvqHUvhuuASAAVVp3iqvCnM-RuJZyxpmBBNKJfIOgwq0eNXkPGq9vxTVRR6q3Z5FWcA-ELSFGFhjDbm1QbihNr_IA&state=L3Zpa2FyL2Jlc3RpbC8= => generated 27 bytes in 6068 msecs (HTTP/1.1 500) 4 headers in 126 bytes (1 switches on core 1)
2019-11-05T09:30:45.886355759Z https://sentry.io:443 "POST /api/1401487/store/ HTTP/1.1" 200 41
pandawankenobi commented 4 years ago

i'm runnning debug=false currently. If you need me to activate it, let me know.

jobec commented 4 years ago

Ok... one step further :) It fails on a POST request, which is not in the retry "whitelist" of urllib3:

https://github.com/urllib3/urllib3/blob/a6ec68a5c5c5743c59fe5c62c635c929586c429b/src/urllib3/util/retry.py#L149

Let's add it and try again.

pandawankenobi commented 4 years ago

ok!. let me know when you are ready.

edit: the retry times are correct. i just confused the logs

jobec commented 4 years ago

Done (https://github.com/jobec/django-auth-adfs/commit/def450cd1c92d6d0b077be8092e6dccefd05efc2)

pandawankenobi commented 4 years ago

ok - now we wait.

pandawankenobi commented 4 years ago

while it is hard to prove a negative things are looking good. I noticed this little new bit of log when a log on took longer than usual (10 secs).

2019-11-05T12:50:40.171724509Z Received authorization code: code
2019-11-05T12:50:40.173755415Z Getting access token at: https://adfs.myjobb.dk/adfs/oauth2/token/

2019-11-05T12:50:45.181397835Z Incremented Retry for (url='/adfs/oauth2/token/'): Retry(total=23, connect=24, read=23, redirect=None, status=None)
2019-11-05T12:50:45.182550438Z Retrying (Retry(total=23, connect=24, read=23, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='adfs.myjob.dk', port=443): Read timed out. (read timeout=5)",)': /adfs/oauth2/token/
2019-11-05T12:50:45.183167240Z Starting new HTTPS connection (5): adfs.myjob.dk:443
2019-11-05T12:50:46.422369635Z https://adfs.myjob.dk:443 "POST /adfs/oauth2/token/ HTTP/1.1" 200 20042
2019-11-05T12:50:46.464033050Z Received access token: jwtToken
2019-11-05T12:50:46.473402875Z Attribute 'first_name' for user 'me' was set to 'myname'.
2019-11-05T12:50:46.474266378Z Attribute 'last_name' for user 'me' was set to 'myname'.
2019-11-05T12:50:46.475017080Z Attribute 'email' for user 'me' was set to 'mymail'.
2019-11-05T12:50:47.389245785Z 172.16.0.1 - - [05/Nov/2019:12:50:47 +0000] "GET /oauth2/callback?code HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36 OPR/64.0.3417.83" "5.186.115.73:1557"
2019-11-05T12:50:47.391137290Z [pid: 14|app: 0|req: 31/124] 172.16.0.1 () {78 vars in 3406 bytes} [Tue Nov  5 13:50:39 2019] GET /oauth2/callback?code=code= => generated 0 bytes in 8213 msecs (HTTP/1.1 302) 7 headers in 407 bytes (1 switches on core 0)
2019-11-05T12:50:47.447134343Z retry decorator active - not retrying
2019-11-05T12:50:49.401781199Z 172.16.0.1 - - [05/Nov/2019:12:50:49 +0000] "GET /vikar/bestil/ HTTP/1.1" 200 13496 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36 OPR/64.0.3417.83" "5.186.115.73:1557"
2019-11-05T12:50:49.403947705Z [pid: 14|app: 0|req: 32/125] 172.16.0.1 () {78 vars in 1739 bytes} [Tue Nov  5 13:50:47 2019] GET /vikar/bestil/ => generated 13496 bytes in 1956 msecs (HTTP/1.1 200) 5 headers in 293 bytes (1 switches on core 1)

2019-11-05T12:50:54.014249438Z 172.16.0.1 - - [05/Nov/2019:12:50:54 +0000] "GET / HTTP/1.1" 200 4214 "-" "AlwaysOn" "[::1]:12373"
2019-11-05T12:50:54.016111444Z [pid: 15|apsp: 0|req: 94/126] 172.16.0.1 () {58 vars in 858 bytes} [Tue Nov  5 13:50:54 2019] GET / => generated 4214 bytes in 4 msecs (HTTP/1.1 200) 4 headers in 124 bytes (1 switches on core 1)

so it appears retries happen now. I will report back tomorrow.

jobec commented 4 years ago

That is indeed the kind of log entries you should be seeing. You'll now need to find a balance in the timeout value and the retry value.

pandawankenobi commented 4 years ago

it works. no errors for a day 👍

Do you know when this might make it's way to pypi?

jobec commented 4 years ago

I released 1.3.1 just now

pandawankenobi commented 4 years ago

Thank you very much for all your help 👍