theelous3 / asks

Async requests-like httplib for python.
MIT License
508 stars 63 forks source link

Asks throwing errors on certain TLS enabled connections that requests doesn't #136

Open PortWhisperer opened 5 years ago

PortWhisperer commented 5 years ago

Results are as follows (I'm unable to find information on how to proxy the traffic from asks to inspect decrypted contents of the HTTPS traffic to give further details).

Wireshark appears to indicate the TLS handshake was successful because application layer data appears in the PCAP after the handshake traffic, but that data is encrypted.

Note: h11 is mentioned in the traceback for the Invalid HTTP Response error I'm getting -- unsure whether it's related to /issues/107

Asks Library


SSLEOFERROR: << fixed by using pip to install latest version of asks (conda-forge is behind) asks.get('https://IntranetResource', auth=BasicAuth(usr_pw)) Invalid HTTP Response Error: asks.get('https://IntranetResource/rest/api/path',auth=BasicAuth(usr_pw)) Body of page is returned as expected: asks.get('https://google.com')

Requests Library


Body of page is returned as expected: requests.get('https://url', auth=HTTPBasicAuth=(usr_pw),verify=False) Proper response returned from API as expected: requests.get('https://url/rest/api/path', auth=HTTPBasicAuth=(usr_pw),verify=False)

PortWhisperer commented 5 years ago

The Invalid HTTP Response stack trace, which i can't easily reproduce because I'm not able to copy/paste from the machine, shows an error in response_obj from asks/request_object.py

[full paths redacted]
File "....\asks\sessions.py", line 185, in request
   sock, r = await req_obj.make_request()
file ....\asks\request_object.py", line 214, in make_request
   response_obj = await self._request_io(req, req_body, h11_connection)
file "...\asks\request_object.py", line 251, in _request_io
   response_obj = await self._catch_response(h11_connection)
file"...\asks\request_object.py", line 599, in _catch_response
   assert isinstance(endof, h11.EndOfMessage)
AssertionError

The above exception was the direct cause of the following exception
[some output redacted]
file "...\trio\_core\_run.py", in 1783, in run
    raise runner.main_task_outcome.error
file "...\my_script_thats_having_this_error, line 33, in main
   n.start_soon(grabber,s)
File"....\trio\_core\_run.py", line 725 in __aexit__ 
   raise combined_error_from_nursery
File "...\my_script_thats_having_this_error, line 30 in request
   r = await s.request(method, url=uri, **kwargs)
File "...\asks\sessions.py", line 215, in request
   await self._handle_exception(e,sock)
File "...\asks\sessions.py", line 253, in _handle_exception
    raise BadHttpResponse('invalid HTTP response from server. ') from e asks.errors.BadHttpResponse: Invalid HTTP response from server.
PortWhisperer commented 5 years ago

The SSLEOFERROR traceback (handtyped so there are maybe a couple of typos) is below. paths are slightly redacted.

Traceback (most recent call last):
line 463 in _retry 
   ret = fn(*args)
line 718 in read
   v= self._sslobj.read(len)
ssl.SSLEOFError: EOF occured in violation of protocol (_ssl.c:2508)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
line 90, in runcode
    exec(code, self.locals)
File "<input>", line 28, in <module> run.py line 1783 in run
    raise runner.main_task_outcome.error
file"<input>", line 26 in main
file _run.py, line 725, in __aexit__ 
     raise combined error_from_nursery
file"<input>", line 15 in grabber
File "asks/sessions.py", line 253, in _handle_exception
     raise e
File ...asks\sessions.py" , line 186 in request
     sock,r = await req_obj.make_request()
file ...request_object.py ", line 217 in make_request
     response obj=await self._request_io(req, req_body, hconnection)
file ..request_object.py", line 254 in _request_io
     response_obj = await 
     self._catch_response(hcconection)
file ..request_object.py", line 596, in _catch_response
     data=await self._recv_event(hcconection)
file "...request_object.py", line 618, in _recv_event
     (await asynclib.recv(self.sock,10000)))
File "..._event_loop_wrappers.py", line 47 in trio_receive_some
     return await sock.receive_some(max_bytes)
file "..._ssl.py" line 657 in receive_some
     return await self._retry(self._ssl_object.read, max_bytes)
File"....trio\_ssl.py", line 468 in _retry
     raise trio.BrokenResourceerror from exc
     Trio.BrokenResourceError
PortWhisperer commented 5 years ago

The SSLEOFERROR seems to have been fixed by updating to asks 2.3.5 (conda-forge still has an older version).

now i'm able to pull pages from this site successfully. Everything is working, except requesting anything from the REST API causes asks to crash with invalid http response errors.

The content-type of responses from the REST API are text/plain; charset utf-8, and according to my proxy server the response code when request from my browser is a 200 with no intervening redirects.

Working https://site https://site/nonexistent/page https://site/nonex https://site/nonexi https://site/nonexis https://site/nonexist

Not working https://site/restapi/ https://site/restapi/payload_to_pass_to_the_api

The only thing im modifying between requests is the URL. Error is the above referenced Invalid HTTP reseponse error related to h11. Considering the server's response is plaintext, the error is surprising.

theelous3 commented 5 years ago

Sorry for the delays around here, I'm swamped IRL at the moment. I'm going to need to take some time to dig in to this and that other odd issue. I agree, this is surprising. Thanks for the write up :)