search

Deepomatic / deepomatic-client-python

Python Client
Other
14 stars 0 forks source link

Network upload fails after ~3s: requests.exceptions.ConnectionError: ('Connection aborted.', timeout('The write operation timed out',)) #77

Closed thomas-riccardi closed 4 years ago

thomas-riccardi commented 4 years ago

When creating a Network we sometimes get a timeout error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.5/http/client.py", line 1106, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1151, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1102, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 936, in _send_output
    self.send(message_body)
  File "/usr/lib/python3.5/http/client.py", line 908, in send
    self.sock.sendall(data)
  File "/usr/lib/python3.5/ssl.py", line 891, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python3.5/ssl.py", line 861, in send
    return self._sslobj.write(data)
  File "/usr/lib/python3.5/ssl.py", line 586, in write
    return self._sslobj.write(data)
socket.timeout: The write operation timed out
thomas-riccardi commented 4 years ago

In #58 we used advanced timeout control in requests: small connect timeout (3.05s) then larger read timeout (idle timeout between 2 effective data reads).

However, it seems that connect timeout does not mean TCP connect timeout as we expected, but in fact it's TCP connect + HTTP request write timeout, which is clearly larger than expected for network uploads (N*100MB to upload...).

thomas-riccardi commented 4 years ago

78 is a hotfix, we should still have a better fix later, once the issue is resolved upstream: https://github.com/psf/requests/issues/5263

thomas-riccardi commented 4 years ago

Also, we are probably impacted by https://github.com/psf/requests/issues/4402#issuecomment-415748368 ...

thomas-riccardi commented 3 years ago

requests project finally responded to my 1 year old issue: it's a behavior change in python 3.5 socket.sendall(): https://github.com/psf/requests/issues/5263#issuecomment-753620169

It seems the current behavior may be there for a long time.