How to increase the timeout threshold for a download?

NathanWailes commented 5 years ago

Hi Itay,

Background

I'm working on a project that is using pySmartDL to download media files from a server.
The server had been using the HTTP protocol but just switched to HTTPS.
Now downloads take a long time to get started (even without using pySmartDL; it seems to be some issue with the server).

The problem

Unfortunately these more-slow download-starts seem to cause pySmartDL to hit a URLError exception on line 129 of pySmartDL.py, where it makes a call to utils.is_HTTPRange_supported(self.url).

A potential solution

I think I might be able to avoid the exception if I could pass in a larger timeout threshold to that function, for example by passing in a timeout argument to the SmartDL() call.

The stack trace:

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python3.6/urllib/request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/usr/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/usr/lib/python3.6/http/client.py", line 1400, in connect
    server_hostname=server_hostname)
  File "/home/vagrant/.local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 392, in wrap_socket
    return GreenSSLSocket(sock, *a, _context=self, **kw)
  File "/home/vagrant/.local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 87, in __init__
    self.do_handshake()
  File "/home/vagrant/.local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 260, in do_handshake
    super(GreenSSLSocket, self).do_handshake)
  File "/home/vagrant/.local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 115, in _call_trampolining
    timeout_exc=timeout_exc('timed out'))
  File "/home/vagrant/.local/lib/python3.6/site-packages/eventlet/hubs/__init__.py", line 164, in trampoline
    return hub.switch()
  File "/home/vagrant/.local/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 297, in switch
    return self.greenlet.switch()
ssl.SSLError: ('timed out',)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/vagrant/src/__init__.py", line 131, in wrapper_function
    thread_function(*args, **kwargs)
  File "/vagrant/src/__init__.py", line 199, in media_downloader_service
    download_queued_media(retry_failed_downloads=True)
  File "/vagrant/src/DKServerAPI/download_queued_media.py", line 52, in download_queued_media
    download_was_successful = _download_media(session, queued_download, tick_step_size, retry_failed_downloads)
  File "/vagrant/src/DKServerAPI/download_queued_media.py", line 104, in _download_media
    smart_downloader = SmartDL(full_url_to_download, media_item_specific_download_folder, progress_bar=False)
  File "/home/vagrant/.local/lib/python3.6/site-packages/pySmartDL/pySmartDL.py", line 129, in __init__
    if not utils.is_HTTPRange_supported(self.url):
  File "/home/vagrant/.local/lib/python3.6/site-packages/pySmartDL/utils.py", line 107, in is_HTTPRange_supported
    urlObj = urllib.request.urlopen(req, timeout=timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib/python3.6/urllib/request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error ('timed out',)>

iTaybb commented 5 years ago

The timeout parameter should really be passed as an optional argument to the SmartDL constructor. Furthermore it seems that the utils.is_HTTPRange_supported is not using the global SmartDL timeout.

I'll make a patch for that later today.

NathanWailes commented 5 years ago

Thank you!

iTaybb commented 5 years ago

I've added the option. Can you check it's solving your issue?

NathanWailes commented 5 years ago

Unfortunately it seems the server is responding quickly enough now that it isn't causing an error even with the version of pySmartDL I was using before this update you pushed, so I can't verify whether this will fix the issue if it pops up again, but I suspect it will. If I see the issue tomorrow (when there may be more load on the server) I'll be sure to try to use the constructor parameter and I'll let you know if it works or not.

Here's the link to the video if you want to test it yourself: https://staging.doohken.com/storage/4/fillervideo-skateboarding_screen.mp4

iTaybb / pySmartDL