Do not crash if downloading a tile temporary fails

[Tapter]

When downloading a bunch of tiles to create an mbtiles file, the process is interrupted (program crash) if downloading one tile fails for an unknown reason (probably a temporary server error or something). It would be nice if this error would just be treated transparently by either retrying a few times or ignoring that tile (and reporting that tiles were skipped in the end).

Here's the stacktrace of such an event:

DEBUG:landez.tiles:tile method called with [15, 17144, 11422]
DEBUG:landez.sources:Request WMS tile (15, 17144, 11422)
DEBUG:landez.sources:Download 'https://...'
WARNING:landez.tiles:''
Traceback (most recent call last):
  File "dtk.py", line 30, in <module>
    mb.run()
  File "/usr/local/lib/python2.7/dist-packages/landez-2.4.0-py2.7.egg/landez/tiles.py", line 322, in run
    self._gather((z, x, y))
  File "/usr/local/lib/python2.7/dist-packages/landez-2.4.0-py2.7.egg/landez/tiles.py", line 374, in _gather
    tilecontent = self.tile((z, x, y))
  File "/usr/local/lib/python2.7/dist-packages/landez-2.4.0-py2.7.egg/landez/tiles.py", line 172, in tile
    output = self.reader.tile(z, x, y)
  File "/usr/local/lib/python2.7/dist-packages/landez-2.4.0-py2.7.egg/landez/sources.py", line 231, in tile
    f = urllib2.urlopen(request)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1241, in https_open
    context=self._context)
  File "/usr/lib/python2.7/urllib2.py", line 1201, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1121, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 438, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 402, in _read_status
    raise BadStatusLine(line)
httplib.BadStatusLine: ''

[leplatrem]

It is a bug I believe.

We are supposed to retry when the download fails: https://github.com/makinacorpus/landez/blob/8df5a9f22284395e01101b526a13609544484f87/landez/sources.py#L172-L188

It looks like this exception is not caught...

[ben-katan]

any progress on this issue?