search

simonw / google-drive-to-sqlite

Create a SQLite database containing metadata from Google Drive
https://datasette.io/tools/google-drive-to-sqlite
Apache License 2.0
152 stars 13 forks source link

httpx TimeoutError #8

Closed simonw closed 2 years ago

simonw commented 2 years ago

Got this exception while testing code from #7:

  File "/Users/simon/Dropbox/Development/google-drive-to-sqlite/google_drive_to_sqlite/utils.py", line 25, in paginate_files
    data = httpx.get(
  File "/Users/simon/.local/share/virtualenvs/google-drive-to-sqlite-Wr1nXkpK/lib/python3.10/site-packages/httpx/_api.py", line 189, in get
    return request(
  File "/Users/simon/.local/share/virtualenvs/google-drive-to-sqlite-Wr1nXkpK/lib/python3.10/site-packages/httpx/_api.py", line 100, in request
    return client.request(
  File "/Users/simon/.local/share/virtualenvs/google-drive-to-sqlite-Wr1nXkpK/lib/python3.10/site-packages/httpx/_client.py", line 802, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/Users/simon/.local/share/virtualenvs/google-drive-to-sqlite-Wr1nXkpK/lib/python3.10/site-packages/httpx/_client.py", line 889, in send
    response = self._send_handling_auth(
  File "/Users/simon/.local/share/virtualenvs/google-drive-to-sqlite-Wr1nXkpK/lib/python3.10/site-packages/httpx/_client.py", line 917, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/Users/simon/.local/share/virtualenvs/google-drive-to-sqlite-Wr1nXkpK/lib/python3.10/site-packages/httpx/_client.py", line 954, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/Users/simon/.local/share/virtualenvs/google-drive-to-sqlite-Wr1nXkpK/lib/python3.10/site-packages/httpx/_client.py", line 990, in _send_single_request
    response = transport.handle_request(request)
  File "/Users/simon/.local/share/virtualenvs/google-drive-to-sqlite-Wr1nXkpK/lib/python3.10/site-packages/httpx/_transports/default.py", line 217, in handle_request
    with map_httpcore_exceptions():
  File "/Users/simon/.pyenv/versions/3.10.0/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/simon/.local/share/virtualenvs/google-drive-to-sqlite-Wr1nXkpK/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout: The read operation timed out
simonw commented 2 years ago

https://www.python-httpx.org/advanced/#timeout-configuration says default is 5s - I'm going to bump that up to 10s.

simonw commented 2 years ago

Bumping it up to 10s is good enough for the moment.

simonw commented 2 years ago

Hit this again, during a really long --folder operation.

It's frustrating because it means I have to start the thing running again - it loses its context.

simonw commented 2 years ago

I think I want an exponential backoff retry here.

simonw commented 2 years ago

Or... bump it up to 30s, which seems to work fine.