gauteh / lieer

Fast email-fetching, sending, and two-way tag synchronization between notmuch and GMail
http://lieer.gaute.vetsj.com
Other
555 stars 62 forks source link

Sync consistently times out #221

Closed nshi closed 2 years ago

nshi commented 2 years ago

I have set up Lieer successfully in two other Gmail accounts with less emails. However, the last one with over 100k emails consistently fails with the following error during the initial sync. The stack trace looks like the HTTPS connection somehow drops in the middle which Lieer doesn't expect and there's no retry. Not sure if anyone has seen this before. Is there anyway to work around this?

I'm using Lieer 1.3-6 with Python 3.9.5 on Ubuntu 22.04.

pull: full synchronization (no previous synchronization state)
fetching messages: 102554it [01:44, 985.43it/s]
removing deleted: 0it [00:00, ?it/s]
receiving content:   2%|█▍                                                               | 2298/102554 [01:52<32:44, 51.04it/s]
remote: reducing batch request size to: 25
receiving content:   2%|█▍                                                             | 2318/102554 [02:01<2:38:43, 10.53it/s]
remote: increasing batch request size to: 50
receiving content:  64%|█████████████████████████████████████████                       | 65773/102554 [41:55<24:14, 25.29it/s]
Traceback (most recent call last):
  File "/usr/bin/gmi", line 24, in <module>
    g.main ()
  File "/usr/share/lieer/lieer/gmailieer.py", line 219, in main
    args.func (args)
  File "/usr/share/lieer/lieer/gmailieer.py", line 303, in sync
    self.pull (args, True)
  File "/usr/share/lieer/lieer/gmailieer.py", line 416, in pull
    self.full_pull ()
  File "/usr/share/lieer/lieer/gmailieer.py", line 661, in full_pull
    updated = self.get_content(message_gids)
  File "/usr/share/lieer/lieer/gmailieer.py", line 756, in get_content
    self.remote.get_messages (need_content, _got_msgs, 'raw')
  File "/usr/share/lieer/lieer/remote.py", line 133, in func_wrap
    return func (self, *args, **kwargs)
  File "/usr/share/lieer/lieer/remote.py", line 334, in get_messages
    batch.execute (http = self.http)
  File "/usr/lib/python3/dist-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/googleapiclient/http.py", line 1455, in execute
    self._execute(http, self._order, self._requests)
  File "/usr/lib/python3/dist-packages/googleapiclient/http.py", line 1386, in _execute
    resp, content = http.request(self._batch_uri, method='POST', body=body,
  File "/usr/lib/python3/dist-packages/oauth2client/transport.py", line 173, in new_request
    resp, content = request(orig_request_method, uri, method, body,
  File "/usr/lib/python3/dist-packages/oauth2client/transport.py", line 280, in request
    return http_callable(uri, method=method, body=body, headers=headers,
  File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1725, in request
    (response, content) = self._request(
  File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1441, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1393, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1374, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.10/ssl.py", line 1273, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.10/ssl.py", line 1129, in read
    return self._sslobj.read(len, buffer)
TimeoutError: The read operation timed out
receiving content:  64%|█████████████████████████████████████████                       | 65773/102554 [51:40<28:53, 21.21it/s]
nshi commented 2 years ago

I finally got it to finish the initial sync by running gmi pull --resume multiple times.