search

gauteh / lieer

Fast email-fetching and sending and two-way tag synchronization between notmuch and GMail
http://lieer.gaute.vetsj.com
Other
497 stars 60 forks source link

gmi pull keeps crashing #43

Closed manishrjain closed 6 years ago

manishrjain commented 6 years ago 
$ gmi pull                                                                                                                                                              /data/Mail/personal
pull: full synchronization (no previous synchronization state)
fetching messages: 139863it [12:33, 153.90it/s]                                                                                                                receiving content: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 57/57 [02:58<00:00,  3.28s/it]receiving metadata:   3%|██▌                                                                                             | 3683/139806 [01:14<38:10, 59.42it/s]Traceback (most recent call last):
  File "/usr/bin/gmi", line 4, in <module>
    __import__('pkg_resources').run_script('gmailieer==0.2', 'gmi')
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 742, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1510, in run_script
    exec(script_code, namespace, namespace)
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/EGG-INFO/scripts/gmi", line 8, in <module>
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 136, in main
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 307, in pull
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 531, in full_pull
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 562, in get_meta
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/remote.py", line 100, in func_wrap
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/remote.py", line 271, in get_messages
  File "/usr/lib/python3.6/site-packages/oauth2client-4.1.2-py3.6.egg/oauth2client/_helpers.py", line 133, in positional_wrapper
  File "/usr/lib/python3.6/site-packages/google_api_python_client-1.6.3-py3.6.egg/googleapiclient/http.py", line 1464, in execute
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/remote.py", line 251, in _cb
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/gmailieer.py", line 560, in _got_msg
  File "/usr/lib/python3.6/site-packages/gmailieer-0.2-py3.6.egg/lieer/local.py", line 326, in update_tags
lieer.local.RepositoryException: tried to update tags on non-existant file: /data/Mail/personal/mail/cur/12e10631fc4211b6:2,

This is right after the initial pull (which also crashed with similar stack trace).

gauteh commented 6 years ago

Do you get the same error with #42?

gauteh commented 6 years ago

Do you have anything working on the maildir at the same time? Anything notmuchy that could have resulted in the maildirflags being changed? (just viewing the email in mutt.. ?)

manishrjain commented 6 years ago

Not using mutt, or offlineimap or anything else. I pulled in all the changes from master. Still the same issue.

$ gmi pull                                                                                                                                 /data/Mail/personal
pull: full synchronization (no previous synchronization state)
fetching messages: 139869it [12:35, 175.62it/s]                                                                                                                
receiving content: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:14<00:00,  4.58s/it]
receiving metadata:  24%|█████████████████████▉                                                                       | 33005/139867 [39:50<1:00:45, 29.31it/s]Traceback (most recent call last):
  File "/usr/bin/gmi", line 4, in <module>
    __import__('pkg_resources').run_script('gmailieer==0.3', 'gmi')
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 742, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1510, in run_script
    exec(script_code, namespace, namespace)
  File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/EGG-INFO/scripts/gmi", line 8, in <module>
  File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/gmailieer.py", line 136, in main
  File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/gmailieer.py", line 307, in pull
  File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/gmailieer.py", line 531, in full_pull
  File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/gmailieer.py", line 562, in get_meta
  File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/remote.py", line 106, in func_wrap
  File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/remote.py", line 277, in get_messages
  File "/usr/lib/python3.6/site-packages/oauth2client-4.1.2-py3.6.egg/oauth2client/_helpers.py", line 133, in positional_wrapper
  File "/usr/lib/python3.6/site-packages/google_api_python_client-1.6.3-py3.6.egg/googleapiclient/http.py", line 1464, in execute
  File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/remote.py", line 257, in _cb
  File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/gmailieer.py", line 560, in _got_msg
  File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/local.py", line 333, in update_tags
lieer.local.RepositoryException: tried to update tags on non-existant file: /data/Mail/personal/mail/cur/132794ce91c4d140:2,F
gauteh commented 6 years ago

Manish R Jain writes on september 15, 2017 9:27:

Not using mutt, or offlineimap or anything else. I pulled in all the changes from master. Still the same issue.

42 is not master.

gauteh commented 6 years ago

Also check if that message exists, but with different maildir flags. Or if it is in 'new'.

gauteh commented 6 years ago

Do these messages exist in more than one gmailieer synced account?

manishrjain commented 6 years ago

Okay. Switched over to 

commit b74da5ff8ab2e69cfbb4977b68a964e11ce3bbb3 (HEAD -> also-new, origin/also-new)
Author: Gaute Hope <eg@gaute.vetsj.com>
Date:   Fri Sep 15 09:59:40 2017 +0200

    get_content(): compact list filter

commit fff8484f41cc90bff0559f121fa68d14a3c6d482
Author: Gaute Hope <eg@gaute.vetsj.com>
Date:   Fri Sep 15 09:48:36 2017 +0200

    local: add newly added messages to cache

commit 6d9cc523f93054ee9d151f77c05329873e37528e
Author: Gaute Hope <eg@gaute.vetsj.com>
Date:   Tue Sep 12 10:42:57 2017 +0200

    local: include messages stored in new in cache as well

commit 8ae3bc794ea8ca682b36bf7275a60e2b9e93f52f (origin/master, origin/HEAD, master)
Author: Gaute Hope <eg@gaute.vetsj.com>
Date:   Wed Sep 13 08:18:08 2017 +0200

    remote: reduce default batch size to 50 (as encouraged by google) to avoid throttling.

    https://github.com/gauteh/gmailieer/issues/40#issuecomment-329023481
    https://developers.google.com/gmail/api/guides/batch
    https://developers.google.com/gmail/api/v1/reference/quota

Installed gmi again. Still an issue.

$ gmi pull                                                                                                                                 /data/Mail/personal
pull: full synchronization (no previous synchronization state)
fetching messages: 139873it [12:47, 166.96it/s]                                                                                                                
receiving content: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:25<00:00,  6.06s/it]
receiving metadata:  24%|███████████████████████▏                                                                       | 34177/139869 [13:27<54:57, 32.05it/s]Traceback (most recent call last):
  File "/usr/bin/gmi", line 4, in <module>
    __import__('pkg_resources').run_script('gmailieer==0.3', 'gmi')
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 742, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1510, in run_script
    exec(script_code, namespace, namespace)
  File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/EGG-INFO/scripts/gmi", line 8, in <module>
  File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/gmailieer.py", line 136, in main
  File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/gmailieer.py", line 307, in pull
  File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/gmailieer.py", line 531, in full_pull
  File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/gmailieer.py", line 562, in get_meta
  File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/remote.py", line 106, in func_wrap
  File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/remote.py", line 277, in get_messages
  File "/usr/lib/python3.6/site-packages/oauth2client-4.1.2-py3.6.egg/oauth2client/_helpers.py", line 133, in positional_wrapper
  File "/usr/lib/python3.6/site-packages/google_api_python_client-1.6.3-py3.6.egg/googleapiclient/http.py", line 1464, in execute
  File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/remote.py", line 257, in _cb
  File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/gmailieer.py", line 560, in _got_msg
  File "/usr/lib/python3.6/site-packages/gmailieer-0.3-py3.6.egg/lieer/local.py", line 345, in update_tags
lieer.local.RepositoryException: tried to update tags on non-existant file: /data/Mail/personal/mail/cur/11518f6bedb78e9a:2,

The file is in cur as you can see. Doing an ls shows this:

$ ls /data/Mail/personal/mail/cur/11518f6bedb78e9a*                                                                                        /data/Mail/personal
/data/Mail/personal/mail/cur/11518f6bedb78e9a:2,S

My work mail doesn't have this file.

gauteh commented 6 years ago

So these are all different files, could you check with notmuch command line if this message has the same id as any other message-files? Perhaps, you have sent this message and received another copy of it using a mailing list or something.

gauteh commented 6 years ago

Gaute Hope writes on september 15, 2017 10:53:

So these are all different files, could you check with notmuch command line if this message has the same id as any other message-files? Perhaps, you have sent this message and received another copy of it using a mailing list or something.

Try #44 - here all messages files from each message are added, #42. has been merged.

manishrjain commented 6 years ago

Ran it. Thought I'll quickly test it and let you know, but after receiving content for 13mins, looks like it's receiving metadata, which would take more time. Heading home from office. I'll know on Monday.

gauteh commented 6 years ago

Manish R Jain writes on september 15, 2017 11:50:

Ran it. Thought I'll quickly test it and let you know, but after receiving content for 13mins, looks like it's receiving metadata, which would take more time. Heading home from office. I'll know on Monday.

Ok, good luck

gauteh commented 6 years ago

Any news on this?