Consistent KeyError in update_tags: self.gids[gid] upon initial sync

[rbutoi]

When I run an initial sync, I get this error (with or without dry-run), where xxx is the same ID each time:

$ gmi sync -c ~/client_secret.json -C ~/.mail/work/ -d
path: .mail/work/
dry-run:  True
push: everything is up-to-date.
pull: full synchronization (no previous synchronization state)
fetching messages: 48414it [01:00, 794.75it/s]
receiving content:   0%|                                                                                                                                                    | 1/48414 [00:00<9:29:00,  1.42it/s$
missing file: reloading cache to check for changes..Traceback (most recent call last):
  File "[...]/bin/gmi", line 8, in <module>
    g.main ()
  File "[...]/bin/lieer/lieer/gmailieer.py", line 172, in main
    args.func (args)
  File "[...]/bin/lieer/lieer/gmailieer.py", line 254, in sync
    self.pull (args, True)
  File "[...]/bin/lieer/lieer/gmailieer.py", line 372, in pull
    self.full_pull ()
  File "[...]/bin/lieer/lieer/gmailieer.py", line 595, in full_pull
    updated = self.get_content (message_gids)
  File "[...]/bin/lieer/lieer/gmailieer.py", line 661, in get_content
    self.remote.get_messages (need_content, _got_msgs, 'raw')
  File "[...]/bin/lieer/lieer/remote.py", line 116, in func_wrap
    return func (self, *args, **kwargs)
  File "[...]/bin/lieer/lieer/remote.py", line 352, in get_messages
    cb (msg_batch)
  File "[...]/bin/lieer/lieer/gmailieer.py", line 659, in _got_msgs
    self.local.store (m, db)
  File "[...]/bin/lieer/lieer/local.py", line 497, in store
    self.update_tags (m, p, db)
  File "[...]/bin/lieer/lieer/local.py", line 542, in update_tags
    fname = os.path.join (self.md, self.gids[gid])
KeyError: 'xxx'

Prior to this I had successfully synced (with some errors along the way that went away after re-runs). Now, I'm trying to sync from scratch, and this occurs consistently. From scratch here means, remove all but the two .gmailieer.json files in the mail directory and rm -rf ~/.mail/.notmuch && notmuch new. Any idea what is going wrong here?

BTW, when I did see it sync, it seemed to be much faster than mbsync, which is awesome.

Thank you!

[gauteh]

Hi, best way to start from scratch is to do gmi pull --force. You might have to add a flag if you want local message that no longer exist on remote to be deleted. This will reset the tags.

[gauteh]

Just to be sure, could you try to run the command from the repository directory without the -C argument.

[rbutoi]

Hi, best way to start from scratch is to do gmi pull --force. You might have to add a flag if you want local message that no longer exist on remote to be deleted. This will reset the tags.

Just to be sure, could you try to run the command from the repository directory without the -C argument.

Having tried both of these suggestions, I get the same errors. Some more data:

• Occasionally, the errors will be a `FileNotFoundError: '~/.mail/personal/mail/tmp/xxx:2,' with the same xxx.
• I can reproduce this on both my home and work Linux machines, on both home/work accounts.

I'll try reproducing on my Mac laptop too.

[gauteh]

Radu Butoi writes on April 14, 2020 20:20:

Hi, best way to start from scratch is to do gmi pull --force. You might have to add a flag if you want local message that no longer exist on remote to be deleted. This will reset the tags.

Just to be sure, could you try to run the command from the repository directory without the -C argument.

Having tried both of these suggestions, I get the same errors. Some more data:

• Occasionally, the errors will be a `FileNotFoundError: '~/.mail/personal/mail/tmp/xxx:2,' with the same xxx.
• I can reproduce this on both my home and work Linux machines, on both home/work accounts.

Do you modify the database in any way while syncing? Or mark messages as read (which would change the read-flags and the file names).

[rbutoi]

Do you modify the database in any way while syncing? Or mark messages as read (which would change the read-flags and the file names).

Not that I know of -- the only thing that might be accessing the notmuch database is notmuch.el, and I tried re-syncing without emacs running but same error. Would there be anything else accessing this DB (I'm new to notmuch as well)?

Wondering, does lieer store/read anything besides in the mail directory and .notmuch directory?

[rbutoi]

Happens on Mac as well.

[gauteh]

I see you tried with and without dry-run, did you try without dry-run on your other tests? It seems from the code that it might work worse in dry-run. Could you check if the file xxx... exists in either cur, tmp or new but with with some letters after the :?

[gauteh]

Do you have a backtrace for the FileNotFoundError? This would only happen at line 494 in local.py as far as I can see. Which is weird, because the file has been written just before... could there be some file system issue? Is this a remote mounted filesystem or is the disk full or something?

The key error with xxx.., what is the full key name except the gid?

[rbutoi]

OK I found my issue. When setting up the lieer mail folder, I would interrupt the process to change the .gmailieer.json file, and end up trying to gmi sync and get a lieer.local.RepositoryException: local repository not initialized: could not find mail dir. To "fix" that, I mkdir a mail directory but no cur, new, tmp below it. This was the problem: after I made these subdirectories pull --force works. It's interesting that you say the file would have been just written before, since that write should fail first?

We could re-purpose this issue to make this flow more robust and prevent other users from running into this problem. I could send you a patch to fix this if you'd like (basically, when doing the lieer.local.RepositoryException check, check the subdirs as well).

Regarding dry run mode, I get the same KeyError as in my first comment even after everything is properly synced.

[gauteh]

Radu Butoi writes on April 16, 2020 1:48:

OK I found my issue. When setting up the lieer mail folder, I would interrupt the process to change the .gmailieer.json file, and end up trying to gmi sync and get a lieer.local.RepositoryException: local repository not initialized: could not find mail dir. To "fix" that, I mkdir a mail directory but no cur, new, tmp below it. This was the problem: after I made these subdirectories pull --force works. It's interesting that you say the file would have been just written before, since that write should fail first?

Yes, maybe the tmp dir is created? But the rename fails because cur is missing? All the mail files have to be in the cur sub-folder. I thought you got the mail files from a previous gmi sync. Then they should have the correct name (the gid) and be in the correct sub-directory.

We could re-purpose this issue to make this flow more robust and prevent other users from running into this problem. I could send you a patch to fix this if you'd like (basically, when doing the lieer.local.RepositoryException check, check the subdirs as well).

Great. That sounds good. I was planning to do a release when your issue popped up, would be good to get your fix included.

Regarding dry run mode, I get the same KeyError as in my first comment even after everything is properly synced.

Was this fixed by creating the dirs? Would this be fixed by your patch?

[rbutoi]

Yes, maybe the tmp dir is created? But the rename fails because cur is missing? All the mail files have to be in the cur sub-folder. I thought you got the mail files from a previous gmi sync. Then they should have the correct name (the gid) and be in the correct sub-directory.

From what I can tell, the only error state I've ran into is a mail/ dir without cur,tmp,new below it. I found it odd that the program would run for about a minute (I guess fetching from Gmail without needing those subdirs, haven't read that part of the code yet) before failing.

Was this fixed by creating the dirs? Would this be fixed by your patch?

No, even after a full proper sync, -d gives me the same error. My patch would just change local.py:243 to check for the subdirs as well, so it wouldn't make a difference there. Does -d work for you?

[gauteh]

Radu Butoi writes on April 16, 2020 19:15:

Yes, maybe the tmp dir is created? But the rename fails because cur is missing? All the mail files have to be in the cur sub-folder. I thought you got the mail files from a previous gmi sync. Then they should have the correct name (the gid) and be in the correct sub-directory.

From what I can tell, the only error state I've ran into is a mail/ dir without cur,tmp,new below it. I found it odd that the program would run for about a minute (I guess fetching from Gmail without needing those subdirs, haven't read that part of the code yet) before failing.

Right. This should be fixed by your patch, which will cause an early return or solve it in some other way.

Was this fixed by creating the dirs? Would this be fixed by your patch?

No, even after a full proper sync, -d gives me the same error. My patch would just change local.py:243 to check for the subdirs as well, so it wouldn't make a difference there. Does -d work for you?

Yes, usually. But with a full proper sync there should not be any changes to update on any messages, nor any files missing, so I don't understand how that happens.

[rbutoi]

Sent https://github.com/gauteh/lieer/pull/135.

Yes, usually. But with a full proper sync there should not be any changes to update on any messages, nor any files missing, so I don't understand how that happens.

Odd. I may look into my dry run problem more later, but I'm fine for now if it syncs without it.