Optimize all_mail_query

[nickwynja]

I currently have ~70k messages in my index which is... notmuch. 🥁

Each time I run mujmap it runs the format!("path:\"{}/**\"", relative_mail_dir.to_str().unwrap()) query, which takes 6+ seconds for me.

NOTMUCH_PROFILE=personal notmuch search path:"jmap/**"  6.19s user 2.07s system 98% cpu 8.355 total

With my basic understanding of how mujmap works, I don't see a straightforward way to use a different query or optimize this query so this can be a place to discuss.

[nickwynja]

Looks like lieer always uses lastmod:

qry = "path:%s/** and lastmod:%d..%d" % (self.local.nm_relative, self.local.state.lastmod, rev)

And I can see we use that when merging changes to local messages.

[nickwynja]

Would it be possible to defer the heavy all_mail_query until we need to do a full sync? And then just query notmuch directly per message instead of local_emails.get(&remote_email.id) and local_emails.contains_key(&x.remote_email.id)?

[elizagamedev]

That makes sense to me. I probably should have anticipated that this would not scale well.

[elizagamedev]

Would you like to take a shot at the implementation for this fix, if you have the time/rust knowledge?

[nickwynja]

I don't have much rust experience (or with any other strongly typed language) but I'll give it a shot! Looks like most of this logic is before anything destructive happens so I should be able to try it using --dry-run without blowing anything up.

[elizagamedev]

Awesome! Please don't feel too much pressure to get it done just by yourself. If you run in to trouble, let me know. 🙂

[nickwynja]

I've been trying to make progress here but have gotten stuck pretty early. I tried to reuse the get_message function but it returns a message type rather than the needed email type. I also tried creating a new query function that returns an email but continue to be confounded by types. Don't think I'll be able to get much done here, unfortunately.

On 06/06/22 at 16:26, Eliza @.***> wrote:

Awesome! Please don't feel too much pressure to get it done just by yourself. If you run in to trouble, let me know. 🙂

-- Reply to this email directly or view it on GitHub: https://github.com/elizagamedev/mujmap/issues/31#issuecomment-1147887067 You are receiving this because you authored the thread.

Message ID: @.***>

[elizagamedev]

I understand. I will add this to the queue for v0.2.1, and once it's implemented, I'll ping you if you'd like to take a look at the code. 🙂