search

johnnyutahh / search-based-email

Documenting one user's conversion to Notmuch/mu from Thunderbird
44 stars 10 forks source link

readme

// vim: set syntax=asciidoc:

// set asciidoc attributes :toc: macro :numbered: 1 :data-uri: 1 :icons: 1 :sectids: 1 :iconsdir: /usr/local/etc/asciidoc/images/icons

// create blank lines, from: http://bit.ly/1PeszRa :blank: pass:[ +]

:sectlinks: 1 //:sectanchors: 1

Converting to Search-Based Email: http://notmuchmail.org/[Notmuch] or http://www.djcbsoftware.nl/code/mu/[mu]

author github.com@johnnyutahh.com 0.1, June 2015: Last updated {docdate} {doctime}

toc::[]

<<< :numbered:

<<<

[id='the_main_questions']

Questions

Here's the most-important questions, given <<the_purpose,the purpose>> and <<the_problem,the problem>>:

. Tools. Am I accurately assessing all available <<tool_choices,tools>>?

. User scenarios. What are the best tools, or trade-offs of each tool, for each <<user_adoption_challenges,user scenario>>?

. Workflows. Do you know a better workflow than what's presented for any <<tool_choices,tool>> or <<user_adoption_challenges,usage scenario>>?

[id='the_purpose']

Purpose

Summary

<<my_user_profile,I>> want to transition my email environment from a <<my_existing_environment,classic setup>> to a tag-and-index-based system (most likely <<core_choice,Notmuch- or mu-based>>), and I'm seeking help.

. Near-term. I'm seeking significant feedback on this document from the <<core_choice,Notmuch, mu, and related>> user communities. Most importantly, I seek answers to <<the_main_questions,these questions>>.

. Longer-term. This document might be reusable by others, including future new users, if someone matures it moving forward. (In such case, the tone would change from being centered on <<my_user_profile,my>> problems to a more-general-purpose document.) <<existing_tag_based_user_intro_docs_appear_to_be_lacking,I couldn't find similar content>> on the web. Please advise if I missed something.

With the right system, I'm hopeful I can make email management a much-less-time-consuming burden on my work... and my life.

Details

DISCLAIMER: this document may be in flux as I learn and attempt to master this domain.

Why write this doc instead of just trudging through the tool testing myself? <<why_spend_the_effort_to_write_this_doc,Read on>>.

See <> for more.

[id='the_problem']

The Problem

I find email classification, recall, and translation to work/task tracking to be much more laborious than it should be. <<computer_please_be_faster_than_me,I expect the computer to be faster than me>>, and tailor to my preferred workflow. Instead, I'm constantly waiting for the computer, or forced to type or mouse-click inefficiently (I'm a <<my_user_profile,vim-loving keyboard jockey>>), and/or experiencing a poor workflow.

More specifically:

** Among other things, <<old_style_imap_folder_paradigm,moving and copying emails to a variety of IMAP folders>>--in order to "classify" said emails--gets cumbersome.

** Searching my <<my_existing_environment,all my IMAP folders>> (many of them quite large) with an MUA like https://www.mozilla.org/en-US/thunderbird[Thunderbird] takes impossibly too long, and in many cases, isn't powerful or granular enough to get the job done.

** Lack of support for several of these <<user_adoption_challenges,user scenarios>> makes general email management much more cumbersome.

** https://www.mozilla.org/en-US/thunderbird[Thunderbird (TB)] in general is <<problem_details,slow, sluggish, and generally annoying in many ways>> in part because it's a GUI-only interface, but TB annoys beyond just the GUI-only problem--including it's struggles to handle a <<my_existingenvironment,large number of IMAP folders>>. Other MUA's like http://bit.ly/1Gn1Ant[Outlook], https://www.postbox-inc.com/[Postbox] (a TB variant), http://en.wikipedia.org/wiki/Mail(OSX)[Mail.app], and http://en.wikipedia.org/wiki/Evolution(software)[Evolution] are either worse (than TB, for <<my_user_profile,me>>) or unavailable in <<my_existing_environment,my environment>>.

[id='my_existing_environment']

See <> and <<user_adoption_challenges,user scenarios>> for more context and requirements.

(An aside: why title this document "search-based email"? http://notmuchmail.org/pipermail/notmuch/2015/020617.html[Suggestion from the community]. It seems better than "tag-based email," my original title. I've also considered "index-based email," but "search" seems a bit more descriptive than "index," and less esoteric. I'm open to any title that best groks with the community.)

Possible Solutions

After reviewing currently-available tools, I'm seeking:

. fast search on my entire email-message/IMAP collection (my "email database"), . to leverage search/index/tag-based email categorization, and . provide easy extensibility with Linux and MacOSX command-line "primitives" or other, custom scripts/commands.

It's fair to ask: did I come up with the above "requirements" by a) adjusting to features from currently-available tools, or b) by independently dreaming up my own specifications? More the former/(a). I'd much prefer http://bit.ly/JARVIS-wikia[Jarvis] be my email interface, but he's not currently (or at least not economically) available.

[id='tool_choices']

Tool Choices

[id='core_choice']

Core tool choice - Indexing

My investigation thus far suggests the implementation path hinges on choosing 1 of the following 2 applications, as they seem to mutually-exclusively represent the best (or at least most-popular) of the core of email-message indexing and tagging tool suites:

  1. http://notmuchmail.org[Notmuch]
  2. http://www.djcbsoftware.nl/code/mu[mu (maildir-utils)]

Is this assessment accurate? What other tools/options might I be overlooking?

[id='notmuch_won'] My comparison analysis:

  1. Initial tests show https://gist.github.com/johnnyutahh/f4e3d2d3fb07de5fa146[Notmuch performing approximately 15 times faster than mu].

    ** Question: were these tests configured and executed correctly? The performance difference is remarkable, generating concerns about correct application setup, environment.

    ** Performance optimizations:

    https://github.com/djcb/mu/issues/22[mu's '.noupdate' feature] http://notmuchmail.org/pipermail/notmuch/2015/020763.html[Notmuch's 'new.ignore' feature, similar to mu's .noupdate] https://groups.google.com/d/msg/mu-discuss/Lgaf4anZjxc/ahqOBAH3V6kJ[mu performance discussion] As of 2015-07-14, I've not yet retested with above optimizations.

  2. mu can embed its metadata (tags, etc) "natively" into the IMAP content/messages. Notmuch can not. However, http://www.muchsync.org/[muchsync] (maybe other tools?) can replicate this metadata, but it takes additional process+infrastructure.

  3. 1 greatly outweighs #2. Because of this, Notmuch "wins" (with me),

    pending the retest results of #1 (above) and additional feedback from others.

What other trade-offs might motivate me to employ http://www.djcbsoftware.nl/code/mu[mu] over http://notmuchmail.org[Notmuch]?

Since Notmuch <<notmuch_won,won>> (for now), the rest of this document may be more http://notmuchmail.org[Notmuch]-specific.

IMAP-to-Maildir syncing

Notmuch seems to work best (or maybe requires?) the http://en.wikipedia.org/wiki/Maildir[Maildir] format. The following tools (presumably) all sync an http://en.wikipedia.org/wiki/Internet_Message_Access_Protocol[IMAP] server to a Maildir filesystem.

Options

Choice

I've currently chosen http://isync.sourceforge.net/[mbsync, aka isync].

Comments

Notmuch initial + automated tagging

(I've not yet started this implementation.)

Auto-detect if IMAP resync needed

(I've not yet started this implementation.)

client->server checking

server->client checking

[id='MUA_choice']

MUA

Summary

Given <<the_problem,the problem>>, the work to find and master the best/better MUA(s) for <<my_user_profile,me>> and concurrently learn a new search-and-tag-based email classification paradigm seems like my biggest challenge.

http://kzak.redcrew.org/doku.php?id=mutt:start[mutt-kz] and https://github.com/pazz/alot[alot] currently present the most-attractive solutions (for me), but it's early.

A dark-horse candidate: http://notmuchmail.org/emacstips[notmuch.el], an Emacs front-end.

Details

** https://github.com/pazz/alot[alot] looks tremendously promising, possibly my best long-term solution, especially given <<my_user_profile,my user profile>> (namely I'm a vim user and a Python programmer--seems to mirror well). However, the available documentation/resources are far more sparse than say mutt-kz. The user-manual content is almost impeccable, and pazz seems to do a great job to stay on top of all issues and offer a professional solution. For example, I significantly appareciate the up-front, informationally-dense, bulleted feature list at the top of the https://github.com/pazz/alot/blob/master/README.md[alot README].

** However, it's thus far been hard to find practical resources like example config files, procedural setup, etc. Maybe this is due in part because it's not yet as popular, or caters to a user base more willing to spend time learning/configuring/tinkering with one tool, or something else?

*** Speculating: a hopefully-small effort to provide setup + config-file examples might go a long way to solve this problem, and boost alot's "new user uptake" populartiy.

*** https://github.com/pazz/alot/issues/776[Related discussion at alot's github project page]

*** https://github.com/pazz/configs/blob/master/.config/alot/config[pazz's alot configuration-file example]

*** https://github.com/johnnyutahh/search-based-email/issues/1[fowlslegs' comparison of alot and mutt-kz]

** Since <<my_user_profile,I'm a vim user>>, I've initially shied away an http://notmuchmail.org/emacstips[Emacs-based front-end]. However, I've now seen it referenced multiple times as a potentially-uniquely-powerful interface.

** Questions: Does http://notmuchmail.org/emacstips[notmuch.el] provide enough uniquely-powerful capability to merit special consideration? And if so, as a vim user will I find special difficulty trying to learn an Emacs-based paradigm?

** <<my_user_profile,I'm a heavy vim user>>, and while this approached seemed initially appealing, it's feature depth seems small enough that I haven't yet attempted to run this application--but to be fair, I haven't "dug deep." Am I overlooking a powerful (in comparison to the others) tool by not vetting this http://git.notmuchmail.org/git/notmuch/blob/HEAD:/vim/README[vim front-end] further?

** ...but none seem as appealing to <<my_user_profile,me>> as the above. Am I overlooking any solutions that might fit well with my <<my_user_profile,user profile>>?

[id='sync_notmuch_metadata_across_machines']

Synchronizing Notmuch metadata across machines

(I've not yet started this implementation.)

** Problem solved: the muchsync author ported to Mac OS 10.9.5 (at least) as of 2015-08-16 although this port may not yet be fully, empirically tested. https://github.com/johnnyutahh/search-based-email/issues/2[Details here].

[id='user_adoption_challenges']

Usage-Scenario Challenges in Priority-First Order

<<my_user_profile,My>> usage-scenario challenges include but may not be limited to:

[id='which_MUAs']

Which MUA(s)?

Decide which MUA(s) to use, particularly deciding on a primary MUA. This is technically not a usage-scenario, but currently represents my biggest challenge. See the <<MUA_choice,MUA options>>.

[id='old_style_imap_folder_paradigm']

Support old-style IMAP-folder paradigm

While I may be be moving to a search/index/tag-based paradigm, I still need to access my <<my_existing_environment,4k+ IMAP folders>> as I did before, at least while I'm transitioning my current <<folder_based_searching,folder-based>> paradigm that I currently employ with <<the_problem,Mozilla Thunderbird (TB)>>, which leverages the https://addons.mozilla.org/en-us/thunderbird/addon/nostalgy[TB's Nostalgy add-on] to do it.

TB-Nostalgy also offers excellent keyboard-shortcut-mapping capability and is one of the few great features of Thunderbird that I'd like to replicate in my <<MUA_choice,new MUA>>.

[id='imap_account_separation']

Support IMAP-account separation

. I have multiple email accounts, which is not uncommon. I want to "view" each one differently, such that emails and folders from account X does not clutter my view of emails/folders when viewing account Y.

. It would be extremely helpful to additionally support a "combined" view/mode of all my accounts. But this is not an absolute requirement, simply because #1 is currently more important than #2.

[id='initial_tagging']

Initial tagging

Moving msgs after applying tags

** I'm used to moving messages between folders in order to classify. Further, I will like to keep a clean Inbox and other folders, for my non-Notmuch-based email clients, thus (presumably) requiring message moving.

** Once I associate Notmuch-metadata (by adding tags, or whatever metadata/etc scenarios might be involved with Notmuch) with a message, I need said metadata/tags/etc to associate with a message forever, regardless of wherever I put said message. Is this the way it works "out of the box" with Notmuch-based systems?

Viewing HTML-formatted messages

[id='folder_based_searching']

MUA folder-based searching with name auto-completion

Piping email-message content through shell commands

I want to http://en.wikipedia.org/wiki/Pipeline_%28Unix%29["pipe"] the content of:

. one email message, . many email messages (by selecting multiple emails at the same time), or . an entire IMAP folder of emails

to any command/script of my choosing.

Example, potential solutions, not yet tested:

Mapping keyboard shortcuts to any command

Example potential solutions, not yet tested:

[id='IMAP_folders_as_tags',reftext='IMAP folders as tags']

IMAP folders as tags

Also, see <>.

Can a Notmuch-based system do this?

Address-book/contacts integration

Highlight folders with unread or new emails

How to do this? Including support for virtual folders. https://bbs.archlinux.org/viewtopic.php?id=191340[Details here].

Sync Notmuch tags with maildir flags

Does anyone use https://github.com/spaetz/notmuchsync[notmuchsync], and does it work well?

Managing message attachments

[id='auto_tagging']

Automatically tagging new messages (after initial tagging)

Offline editing - postponing drafts

Writing HTML-formatted messages

[id='forwarding_msgs_w_attachments']

Forwarding messages with attachments

[id='purpose_more_details']

Appendix 1: Purpose: More Details

[id='my_user_profile']

My user profile

** I've significant experience with programming in a variety of programming languages and system-administering a variety of OSes including but not limited to: C, C++, Java, Ada, perl, Python; Windows, many commercial Unix-es, Linux, VMS, MacOSX. My favorite "Swiss army knife" language is Python. If I've time, I'm open to extending/fixing Python programs. I'd like to learn https://www.ruby-lang.org[Ruby] and https://golang.org[Go].

vim remains my primary editor (I hate moving my hand from the keyboard to the mouse or trackpad), Mac OS X is my primary computing machine, and I still significantly code in Python to solve "glueware" problems. I also still dabble in Linux (mostly Debian/Ubuntu) and MacOSX sysadmin.

[id='computer_please_be_faster_than_me']

Despite my history assimilating to new applications/environments, the search-and-tag-based-classification paradigm still seems significantly different and a bit daunting to this "old school IMAP-folder user", and may (or may not?) take some time to master. See <>. For example, opening https://github.com/pazz/alot[alot] for the first time and looking at a staggering 50k+ emails in my "inbox" can give someone pause; hopefully <> will take care of that.

[id='existing_tag_based_user_intro_docs_appear_to_be_lacking']

Additionally, the search-and-tag-based documentation resources--to describe new-user-paradigm shifts and present the most-popular toolsets--seem disjointed and/or non-existent. Hence some of the motivation to present this document.

[id='why_spend_the_effort_to_write_this_doc']

Why write this doc?

Why did I spend the time to write this document, instead of just trying all the tools?

. Email is too important not to "get it right." Or at least, email is too "frequent," probably my most-frequent life activity (very unfortunately).

. Brute-force "experience" may be too inefficient. I'd rather learn from others' experiences rather than inefficiently reply them all myself.

. This document may help future newbies. And possibly accelerate new-user population growth.

. Defining requirements up front: this usually works. Rarely have I regretted taking the time to well-define requirements (separate from design and/or solution) for any significant software or tool-adoption project.

. I might learn something I wouldn't have previously found. It's possible this document might attract enough attention for people to offer solutions (applications, workflows, or whatever) I might not have otherwise discovered.

. Breaking my production email "IMAP database" testing new apps would be very... bad. My businesses and projects rely on my email system to be top-notch solid. If my email gets corrupted, lost, etc - things go very bad, very fast. Especially if I'm unknowingly messing up my email. Hence, I'm rather cautious about correct implementation.

In any case, I'm hopeful that experienced and diverse feedback from the search/index/tag-based-email-using communities can help avoid these problems. At least, it seemed like the most-effective way, as the space <<existing_tag_based_user_intro_docs_appear_to_be_lacking,doesn't (yet) seem friendly to newbies>>.

[id='problem_details']

Appendix 2: More Problem Details

(DISCLAIMER: This section's under construction, and not complete.)

OS X is great, but TB is difficult. https://www.mozilla.org/en-US/thunderbird[Thunderbird] is old, buggy, troublesome, slow, basically inextensible (for me, anyway), and as I understand it, feature frozen. I'm tired of debating with the mozillaZine support team about TB's bugs and limitations. Among other things, it's IMAP sync is slow and unreliable. It literally (and unfortunately, inconsistently) deletes IMAP folders on it's own whim, asynchronously, sometimes when I least expect it. Sometimes it loses track of the folders it didn't delete, and simply creates new ones, bloating my mbox (TB only reliably supports mbox) files terribly over time.

Additionally, the TB text/formatting editor is legendarily bad/buggy. I'd desperately prefer to simply edit in vim, and edit rich/html text in markdown or asciidoc and convert to html with a rendering engine, and I suspect I could script-integrate such capability... if I had an MUA that could play nicely with external scripts.

Further, I'm a keyboard jockey--eg: vim lover--and Python programmer. I've maxed out TB's keyboard-shortcut-ness (eg: https://addons.mozilla.org/en-us/thunderbird/addon/nostalgy[TB's Nostalgy add-on]) best I can tell, and it's still limiting. I have external tools (some developed by me and/or my team) to parse and perform "magic" (like task-tracking and bug-report integration) on email folders and individual messages, and TB--with it's lack of proper maildir support and difficult extensibility--makes it extremely difficult if not impossible to integrate with the external tools.

In short, it's time to move on from https://www.mozilla.org/en-US/thunderbird[Thunderbird].

Appendix 3: General Setup Guides

(Previously-referenced guides or sections of guides listed elsewhere in this doc are not duplicated here. The following is provided here for my general reference; maybe others will find these references useful.)

More Notmuch stuff

mutt-kz-applicable configuration

generic-mutt stuff

mutt and Maildir

mbsync + mutt

From the mbsync(1) man page as of 2015-07:

Flatten delim

      Flatten the hierarchy within this Store by substituting
      the canonical hierarchy delimiter / with delim. This can
      be useful when the MUA used to access the Store provides
      suboptimal handling of hierarchical mailboxes, as is the
      case with Mutt. A common choice for the delimiter is ..
      Note that flattened sub-folders of the INBOX always end up
      under Path, including the "INBOXdelim" prefix.

Does this mean mutt does not work with Maildir hierarchical subfolders?

Misc

{blank} {blank} {blank} {blank} {blank} {blank} {blank} {blank} {blank} {blank} {blank} {blank} {blank} {blank} {blank} {blank} {blank} {blank} {blank} {blank} {blank} {blank} {blank} {blank} {blank} {blank} {blank} {blank}