Refactor mailbundle for an improved cli experience

[leophys]

I am opening this issue to propose an overhaul of the shape of this tool. If I get it right, the initial tenets of this project revolved around the unix philosophy[^1] and portability. And also, do not reinvent the wheel, but I give it for granted. In my understanding, from that derived the current status: many small programs, assembled to work together to orchestrate a modern email experience. Moreover, when possible, only shell/bash scripts or python scripts have been used (apart from the core dependencies of a MTA, MUA and MDA), and the python runtime should have been enclosed in a portable virtualenv, so that all that one would have needed was to copy the directory containing all this from one computer to another to use it.

I am fine with the unix philosophy, and also the core concept of portability is nice. What irks me a bit is the bootstrap experience and the overall cli interaction, that is a bit poor. I would like to make some proposals that, in my opinion, could improve the usage of mailbundle.

The cli

I propose to refactor the existing code in a python package that, once pip-installed, offers a cli: mailbundle. Such cli will have a number of subcommands, and its configuration file will be mailbundle.json (the one currently produced is good enough). This cli will have the following subcommands:

• start: to spawn the various processes (as now, inside a tmux session)
• pack: to create a tarball with the configurations and the mailboxes
• unpack: the opposite of the previous, but invoking implicitly the following
• bootstrap: takes the configurations (those stored in src/vars) and generates the runtime (config/ and everything in dep/)

The details

Two things have to be preserved: the structure in small programs and the possibility for the user to override the default scripts and templates. To do so, the configuration (src/) may include also override/{config,templates} paths that will be considered with higher priority when bootstrapping.

The mailbundle cli should look for a mailbundle.json file (to look for example for the base path of execution) in a deterministic set of paths (I am thinking to the usual $PWD -> $XDG_CONFIG -> $HOME -> /etc hierarchy).

My favorite cli library is click, but of course I am open for discussion.

All the currently used templates and static assets should be included in the package statically.

[^1]: Write programs that do one thing and do it well; write programs to work together; write programs to handle text streams, because that is a universal interface.

[boyska]

I propose to refactor the existing code in a python package that, once pip-installed, offers a cli: mailbundle. Such cli will have a number of subcommands, and its configuration file will be mailbundle.json (the one currently produced is good enough). This cli will have the following subcommands:

cool. I propose to distinguish between the current mailbundle.json and a mailbundler.json, which configures the tool itself (ie: verbosity, tunables, default format for packing, etc.)

• start: to spawn the various processes (as now, inside a tmux session)
• pack: to create a tarball with the configurations and the mailboxes
• unpack: the opposite of the but invoking implicitly the following
• bootstrap: takes the configurations (those stored in src/vars) and generates the runtime (config/ and everithing in dep/)

nice interface

The details

Two things have to be preserved: the structure in small programs and the possibility for the user to override the default scripts and templates. To do so, the configuration (src/) may include also override/{config,templates} paths that will be considered with higher priority when bootstrapping.

yes, improving the template structure is very much a good thing to do. Right now, it sucks.

In theory, that's orthogonal to the other point, but let's do a major overhaul of it all.

The mailbundle cli should look for a mailbundle.json file (to look for example for the base path of execution) in a deterministic set of paths (I am thinking to the usual $PWD -> $XDG_CONFIG -> $HOME -> /etc hierarchy).

yes

My favorite cli library is click, but of course I am open for discussion.

I tent to dislike additional dependencies that are not strictly needed. However, click is very widely used. I hope we can be compatible with old versions of it. But really: we all want autocompletion.

All the currently used templates and static assets should be included in the package statically.

nice!

So: feel free to start a "v2" branch, where we can go wild about it. happy hacking!

[leophys]

Writing here what we discussed OTR. The following points stand out clear:

mailbundler

The cli will be named mailbundler, and its configuration file will be mailbundler.json. It will be sought by the cli in the following order:

• $XDG_CONFIG_HOME/mailbundler.json
• $XDG_CONFIG_HOME/mailbundler/mailbundler.json
• $HOME/.config/mailbundler.json
• $HOME/.config/mailbundler/mailbundler.json
• /etc/mailbundler.json This will be overridden manually specifying a path with the cli --config switch (the switch is global, i.e. mailbundler --config <path> [subcommand] ...).

The usage flow

There are various steps in the usage of mailbundler. A first-time user installs mailbundler (together with its dependencies) and creates an empty bundle (this name is user-controlled and bundle is used here just as example) with the new subcommand. It will contain the following structure:

bundle
├── environment
├── settings
│   ├── overrides
│   │   ├── static
│   │   └── templates
│   └── vars
└── var
    ├── lib
    └── mail

The user then places their configurations in settings/vars and their (optionally) overriding builtin templates with files placed in settings/overrides/templates and static assets with files placed in settings/overrides/static.

After that, and any time they change anything in settings/, the user may then derive a new environment with the apply subcommand. This will create all the needed machinery in the bundle/environment subfolder.

The whole set of programs may then be invoked with mailbundler start (the equivalent of the current ./config/bin/autorun).

If a user wants to move the bundle on another machine, they may do so with mailbundler pack. This will backup in a tarball the whole bundle/settings/ and bundle/var. Optionally, with a --light flag, only bundle/settings may be backed up. On the other machine, having installed there mailbundler and copied the tarball, the user may invoke then mailbundler unpack. This will extract the content of the tarball and place it where chosen and then implicitly invoke apply to rebuild the new environment/.

The folder structure

The above structure has the following logic:

• settings/ contains all the configurations needed to derive a fully working environment
• var/ contains all the state: var/mail/ contains the mailboxes, var/lib/ the possible runtime status (like the mutt alias file, the notmuch database, ...)
• environment/ holds all the runtime information, i.e. everything that is needed to invoke the programs that mailbundler depends on, but that is completely derivable from the settings/

mailbundler.json

The scope of this config file is to hold the information regarding one or more bundles. It is possible indeed to create on the same machine more than one bundle (think of example of a personal and a work bundle) and to use them separately, as different profiles passed to mailbundler. It is mailbundler responsibility to update this mailbundler.json file anytime new is invoked. The structure is the following, as of now (any further information useful for the execution of mailbundler will be placed here):

{
    "default": "main_account",
    "main_account": {
        "path": "/path/to/bundle"
    },
    "work_account": {
        "path": "/other/path"
    }
}

subcommands

The following subcommands will exist.

new

mailbundler [--config <path/to/config>] new [--profile <profile_name>] [--default] <path/to/bundle>

This will create the above mentioned folder structure and add the chosen path to the mailbundler.json config file. If --default is used, this bundle will be set as default profile. If no --profile is provided, the profile name will be the last component of the given path (i.e. the directory basename).

apply

mailbundler [--config <path/to/config>] apply [--profile <profile_name>]

This will create/update the contents of the environment/ subfolder. If no --profile is specified, the default one is used.

start

mailbundler [--config <path/to/config>] start [--profile <profile_name>]

This will start the screen/tmux session. If no --profile is specified, the default one is used.

pack

mailbundler [--config <path/to/config>] pack [--profile <profile_name>] [--light] [--all]

This will pack in a tarball the bundle. If --light is specified, the contents of var/ will not be packed. If no --profile is specified, the default one is used. If --all is specified, all the profiles in mailbundler.json will be tarballed in separate tarballs (--profile will be ignored and --light, if present, will be applied to all the packs).

unpack

mailbundler [--config <path/to/config>] unpack [--profile <profile_name>] <path/to/tarball> <path/to/new/bundle>

This will unpack the tarball in the given destination. The destination may not exist and in that case it will be created. The profile will be added to mailbundler.json.

[leophys]

After another round of useful discussion, we decided to drop the idea of profiles, and steer towards a git-like approach: mailbundler will behave differently whether in a bundle or not and will look for an env variable that sets a default bundle (MAILBUNDE_DEFAULT_DIR). So, for example, if I have two distinct bundles ~/workmail and ~/mail, I can:

• cd to either one of them and launch from there mailbundler (like git, this shall work in any subdirectory of the bundle)
• specify the path with a flag --bundle-dir|-d: mailbundler start -d ~/mail
• set an env variable (MAILBUNDLER_DEFAULT_DIR=~/mail) and use mailbundler start (or any other subcommand) straight away (of course, setting --bundle-dir will have higher priority)

The mailbundler.json file will still exist, but will hold only configurations used by the cli itself to run (for example the verbosity).