BSD/Mac OS compatibility

[LukeSmithxyz]

I'm open to making this script compatible with BSD and Mac OS although I have neither OS so can't really be bothered to test it.

As shown in this post Mac OS can be done with a couple steps. I'll mention those relevant for making mw compatible here.

1. after cloning this repo, edit Makefile and change the PREFIX to
   `PREFIX = /usr/local`

Just add an if statement to the makefile to test the OS.

4. edit mw to change a few **hardcoded** paths
   `vim $(which mw)`
   find **muttshare** and set it to:
   `muttshare="/usr/local/share/mutt-wizard"`
   samething for mwconfig:
   `mwconfig="/usr/local/share/mutt-wizard/.muttrc"`
   4a. on of make install commands is:
   `cp -f mutt-wizard.muttrc $(DESTDIR)$(PREFIX)/share/mutt-wizard` which, on macos doesn't create ../share/mutt-wizard/.muttrc but ../share/mutt-wizard/mutt-wizard.muttrc so you have to rename it to /usr/local/share/mutt-wizard/.muttrc

Same principle, but for the script.

5. about certificates:
   on /etc/ssl there's a cert.pem file which should be fine. I had to symlink it to /etc/ssl/certs/ca-certificates.crt so:
   `sudo ln -s /etc/ssl/cert.pem /etc/ssl/certs/ca-certificates.crt`
   although I found out later you don't need this as SystemCertificate is already in use.

In a bug fix earlier today, mw now searches common certificate locations for the file, so all you'd need to do is add this location and make sure it works.

8. `mw add` and follow the program's instructions. it should fail with gmail
   though it was because of certificate and removed the CertificateFile entry on ~/.mbsyncrc.
   tried to setup an old Gmail account with no success at first. I have 2fa turned on and had to generate an "App password"
   as Luke mentioned. but also, had to through
   `AuthMechs LOGIN` in ~/.mbsyncrc for it to work
   you can also download any certificate for your connecting imaps server with
   `mbsync-get-cert imap.gmail.com` (change imap.gmail.com with yours of course).
   after **AuthMechs LOGIN** I could finally do:
   `mbsync -l gmail` with success (called my account "gmail" during mw setup)

Not sure what parts here are user error or just Gmail problems, but anything else can be fixed.

So if someone wants to open a PR adding tests in here to enable Mac OS compatibility or BSD, feel free to. Should be easy and I would do it myself, but I'm sure there's another detail that would necessitate actually having the OS! Be sure not to break Linux compatibility of course!

Actually, for all I know it's already compatible with BSD, but I haven't tried it. I'm just assuming that their certs are somewhere else or something.

[mikeziri]

will give it a try to open PR later. thanks!

[mikeziri]

at this moment I'm not sure if it isn't better to just fork bin/mw to bin/macos/mw.

paths are somewhat simple to fix but there are things like grep that is slightly different between BSD versionn (macos) and GNU's.

one can add dependency of gnugrep to the list though.

I say this because mw add gives me a bunch of errors on grep commands in the end. but it does most of the work anyways so no big deal.

[mikeziri]

bin/openfile should be considered as well to fix mktemp and xdg-open with macos' open or simple use open in mailcap

[LukeSmithxyz]

No, that's ridiculous. Just replace the grep commands with something posix/portable.

I assume you're talking about lines like this:

echo "$boxes" | grep -i -m 1 inbox | formatShortcut i inbox

As I think -m is GNU specific. Just do:

echo "$boxes" | grep -i inbox | head -n 1 | formatShortcut i inbox

etc.

As for xdg-open, use an if statement. No reason to double up scripts.

[mikeziri]

@LukeSmithxyz are you using spaces or tabs? are there any guidelines?

[LukeSmithxyz]

Spaces are for separating words; tabs are for indenting text.

[mikeziri]

I'm having some sed problems now. not sure why:

this one is on mw delete. but after mw add it has some sed commands that fail with same code "m"

sed: 1: "/Users/<user>/.mbsyncrc": invalid command code m sed: 1: "/Users/<user>/.config/mut ...": invalid command code m sed: 1: "/Users/<user>/.config/msm ...": invalid command code m

[LukeSmithxyz]

Oh yeah -i as a flag for sed is GNU specific I believe. -i means "change the file in its place". You should be able to replace something like this:

sed -i "/[0-9]-$title.muttrc/d" "$muttrc"

with this:

sed "/[0-9]-$title.muttrc/d" "$muttrc" > "$muttrc"

EDIT: Won't work. sed will open pipe to file immediately, deleting it.

[mikeziri]

my man says:

-i extension Edit files in-place, saving backups with the specified extension. If a zero- length extension is given, no backup will be saved. It is not recommended to give a zero-length extension when in-place editing files, as you risk corrup- tion or partial content in situations where disk space is exhausted, etc.

will try to figure out later.

[LukeSmithxyz]

Yeah exactly, that's the problem. GNU sed is totally different. Just remove the -i and > to the same file.

EDIT: Won't work. sed will open pipe to file immediately, deleting it.

[Kr1ss-XD]

Actually, GNU sed -i would behave like this man snippet describes. What I could imagine is that on Mac you'd have to specify an empty string explicitely, like : sed -i'' "/[0-9]-$title.muttrc/d" "$muttrc" which would work on Linux as well.

EDIT Probably better (:question: ): sed --in-place="/[0-9]-$title.muttrc/d" "$muttrc"; not sure if this is valid on Mac, though...

[mikeziri]

I'm done with BSD binaries. I'm going to alias sed, du and grep with gsed, gdu and ggrep and add dependencies to them via brew.

[mikeziri]

Ok. done. everything is fixed and running like on linux. will do pull request later today.

[LukeSmithxyz]

@Kr1ss-XD Someone can check, but if it starts with -- it probably ain't portable. -- is the calling card of GNU bloat!

@mikeziri That's fine on your own system, but I'm not about to accept a PR requiring an entire clone of unix core utilities when we could just replace one command type with a portable equivalent. I cannot think of something that more severely violates all my design principles, even though we're talking about Mac OS here.

I suppose it's my fault for getting used to non-portable utilities: Serves me right, I trusted GNU!

[Kr1ss-XD]

Someone can check, but if it starts with -- it probably ain't portable. -- is the calling card of GNU bloat!

My bad... you're right these are called "GNU options" :smiley:

The short option should work though; for sure it does on GNU/Linux. The only caveat I could think of here is that I'm not sure if BSD-sed needs a space between -i and '', on Linux there mustn't be one.

[LukeSmithxyz]

I'm looking at the BSD sed manual and it seems that it works like GNU. So

sed -ibu "s/word/WORD/" file

means replace word with WORD in file using the file filebu as a backup.

I'm going to go ahead and start replacing the sed -i commands with commands with backups (and deleting them I suppose)

[Kr1ss-XD]

Hmm seems you could also encounter issues on different versions of FreeBSD-sed. I've found some info on Stackoverflow about this.

The perl command suggested in the accepted answer would probably be fine, since perl is a dependency of neomutt anyways.

[mikeziri]

@Kr1ss-XD Someone can check, but if it starts with -- it probably ain't portable. -- is the calling card of GNU bloat!

@mikeziri That's fine on your own system, but I'm not about to accept a PR requiring an entire clone of unix core utilities when we could just replace one command type with a portable equivalent. I cannot think of something that more severely violates all my design principles, even though we're talking about Mac OS here.

I suppose it's my fault for getting used to non-portable utilities: Serves me right, I trusted GNU!

sure. anyway, if you want a patch with where I was at, before just adding alias to the top:

https://gist.github.com/mikeziri/845f948c6d40d5b2dfd62b79a0eb0cae

it includes some typos fixes as well.

for anyone who just wants to run it: install gnu grep, sed and du. it doesn't override bsd ones. it links them with a preceding 'g'.

brew install grep sed du

If you need to use these commands with their normal names, you can add a "gnubin" directory to your PATH from your bashrc like:

PATH="/usr/local/opt/grep/libexec/gnubin:$PATH"

bin/mw changes: os=$(uname)

if [[ $os == "Darwin" ]];then muttshare="/usr/local/share/mutt-wizard" alias sed='gsed' alias du='gdu' alias grep='ggrep' fi mwconfig="$muttshare/mutt-wizard.muttrc"

in both cases, don't forget the Makefile changes:

ifeq ($(shell uname), Darwin) PREFIX=/usr/local $(info Running for Macos) else $(info Running for Linux) endif

[LukeSmithxyz]

I've already pushed the sed and grep portability changes up. If the syntax of du is different, I might just switch to ls or find to get that done.

EDIT: Replaced du with find with portable options.

[mikeziri]

dont' forget mailcap path:

set mailcap_path = /usr/local/share/mutt-wizard/mailcap

and

change repos mailcap for macos:

to change openfile with open %s

although I still haven't fixed opening a copy of the file. preview simply ignores the temp file form neomutt's attachments.

[passaca]

Hi,

First of all, thanks for your outstanding wizard, this was really needed. Before I found it, I checked several tutorials and manuals but setting up mutt is really complicated.

I used it on Mac OS 10.14 and most of the things work, but here are some quirks:

The path to "mutt-wizard.muttrc" is incorrect in the "muttrc" file. It points to "/usr/share..." but it should point to "/usr/local/share".

The "mailboxes" in the account config file in my case have spaces which are not escaped ("\ "), leading to wrong mailbox folder names.

The "mailsync" script has two problems with unknown arguments:

pgrep: illegal option -- c

pkill: illegal option -- R

And the generated notification is a bit strange:

The subtitle shows the full path to the account, I guess that's a bit too much. In the actual notification text, the account is correctly displayed with just its name, but the number of new messages is wrong. With one new message it shows "3", with two new messages it shows "6".

If you need me to test something, feel free to ask :)

Apart from that, it looks fine so far. But I just started, so I might come across some more things.

[LukeSmithxyz]

Thanks for the heads up. The issue of /usr/ vs. /usr/local/ was supposed to have been solved, but it ends up we made a change that broke it a couple days ago. It should be fixed now.

The pkill error I've suppressed: it really only is for Linux (specifically for updating the status bar) and doesn't do anything else. I've removed the pgrep extension, which wasn't actually necessary anyway.

Would you mind showing me exactly what you mean as the problem with your mailboxes line?

[passaca]

Thank you very much :)

I have an IMAP folder which is called "Account Archive" and it is added to the "mailboxes" line like so:

=Account Archive

But it should be:

=Account\ Archive

But this is only a problem if the IMAP folders have spaces or other characters which need to be escaped. It wasn't a good idea to give that folder a name containing spaces in the first place, though...

[LukeSmithxyz]

Okay, thanks. I've added in a line that will now escape spaces in mailbox names.

Did you have any other problems that could be related to running it on a Mac? We might have about full compatibility now.

[passaca]

Thanks

One more thing just came up in the openfile script:

mktemp: illegal option -- -

[LukeSmithxyz]

Okay. I've updated the script to make it portable.

[passaca]

Thanks, I checked it, it works.

One more thing: muttimage doesn't work. This is because "w3mimgdisplay" is not available on Mac by installing w3m via Homebrew. I don't know if it can in principle be installed, but at least it's not there out of the box.

[LukeSmithxyz]

Yeah I'm aware. It barely works on Linux too. Luckily, I'll probably be replacing it with previews with the python module überzug sometime soon. You'll just have to view attachments manually until then.

I'm going to close this issue because that should be all the major Mac OS issues and I haven't had any BSD users reach out to bug test. Once I get my hands on a BSD machine, I'll try it out, but the script should work with BSD utils and the rest.

[mikeziri]

@LukeSmithxyz I guess your latest video on GNU is bloat came in good time :+1: hate these discrepancies between unix systems.

[passaca]

Do you know why the calculation of the number of new e-mails fails? It seems to be: (actual number)*3

[passaca]

And there is one more thing, the automatic addition of the cronjob for mailsync does not work.

As far as I can see it, the check if cron is present fails, but I don't know if this check is really necessary as cron will automatically be started if you add something to the crontab.

But on Mac it would be probably cleaner to use launchd instead of cron.

[Kr1ss-XD]

Do you know why the calculation of the number of new e-mails fails? It seems to be: (actual number)*3

My best guess is that it probably has to do with this line in the mailsync script :

newcount=$(find "$HOME/.local/share/mail/$acc/INBOX/new/" "$HOME/.local/share/mail/$acc/Inbox/new/" "$HOME/.local/share/mail/$acc/inbox/new/" -type f -newer "$HOME/.config/mutt/.mailsynclastrun" 2> /dev/null | wc -l)

Is there any chance that the find utility is not case sensitive on your OS ? If so, it would count each inbox folder's content thrice...

[mikeziri]

And there is one more thing, the automatic addition of the cronjob for mailsync does not work.

As far as I can see it, the check if cron is present fails, but I don't know if this check is really necessary as cron will automatically be started if you add something to the crontab.

But on Mac it would be probably cleaner to use launchd instead of cron.

I needed to create a script that calls mailsync to run as user level:

~/scripts/mailsync `#!/bin/bash

$(/usr/local/bin/mailsync >> /Users/username/.mailsync.log 2>&1) echo "end run." ` then crontab -e: set your PATH with common bin dirs: PATH=/usr/local/bin:/usr/bin ...

• • • • • ~/scripts/mailsync # run every minute. you might want to add */5 for every 5 minutes

you also need to go to your security & privacy settings and under privacy tab, add full disk access to cron. stupid macos protecting their users.

[passaca]

Is there any chance that the find utility is not case sensitive on your OS ? If so, it would count each inbox folder's content thrice...

Yes, that's the reason. Thank you. So just leaving the first path fixes the problem. But this would of course mean trouble if the used filesystem is case-sensitive...

[passaca]

And there is one more thing: Ctrl+o (open IMAP folder) didn't go through to the application unless I did this before opening neomutt:

stty discard undef

I put this into my .zshrc (or bash_profile) to make it persistent.

[passaca]

This is probably unrelated to Mac OS, but I just want to make sure:

Is it normal that tagged messages do not get marked in any way? When I tag a message (press "t"), the tag-count in the status line goes up, but the tagged message itself does not change at all.

[LukeSmithxyz]

On tags, this actually just the default settings (I don't use tags myself so I didn't have any UI that shows what's tagged or not).

You can check the neomutt manual for the index_format command and change it to show tags or other things. This is the default set by mutt-wizard:

set index_format="%2C %zs %?X?A& ? %D %-15.15F %s (%-4.4c)"

[passaca]

Ok, thank you very much.

I'm new to mutt, so I didn't know tags were practically not very important as it looked to me as if that's how you select several emails for move/copy/delete.

[Kr1ss-XD]

You can check the neomutt manual for the index_format command and change it to show tags or other things.

This^, and you could also define a color highlight for tagged messages. This is what I did :

color index             brightyellow  blue        '~T'
color index_author      brightgreen   blue        '~T'
color index_subject     white         blue        '~T'

There are more patterns, e.g. '~N' for new mail, '~F' for flagged, '~D' means to be deleted etc., and you can combine them: '~T ~N' for example matches new and tagged messages a.s.f. (There's even a way to define colors epecially for mails that have some specific pattern in the message body !)

There's actually a neomuttrc(5) man page, which is quite extensive but very helpful regarding these and other settings. In case you don't have the manpages installed, you should definitely get them, it's more than worth the effort if you want to customize neomutt to your personal preference, as the wizard can only create a generic framework to get you started.

[Kr1ss-XD]

@LukeSmithxyz

I'll probably be replacing it with previews with the python module überzug sometime soon.

I would appreciate this being optional, b/c w3mimgdisplay works in a tty, while Überzug does not since it depends on X.

[passaca]

@Kr1ss-XD Thank you very much for elaborating on the highlighting issue and on sharing your own settings. This is extremely helpful.

Depending on how "deep" you want the integration into Mac OS to be, it might make more sense to use the System Keychain to store the account passwords instead of "pass" and GnuPG.

Then you would add a new Keychain Item to the "system" keychain in the "Keychain Access.app" and then you can access this via the following command:

security find-generic-password -w -s [chosen-item-name] -a [chosen-account-name]

This command would then replace the settings of PassCmd and passwordeval in the isync and msmtp configs, respectively.

[LukeSmithxyz]

@Kr1ss-XD Well if you want my most evil opinion, I really want to drop image handling entirely... but anyway I could probably make a mailcap wrapper that selects the image display based on if DISPLAY exists though to force w3m in the tty.

As an aside, what exactly do you need to have w3mimgdisplay running in the tty? I've noticed I've had it in the past, but sometimes I haven't (probably related to some package I had temporarily installed). I never looked into it.

@passaca I'll consider it. Obviously the big problem is me not having a Mac to test it, and I have no clue what the commands would be.

Honestly at some point it might be better to fork/branch the project to have a version targetted to Mac. When I first wrote this I never expected Mac-users to want it, but a lot I know do use it.

[Kr1ss-XD]

Well if you want my most evil opinion, I really want to drop image handling entirely... but anyway I could probably make a mailcap wrapper that selects the image display based on if DISPLAY exists though to force w3m in the tty.

I admit image support is not vital, and besides that folks using mutt in an environment w/o display server could probably be supposed of being able to set up a mailcap file on their own :laughing: Nevertheless, this would be greatly appreciated !

what exactly do you need to have w3mimgdisplay running in the tty?

I'm quite sure you don't need any extra packages installed, besides of those which are anyways necessary for image support in X, too. So you should be fine when your kernel has framebuffer support enabled, which is the case w/ standard Arch/Arch-LTS kernels. If you compile your own custom kernel, you need to make sure to include the according modules. On the other hand, even when compiled into the kernel or provided as a module, FB support can still be deactivated at boot time by kernel command line options, so this might also be something you'd like to check.

[passaca]

Honestly at some point it might be better to fork/branch the project to have a version targetted to Mac. When I first wrote this I never expected Mac-users to want it, but a lot I know do use it.

Yeah, I think it might be overkill to go past the point of "it installs and runs properly" for Mac OS.

I guess it's sufficient to mention in the README that pam-gnupg won't work for Mac (as far as I know), but that one can use the system Keychain if one wants to have similar behavior.

The message count problem on case-insensitive filesystems could be resolved by adding something like the answer proposed by "John" here:

https://apple.stackexchange.com/questions/71357/how-to-check-if-my-hd-is-case-sensitive-or-not

[passaca]

If anybody is interested, I will post the script (based on mailsync by @LukeSmithxyz ) I use to periodically sync mail, to notify about new mail using the native Notification Center and to show the current number of unread e-mails per account in the Mac OS menu bar:

It uses BitBar to run the script in definable intervals (so no need for cron) and to update the number of unread mails in the menu bar. It also uses terminal-notifier to show notifications when new mails arrive (containing the From:-address as title and Subject: as text).

#!/bin/bash

# without this line, the unicode "pipe" character used to concatenate the texts breaks the script
LC_ALL="en_US.UTF-8"

# redirect stdout to log file
exec 4<&1
exec 1>>$HOME/.mail/mailsync.log

# print date for logging purposes
echo $(date +"Date: %d/%m/%Y Time: %H:%M:%S")

# Check for internet connection
ping -q -c 1 1.1.1.1 > /dev/null || exit

# run mbsync for all accounts
/usr/local/bin/mbsync -a

# run notmuch
/usr/local/bin/notmuch new 2>/dev/null

# loop over all mails newly added since last sync and output a notification message for each of them 
show_notifications () {
    find "$HOME/.local/share/mail/$1/INBOX/new/" -type f -newer "$HOME/.mail/.mailsynclastrun" -print0 |
    while IFS= read -r -d '' file; do
        fromaddress=$(cat $file | sed '/^$/q' | sed '/^From: */!d; s///;q' | perl -CS -MEncode -ne 'print decode("MIME-Header", $_)')
        subject=$(cat $file | sed '/^$/q' | sed '/^Subject: */!d; s///;q' | perl -CS -MEncode -ne 'print decode("MIME-Header", $_)')
        /usr/local/bin/terminal-notifier -title "$fromaddress" -message "$subject" 
    done
}

show_notifications "acc1"
show_notifications "acc2"

# record last sync time
touch "$HOME/.mail/.mailsynclastrun"

# re-route the output back to normal stdout
exec 1<&4

# count all unread mails in the accounts
get_unread_count () {
    echo "$(find "$HOME/.local/share/mail/$1/INBOX/new/" -type f 2> /dev/null | wc -l | sed 's/^ *//')"
}

unread_count_acc1="$(get_unread_count acc1)"

unread_count_acc2="$(get_unread_count acc2)"

# set menu texts for accounts if there are unread mails
if [ "$unread_count_acc1" -gt "0" ];
then
    text_acc1=" A: $unread_count_acc1 "
fi

if [ "$unread_count_acc2" -gt "0" ];
then
    text_acc2=" B: $unread_count_acc2 "
fi

# function to join strings with separator
function join_by { local IFS="$1"; shift; echo "$*"; }

# join account texts if they are defined and prepend with red mail symbol
concatenated="$(join_by "┃" ${text_acc1:+"$text_acc1"} ${text_acc2:+"$text_acc2"})"
concatenated="${concatenated:+"📮$concatenated"}"

# show joined texts (in case there is mail) or grey mail symbol (in case there is no mail) in menu bar
echo "${concatenated:-"✉️ "}"

As I'm not very experienced, the coding style is lacking and it is hardcoded to two accounts, but that can easily be extended to as many as you like, of course.

Unfortunately, only one Notification can be shown at any time if the Banners style for terminal-notifier is selected in "System Preferences->Notifications->terminal-notifier". So if you receive more than one new e-mail per sync-cycle, only one of them will persist for a few seconds as notifications, the other ones are immediately overwritten. I don't think that's a big deal, as this is also the case for Thunderbird, for example.

It is possible to show all of the notifications by changing to the Alerts style. But this has the big drawback that one has to dismiss each of the notifications explicitly, otherwise they just stay there.

[unwillexist]

BSD does not support negative input to head like head -n -1 it must be written different ways I don't know which one is more portable

$file | tail -r | tail -n +1 | tail -r

There is more ways at https://unix.stackexchange.com/questions/169079/negative-arguments-to-head-tail

[unwillexist]

Thats what I got with as notify output with | tail -r | tail -n +1 | tail -r in FreeBSD. Not getting any illegal count errors but still not perfect. https://ibb.co/10v42Xg

[unwillexist]

https://ibb.co/ydJFB9L https://ibb.co/Q9MCbrG there are the mail I've sent to myself and output that I got.