Rework our Mac OS X integration

[BjarniRunar]

The current Mac OS X package is not really usable. Known issues:

1. The package is unsigned (see #1366)
2. If multiple users launch Mailpile (using user switching), ports conflict (#1139, #877)
3. The current approach of launching a terminal and then spawning a browser confuses users

Item 3 above is the biggie; we are violating the principle of least surprise in a big way. As I understand it, Mac OS X users expect the following:

1. The app launches and can be started/stopped using an icon in the dock at the bottom.
2. When clicked, the dock icon should launch a window with UI elements (there may be more than one)

So that's what we should do. The dock icon can represent the fact that Mailpile is running, clicking it could launch a browser tab (1st iteration?) or a webkit wrapper (2nd iteration?). In addition there should be an option to open a "debug window", which would be the terminal CLI interface. This should not launch by default.

I think implementation-wise, this means we need a custom app which in turn launches Mailpile in the background and coordinates things. This app can also take care of choosing which port to run on (issue #887). If I recall, the screen utility is included on the Mac, so this launcher app could use the same strategy as mailpile-admin.py, of launching Mailpile itself inside a named screen session that the user can then attach to for debug purposes (and provide a context-menu item for doing exactly that).

[NoahAndrews]

I see a few different ways we could take this. One is to make it seem as much like a normal app as possible. When the user launches the app, a webview comes up containing Mailpile. Opening a browser tab when the app is launched would feel pretty jarring. When the app is quit, the server quits with it. Obviously, this isn't good if a user wants to host Mailpile for other machines.

Option two is that we write a native Mac app that has options for stopping and stopping the server, changing the port, opening the debug window, and opening Mailpile in a browser tab. The server would continue in the background even after the app was quit, with an icon in the status bar for easy access to the control panel app. This is probably the most complicated option.

Finally, we could operate like (for instance) the Dropbox app. On first launch, we have some kind of interface to explain what's going on, but after that, the app lives entirely in the status bar. When the user launches the app, the only thing that visibly happens is that an icon is added to the status bar, and they know that they can now go to Mailpile from their browser, or click an option in the status bar menu to open it for them.

Issue number 2 could be a bit tricky for options 2 or 3, since both of those involve accessing Mailpile directly through the normal browser. If we can't depend on having a consistent port number, the user can't bookmark it, and we shouldn't encourage them to access it through the browser. Choosing a random port upon setup would work.

Really this comes down to whether we want the app to act as an app or as a server. How do we get around issue 2 on the Windows side?

P. S. Did you reference the right issue about user switching? Ah, found the right issue. Option 2 from over there ought to work pretty nicely. I'm honestly not sure why there needs to be anything Mac-specific about solving that.

[BjarniRunar]

I referenced the wrong issue wrt ports - fixed!

I think the first option - the webview - is the one to aim for, although I think that we could get away with leaving Mailpile running in the background as long as the dock icon is "lit up" to show it's still running. I think a bunch of apps behave more or less that way already - you close the last window but the app is still "running". Am I wrong?

[BjarniRunar]

Regarding the dropbox-solution, if we have to explain things, then we're failing our job here. We want to match user expectations, not force them to learn that Mailpile is special and weird. Most users wouldn't read an explanation anyway, so we'd have to invest a fair bit of effort into design and visual communication to make sure that actually works. It's harder than it looks!

[NoahAndrews]

You're not wrong about being apps continuing to run after the last window is closed, but that doesn't mean it's a good way to get the functionality we want. For one, I don't like the idea of a background service taking up precious Dock real estate. But the bigger issue I see is that it's too easy to quit. As a user, when I'm done using an app, I quit it. I don't think to leave it open so that it's accessible from other computers. This would be an example of having to explain things.

I think somewhere we're going to need to put an option to run as a background service. If that option is checked, we also put an icon in the status bar, which will remain after the webview-app is quit.

[NoahAndrews]

You've got a great point about matching expectations. It should work just like any other email client, so option 1 it is.

[NoahAndrews]

I'm thinking we may need to use something like Electron or nw.js. I do have some node.js experience. My research indicates that Electron is more actively developed, starts much faster, and results in a somewhat smaller app. Unfortunately, Electron requires OS X 10.8, and nw.js requires OS X 10.7.

One interesting thing about this route is that these are cross-platform solutions. If we like the end result on Mac, we could easily bring the same packaging to the other platforms.

I just found a couple of closed Electron issues that explain how we should be able to get the functionality I talked about before with that combination of status bar (tray) icon and dock launcher.

https://github.com/atom/electron/issues/1154 https://github.com/atom/electron/issues/422

As long as we can find a good way to communicate between the Python code and the node.js code, I think this will be the best way if we want to integrate with the operating system as much as possible.

Any thoughts?

[NoahAndrews]

I've officially got a start on Electron-enhanced Mailpile! http://imgur.com/qYMMyXs

It's not too late to turn back if you have an objection of course.

The next thing I need to know is how to add a command line flag to the mp command without making it available as an option in the interactive interface. Specifically, I need a way to start the server without opening a new browser tab. It seemed like --www was what I wanted, but it turns out that that just changes the socket the server will listen on. I'll let you decide if that deserves an issue or not.

[NoahAndrews]

Perhaps we could also have a download option for old versions of OS X. That package would be generated using platypus, and would use a standard webview (Electron uses Chromium). It would lack the option to run in the background and the host OS integration that Electron provides us with.

[BjarniRunar]

Electron looks amazing, but incredibly bloated for our use case. Instead of shipping a 200k native Mac wrapper, we're going to ship 100 megs, including node.js and Chromium - not counting Mailpile itself which may include gnupg, tor and possibly Python? Yikes. This adds a huge support burden to us upstream, as we'll have to repackage and update every time a security vuln is found and fixed in Chromium, Electron or Node, instead of relying on Apple's OS updates for the OS-provided embeddable webview (Safari).

I'm not sure this is a reasonable thing to do, except perhaps only as a prototype to work out exactly what the "real" thing needs to look like.

[BjarniRunar]

A bunch of discussions happened on IRC today, some of the main take-aways:

We agreed that Electron was maybe too heavy.

Embedding a webview vs. using the browser was debated, our 1st attempt will be to use the browser but give the user control over things; the wrapper app will pop up a window offering the user buttons to view in browser or view the CLI, and a tickbox to make launching in the browser automatic from then on - this should hopefully remove the element of surprise (few apps launch the browser) without adding any inconvenience once the user has ticked the box.

The app will have a dock icon like other apps, closing the launcher/wrapper/intro window won't terminate the app, instead the user must press an explicit Quit button.

The wrapper will launch Mailpile in a screen session, much like the Apache integration script does.

@NoahAndrews made a mock-up, I am pretty sure he has started working on the wrapper app!

[pix2D]

Please consider adding an option to hide the dock icon and have it live in the status bar instead. Dock space is always at a premium on smaller resolutions and since this is effectively something most people would leave on permanently (I think?) it would be nice to have the ability to not have it in the dock.

[BjarniRunar]

@pix2D Thanks for the suggestion. As an option, that would make sense. It can't be the default mode of operation because of user confusion, but having it as an option shouldn't hurt (as long as there's a resonable place to communicate the option itself, which isn't quite a given). I don't consider this high priority, but it would be nice to have.

[BjarniRunar]

Note: work on this issue should happen in the gui-o-matic project, so the results can be re-used by other projects: https://github.com/mailpile/gui-o-matic

[marksev1]

Why is this issue Mac OS specific. Isn't this project crossplatform?

[caesar]

If MailPile is to be familiar to users of existing mail clients (and of existing software in general), it should run as a standalone app. It shouldn't have to be loaded in the browser. Certainly it shouldn't leave a standalone app running and have to use the browser for its UI. I appreciate that you might want to use the browser in the short term before putting work into developing a standalone UI wrapper. But your long term goal should be for the UI to run in its own process if you want users to be familiar and comfortable with it.

Plus, there is a huge security benefit to avoiding the browser. Personally, I would not trust my PGP keys to a script running in the browser, and I know a lot of security experts have made the same recommendation.

As a side note, while I understand and agree with your concerns about Electron being bloated, it's important to note that it exists. Maybe it would be a good idea to use it for now, to focus your efforts on the unique code in MailPile rather than on creating a new wrapper framework? This can be done at a later date if/when developers have free time for it.

Incidentally, Nylas N1 (based on Electron) is about the same size as Thunderbird. Doesn't mean we can't do better (and we should!), but just a note of comparison that existing mail clients are already pretty big.

[DaLynX]

From my understanding, the keys are never used by a script in the browser (that would be javascript, I guess?), but directly by python calls to GPG. No?

Caesar Schinas notifications@github.com wrote:

If MailPile is to be familiar to users of existing mail clients (and of existing software in general), it should run as a standalone app. It shouldn't have to be loaded in the browser. Certainly it shouldn't leave a standalone app running and have to use the browser for its UI. I appreciate that you might want to use the browser in the short term before putting work into developing a standalone UI wrapper. But your long term goal should be for the UI to run in its own process if you want users to be familiar and comfortable with it.

Plus, there is a huge security benefit to avoiding the browser. Personally, I would not trust my PGP keys to a script running in the browser, and I know a lot of security experts have made the same recommendation.

As a side note, while I understand and agree with your concerns about Electron being bloated, it's important to note that it exists. Maybe it would be a good idea to use it for now, to focus your efforts on the unique code in MailPile rather than on creating a new wrapper framework? This can be done at a later date if/when developers have free time for it.

Incidentally, Nylas N1 (based on Electron) is about the same size as Thunderbird. Doesn't mean we can't do better (and we should!), but just a note of comparison that existing mail clients are already pretty big.

---

You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/mailpile/Mailpile/issues/1514#issuecomment-217752665

[s4y]

(I landed here via the "Packages are being reworked" note on the Mailpile homepage and I'm chiming in as a Mac developer. Excuse me if it's not the right place to discuss this anymore, or if an opinion without a PR on this issue is unhelpful 🙃.)

---

Electron looks amazing, but incredibly bloated for our use case. …instead of relying on Apple's OS updates for the OS-provided embeddable webview (Safari).

WKWebView is available on macOS 10.10+, which is a reasonable deployment target with 10.13 as current. The main benefit of Electron is the platform-agnostic API, but I would favor WKWebView for the bloat and security patch reasons you mentioned.

If multiple users launch Mailpile (using user switching), ports conflict (#1139, #877)

In addition to making it tricky to run more than one instance of Mailpile on the same machine, using a port at all is undesirable from a security standpoint because nothing stops users on the system from connecting to each other's ports. One alternative would be to load the web app directly from the filesystem and expose a JavaScript pipe to the backend (which could be its stdin/stdout, another fd, or a UNIX domain socket) via a user script message handler.

I think that we could get away with leaving Mailpile running in the background as long as the dock icon is "lit up" to show it's still running. I think a bunch of apps behave more or less that way already - you close the last window but the app is still "running". Am I wrong?

If clicking the Dock icon opens a window within the app, staying running in the Dock is OK. The built-in Mail app works that way. If clicking the icon opens a browser or configuration window, that would be strange/surprising to me. Here are some interesting points of reference (the best one for Mailpile IMHO is at the end):

• Apps that are just document viewers/editors, like TextEdit and QuickTime, disappear from the Dock if you switch away from them with no windows open.

• Launchpad, Siri, and Time Machine all have .apps in the filesystem which can be launched to trigger some action (like showing Launchpad). If a user adds one of these apps to their Dock, it can be clicked, but it never shows up as running in the Dock.

• Time Machine is managed through a pane in System Preferences. It has an optional status item that can be used to monitor it and perform actions like "stop the current backup" and "open the preference pane", but it runs in the background even if the user turns off the status item. Launching the Time Machine app kicks either the current application or the Finder into Time Machine without appearing in the Dock itself.

• Backblaze (third party) mostly behaves like Time Machine: it has a preference pane, an optional status item, and a .app. Launching the app opens System Preferences to its preference pane.

• ✨Messages ✨ (I wish Mail.app worked like this, too). It's an app. Once you've configured an account, it spawns a background process which runs, with no status item or Dock icon, all the time. You launch Messages.app to read and write messages, but the background process handles fetching inbound messages and delivering outbound ones, so you never have to wait for things to sync up when you launch the app. The background process is also responsible for message notifications. In the past, when Messages supported other kinds of accounts, like Jabber, the background process had an optional status item with a menu to see your buddy list and set your status message. But, even with the status item turned off, you'd be online and it would launch the app to a conversation window when someone messaged you.

So, in my perfect world, Mailpile would:

• Have a .app that showed its own windows, probably containing WKWebViews.
• Run its backend in a background process, probably via SMLoginItemSetEnabled, ideally without listening on a port unless the user explicitly wants it to. It could have an optional status item, but I'm not sure if there's anything useful to put in there.

I'm happy to answer other Mac questions, if you don't already have someone helping out on that front.

[BjarniRunar]

Thank you for your comments @s4y ! It's great to get feedback from a Mac developer.

First of all, to all the people suggesting we don't use the system browser (hi @caesar!): that's not going to happen. This isn't up for debate at this point. We've gone back and forth on this a lot (both on GH and elsewhere), and although yes, this does imply a slightly higher learning curve for new users, I have become convinced that it is worth it. Why boils down to two things: security and uniformity across platforms. Mailpile is such a small effort that we cannot afford to document and support and maintain radically different user interfaces across the different platforms: we have to keep things as similar as possible between the Mac, Windows and Linux offerings. Browsers are very security-critical apps and historically any "system webview" or "webkit binding" has lagged behind the main browsers in terms of patches, if not on the Mac then on some other platform. So overall, this just isn't an option for us.

Your idea of not using a port at all is interesting, but also a non-starter. Mailpile just isn't written with any other mode of access in mind, and in particular the things we want to do with remote access (so you can access your Mailpile from your phone) depend on Mailpile providing a standard HTTP interface. Again, even if this may not be what most Mac users want to do, uniformity across platforms is important here.

The rest of your points are very well taken and useful. Thank you so much for taking the time to post, we will take all of this into consideration.

As a general update, we are working on putting together a job description and plan to actually hire someone to work on this soon (funded by selling some of the Bitcoin folks donated to us back in 2013).

[ghost]

In reply to what BjarniRunar commented on Jan 5, 2016

The app launches and can be started/stopped using an icon in the dock at the bottom. [...] The dock icon can represent the fact that Mailpile is running, clicking it could launch a browser tab (1st iteration?) or a webkit wrapper (2nd iteration?).

I understand there has been a debate on whether to use a tab in the default browser or WebView. Launching a browser tab integrates poorly with macOS, because:

• Users expect a "new" window to appear when clicking a the dock icon.
• When tabbing between applications using cmd+tab (eqv. to alt+tab on linux), the user expects to see Mailpile's icon on the preview. However, if a browser tab is launched, then the user will see his browser's icon and not Mailpile. Worse, the tab preview shows a single icon per app, irrelevant of how many browser tabs or browser windows are open.
• Shutting down Mailpile will not close the browser tab. Closing browser tabs can be achieved via AppleScript as to issuing a "close command" to any browser tab which is showing the Mailpile UI. However, doing so leads to an integration overhead in which coupling with multiple browser (Safari, Firefox, etc.) must be maintained.

While I do understand the security concerns of using WebView,

macOS Integration Proposal

Status bar

A black and white variant of the Mailpile icon is shown in the status menu (top right of the screen). Clicking that icon leads to a drop down menu from which the Mailpile GUI-client can be launch, the console can be accessed, the daemon started and stopped.

Other applications, such as Postgres.app, Little Snitch, Micro Snitch and Arq take this approach. The following figure depicts a status menu in which Postgres.app has been clicked.

Dock

Ideally a dock icon shall display the number of unread emails.

The following figure depicts a portion of a dock. The third icon from the left is Mail.app's icon (Apple's email client), on which the number of unread email messages is marked as one.

The following figure depicts Postgres.app on the dock as a blue elephant (the fifth icon from the right) and the GUI which appears in response to a click on that icon.

This is to say that Mailpile resides both on the status bar and in the dock. (Note that it is possible to dismiss the dock icon without closing down the daemon, as is the case with other apps such as Postgres.app which still "runs" in the status bar.) This approach remediates the following concern:

When the app is quit, the server quits with it. Obviously, this isn't good if a user wants to host Mailpile for other machines. -- NoahAndrews on Jan 5, 2016:

Summary

As a user, I expect the following behaviour upon a clicking Mailpile in the status bar, given that the daemon has previously been configured with currently valid settings:

• The mailpile daemon is started, should it not be running already.
• The mailpile GUI appears on the screen.

---

In reply to what BjarniRunar commented on Jan 10

Browsers are very security-critical apps and historically any "system webview" or "webkit binding" has lagged behind the main browsers in terms of patches, if not on the Mac then on some other platform. So overall, this just isn't an option for us.

Today, 18th of February 2018, Safari webkit's version is identical to the one of WKWebView. It might be that both are using /System/Library/Frameworks/WebKit.framework. I guess the only way to be sure is to contact Apple, as such I raised a developer support ticket with them -- will comment on this thread after they have replied.

[BjarniRunar]

Hi @peturingi ! Thanks for your comments. Your proposal above is very similar to what I had in mind!

However I do want to re-stress my point from the comment above: it is valuable to us if we can end up with a similar solution on all platforms, Windows, the Mac and Linux. So even if the Mac doesn't have the security issues I discussed above, if the other platforms do that effectively vetoes the idea of using a separate window and a WebView. By my understanding, this is currently the case, so that idea is vetoed, not by the Mac itself, but by other platforms. In particular, I think the situation on the Linux desktop is actually pretty bad and I would hesitate to go this route there.

Once we have hired for both the Windows and Mac roles, one of the first tasks will be to explore and discuss what each platform offers and what users of each platform expect (I myself will be representing the Linux universe). We'll then try and find a balance which keeps things as similar as possible across all three, while minimizing the learning curve for users. That conversation is inevitably going to override whatever we discuss here, so it doesn't make sense to invest a lot of time in this issue just now.

[BjarniRunar]

This is Done. Closing; bugs belong in issues of their own.