Can macOS's native WebKit be used for the macOS build?

[c-alpha]

The discussions in the comments on commit astroidmail/astroid@0d24d8330b3cd908e2d016d05bb2161b6163fae7, which was about updating to the latest WebKitGTK release, sparked a discussion on the use of WebKitGTK on the macOS platform. This issue was created to give this discussion a proper home of its own, and to stop polluting the commit comments for astroidmail/astroid@0d24d8330b3cd908e2d016d05bb2161b6163fae7.

Building WebKitGTK on macOS is a time-consuming task (several hours), and a Homebrew formula for it needs to be crafted (even more time-consuming), since it is not part of the Homebrew core.

macOS already comes with a prebuilt WebKit as part of the platform, however (cf. Apple's WKWebView developer docs). So what is the point of having two WebKit implementations on the same platform?

The purpose of this issue is to discuss, share research results, and share ideas about how a wrapper layer could be made around macOS's native WebKit implementation, so that astroidmail/astroid can build & run using the native WebKit on macOS.

Things to do (evolving list):

• [ ] research how WebKitGTK is used by astroid today
• [ ] benchmark this against the macOS native WebKit API
• [x] research prior art in WebKit wrappers (e.g. zserge/webview)

[c-alpha]

• https://www.paulirish.com/2013/webkit-for-developers/
• https://electronjs.org

[gauteh]

@yeled: ping ;)

[c-alpha]

In addition to ElelctronJS, which I mentioned in my previous comment, I found a couple of more candidates for replacing WebKitGTK:

• https://github.com/zserge/webview
• https://github.com/r0x0r/pywebview
• https://github.com/ultralight-ux/ultralight
• https://github.com/pojala/electrino

Here is an (admittedly arbitrarily chosen) set of statistics:

	ElelctronJS	zserge/webview	r0x0r/pywebview	Ultralight	electrino
Project Status	active / stable	active / rewrite	active /	active / stable	active / development
Last Commit	3 hours ago	January 23	8 days ago	3 hours ago	June 9, 2017
Licence	MIT	MIT	BSD-3-Clause	Custom	MIT
Issues (open/closed)	1133 / 9301	108 / 79	19 / 201	63 / 18	14 / 5
Pull Requests (open/closed)	97 / 6806	17 / 41	2 / 97	0 / 2	2 / 1
Contributors	837	20	25	2	3

Given that electrino is marked as "do not use this in production", and has seen its last commit a year and a half ago, I think we can safely discard it, and focus on the other four.

Here is an (again admittedly arbitrarily chosen) set of more or less technical parameters:

	ElelctronJS	zserge/webview	r0x0r/pywebview	Ultralight
Language Bindings	JavaScript	C, C++, Go, Rust, Python, Nim, Haskell, C#	Python	C++
Required Build System	npm	CMake	none	CMake
Target Use-Case	app framework	web view wrapper	web view wrapper	app framework
Used by other projects?	shedload	a couple	???	???

Despite being on GitHub, Ultralight is not open source (they upload binaries), plus it seems to do too much. So I think I'll give it a try with zserge/webview in a new branch.

[gauteh]

Webview looks promising, would be great to replace webkit with something of appropriate size compared to what we are using! I think webkit is fairly well separated from the rest of the application so it should be possible without completely redesigning things. Very interesting.

[c-alpha]

Just came across another comparison: https://blog.budgetwithbuckets.com/2018/12/13/making-an-app-2018.html

[c-alpha]

@gauteh, here's one for the GTK expert: 😉

In the ThreadView::ThreadView constructor, I am calling the webview create API:

webview_t webview_create(int debug, void *wnd);

Now my trouble is: what do I pass in as the wnd pointer? If I use the MainWindow * which the constructor gets passed in, it crashes because the webview implementation expects a native window for the platform. If I pass NULL as the wnd parameter, the webview implementation creates a new window for me; but that's not what I need.

I guess I need a way to obtain a pointer to the native window object behind the MainWindow *, so I can pass it to webview_create(). Any clues?

[gauteh]

Do you mean the C-binding GtkWindow (as opposed to Gtk::Window) or one of the XWindow-ish things?

[gauteh]

You can use Gtk::Window->gobj() to get the GtkWindow. Where is the docs for webview_create? Also seems like it is currently being re-written: https://github.com/zserge/webview/tree/webview-x

[gauteh]

The window you want is probably the one from the widget where the webview should be drawn: https://developer.gnome.org/gtk3/stable/GtkWidget.html#gtk-widget-get-window, but the container might have some notes about how to get it.

[gauteh]

Another alternative: https://github.com/cztomczak/cefcapi/tree/master/examples, haven´t checked whether it works on Mac.

[gauteh]

I´m considering moving to a JS/typescript/node lib for the thread-view. Basically in the same way as @ramblurr did in https://github.com/astroidmail/astroid-js. Hopefully https://www.npmjs.com/package/sanitize-html or similar can be enough to guarantee that emails wont be able to include JS. Must be enough projects that already have that problem, and it seems that most APIs go via the JS engine anyway (e.g. the new webkit does). This would make us more agnostic to the web engine we are using. I dont know if this is the way to go or not, but it would also be nice to have the thread-index in the same UI. But have to make sure it is fast enough.. Maybe I should just do some tests. Would be easier with only one UI toolkit to manage.

Should clearly define what separates us from e.g. https://electronjs.org/apps/mailspring. Speed and lightweight-ness of course. Vim or custom editors. Although, embedded would probably have to go. Small inline editor will be easier.

[c-alpha]

I've been mute on this for quite a while now, for which apologies. I spent the time doing some small experiments, and mostly researching & thinking.

It seems we are looking at a conceptual problem here: how to make it cross platform? And which platforms do we include? Thinking in the terms of astroid, it seems we're caught in a Bermuda triangle of competing goals: we need a web view we can configure to be a sandbox to render the email messages; then there is the native UI API of the platform (GTK, AppKit, UIKit, .NET, you name it); and thirdly we want to be platform independent.

This is a totally subjective overview of my findings:

I put zserge/webview in the middle, because it is kind of a tweener: it offers a very reduced but cross-platform C/C++ API for creating a web view, and the rest is JavaScript. As opposed to the more platform-specific options, there is no access to "advanced" features like controlling whether JavaScript is enabled, or whether remote resources are loaded or not. To base our architecture on this, we would need to write extensions to zserge/webview. And making it coexist with GTK+, zserge/webview still has the issue of needing access to the low-level, native window object, and the resulting question of how GTK+ reacts to someone else fiddling with the native window under the hood. And there's no DOM API in zserge/webview, but which is needed for astroid.

An even less attractive (IMO) alternative would be to develop our own wrapper. Lots of code, an extra maintainer per UI platform, ...

The next thing to look at were the web app frameworks. These are entirely portable, of course. The drawbacks are that either huge and slow, or designed for mobile devices, or both.

One further contender, that I only discovered recently, deserves a special mention: the Chromium Embedded Framework (CEF). In terms of the diagram above, it would probably sit between zserge/webview, and the web app frameworks. If you check out their tutorials, you will see that the code still has #ifdef LINUX type stuff, but it would seem to me at an acceptable level. What I have not yet been able to establish is whether the browser features can be controlled closely enough to be suitable as a sandbox environment for rendering email messages. Looking at the list of apps that use CEF on Wikipedia, notable runners up include Foxmail, Adobe Acrobat, Matlab, Facebook Messenger, Mailbird, and Spotify.

So I think CEF could be an interesting candidate for replacing WebKitGTK+, provided that the sandbox restriction features can be implemented (which is still to be confirmed). It also offers a C/C++ API (small/fast), isolates the app from the graphical UI platform, and provides a web view.

And I think it would be a less drastic change for astroid than moving to a web app framework, since in that case all (most of) the business logic would probably need to move to JavaScript.

CEF's licence is an MIT-style one, so quite friendly.

CEF could be "the right cut". Looking forward to your thoughts!

[c-alpha]

Small update: looks like we're "good" wrt. to the browser sandboxing: https://magpcss.org/ceforum/apidocs3/projects/(default)/_cef_browser_settings_t.html

[gauteh]

Hi, I just invited you to an experiment I has been working on. It uses a backend in rust and a web-frontend. It can run in browser (tested on chrome/chromium), electron, or nw.js. But could use anything. I tested several frameworks, and found that https://infernojs.org/ can be made really fast (easily showing 100k rows). I haven't tried running it in a while, but please try. I developed it on Mac OS X, but it works(worked) on linux. I could just open it, but it is not really ready. If more people are interested I could make it public as it is.

[gauteh]

Thanks for the through research. Really good.

[gauteh]

Doesn't seem to compile at the moment, but it is not much code that needs to be updated. actix might not be the best choice as backend. But it is fast. Lists threads at the same speed as the notmuch search command. Which it is using in the background anyway.