Plan

[leeoniya]

Gmail has its main, bloated/slow web interface [1]. It has fast html-only interface [2], which sucks. It has another much better mobile interface [3] that looks good but lacks features, still serves almost 1MB of assets and is clumsy to use on desktop. It has a native Android Gmail app [4] which has been getting slower and more resource hungry with every update.

The plan is to do better, much better:

• Total payload < 200k gzipped (perhaps < 100k)
• Network load time over chrome's throttled "Good 3G" connection: <= 2s
• Layout that can be used on desktop OR mobile (via normalize.css & reflex.css flexbox grid)
• Show message size column, maybe column reordering
• Multi-stage sortable columns
• Grouping / threading / labeling
• Filtering / searching
• Markdown composition (via marked.js)
• Autocomplete for some contacts store or ajax endpoint during reply/compose
• Support for attachments + image preview (File api)
• use domvm.js for vdom view layer & router
• use xr.js for ajax sugar
• Support for touch gestures (via hammer.js)
• Support for keyboard shortcuts (via mousetrap.js)

Maybe use Google's CORS apis [5], else a demo using an sqlite backend with a REST router to issue queries. To accomodate missing apis (like getting contact list) a local websocket server can run in background to deliver data, maybe https://github.com/TooTallNate/Java-WebSocket

[1] https://mail.google.com/mail/u/0/#inbox [2] https://mail.google.com/mail/u/0/x/ [3] https://mail.google.com/mail/mu/mp/138/#tl/Inbox [4] https://play.google.com/store/apps/details?id=com.google.android.gm&hl=en [5] https://developers.google.com/api-client-library/javascript/features/cors

[l-cornelius-dol]

I won't be able to contribute much over the next two to three months, as we cope with Firefox dropping Java applets... but sign me up; at the least I'd like to follow and early next year I should get my life back.

[yosbelms]

I would like to contribute but https://developers.google.com returns a 403 error to me because U.S. government restrictions to my country. Anyways, I'll be watching, cheers.

[leeoniya]

@yosbelms there is plenty to do without relying on Google or its OAuth2 layer.

I've made an IMAP import script that sucks raw messages into a database & dumps decoded attachments onto disk. This script will simply run on a 1min cron to poll for and import new messages. The rest of the app will use this as a backing store for filtering/sorting/grouping.

I'm trying to make the setup process easy and portable but have run into some issues.

For an index i'm using SQLite3 with FTS5 module [1]. The current problem is that the bundled pdo_sqlite.dll in PHP7 is not compiled with FTS extensions, so fails to insert records with "unknown module fts5". I have to set up an MSVC env and recompile it to continue.

Another option is using mysql or postgres since both have fulltext indexing built-in but they're not lightweight and require setup/daemons or a lot of storage to run portably.

[1] https://sqlite.org/fts5.html

[leeoniya]

good news. apparently FTS3 is enabled in the bundled sqlite builds in php7, and now FTS4 and FTS5 will be in upcoming releases [1].

EDIT: back in business using this snapshot build: http://windows.php.net/snapshots/#php-7.0-nts-windows-vc14-x64

[1] https://github.com/OSTC/php-sdk-binary-tools/issues/4

[leeoniya]

Okay, I did a test import to gauge what the perf might be like on the backend.

Ready for some stats from my average 2013 desktop?

• 5001 messages (about 40% of my Inbox)
• Import took ~ 5min via Gmail's IMAP (including all attachments)
• Indexed fields: message_id, subject, from, to, cc, bcc, reply-to, date, received, body (text or html->text)
• Index size: 27.8MB

Searching for messages containing a specific word in the message body:

Results: 310 emails.

Gmail's search: 619ms SQLite3 query: 3ms

That's no mistake. There will be some minor added overhead for network transfer/latency, so let's round it to ~40ms.

[leeoniya]

Working code for the IMAP importer has landed [1].

• It properly detects and tags threads in the DB.
• It extracts attachments and should do proper decoding of everything.
• PHP <= 7.0 on Windows has the shitty property of being unable to create filenames in UTF-8, so attachment names that aren't ASCII might get mangled (but contents will be ok). Supposedly this is fixed in PHP 7.1.
• It's as fast as your disk can write and internet can download. It's somewhat memory hungry (there's some leak in this loop [2]) though it never went over 1.5GB RAM usage when importing 12,000 messages from my Inbox and dropped back to ~100MB when that function exited to scan Sent Items. Should be simple to fix.
• DB size was ~70MB for 14,000 imported messages.

Instructions are in the readme for the brave.

[1] https://github.com/good-3g-apps/email/tree/master/imap-importer [2] https://github.com/good-3g-apps/email/blob/master/imap-importer/ImapSqlite.php#L91

[leeoniya]

Apparently imap_fetchbody() leaks memory (as does imap_savebody()). Seems like there's a history of this in the past [1]. Implemented a work-around by disconnecting and reconnecting imap every $batch_size = 100 messages. It's a hack, but it's actually very fast, so fine by me. This keeps mem at a consistent 20-40MB depending on what's fetched in that batch. Better than 1.5GB peak :)

EDIT: submitted at https://bugs.php.net/bug.php?id=73493

[1] https://www.google.com/search?q=imap_fetchbody+memory+leak

[leeoniya]

apparently these "leaks" were due to built-in caching in the imap extension. solved by calling http://php.net/imap_gc

now mem usage stays at ~15MB without any batching gymnastics.