begin the notification service and app v2 initiative

[dmitshur]

I'm very used to visualizing notifications from GitHub (and lately, Gerrit) in the form of threads. Each issue/PR/CL with new activity shows up as a single entry, grouped by repo:

Notification Threads Screenshot


About 2 years ago, I had an idea. What if I could visualize each individual comment/event as a standalone notification, on a single page that streams them in real-time as they come in? The goal would be to provide a little context without having to open the thread. Twitter does something similar for its notifications UI.

About a year ago, I decided to give it a shot and try to build a prototype on my personal site. It has taken a while, but by now I have something functional that I'm happy with. It's far from finished, but I use and rely on it every day, and it saves me time having to deal with notification emails or to check GitHub and Gerrit manually.

The notifications are streamed to the frontend, written in Go and compiled to WebAssembly, which displays them when the notification stream page is open. The UI is very simple and can be improved:

For both GitHub and Gerrit, the new notification service v2 implementation combines various sources of information to emit the final notifications with as little delay as possible.

For GitHub, these sources are used:

• email notifications
• GitHub REST API v3
• GitHub GraphQL API v4

And for Gerrit:

• email notifications
• Gerrit REST API

Additional work is done to unify/de-duplicate the events from multiple sources, and to emit self-activity events as well.

Development, iteration and testing was done on a feature branch. A lot of WIP commits with brief messages were squashed along the way while I was rebasing the feature branch, and so all I have by now is the following rough combined commit message log from development:

Working Log

``` WIP: beginning+16.5 of activity v2 (notification v2 + events + mail) also fix internal panic internal/exp/service/notification/githubapi: implement MarkNotificationRead include inline comments in comment body WIP: improve display of notifivation stream; use hacky import path prefix WIP 2: start using state.Review directly WIP: also do MarkRead from v1 Because we're not yet tracking all threads... E.g., on initial fetch, no pagination is being done. Need to fix that, then this can be removed. Look without cache right away. There's no cache for this stuff in v2, so there's no point in trying to use it. WIP: display change body when new change opened WIP: start using notification service v2 everywhere WIP: battle with duplicate GH and Gerrit notifications also fix another internal panic WIP: improve issue/change notifications rendering with empty body internal/exp/service/activity/gerrit: detect direct mentions internal/exp/service/activity/gerrit: save/load lastReadAt to/from disk So it persists across server restarts. fix bug internal/exp/service/notification/http{client,handler,route}: add render /notificationsv2 on frontend instead of backend add ?debug query parameter to show debug info CHK: StreamNotifications returns channel CHK: before MarkRead button WIP: make MarkRead button functional WIP: mark all notifications in thread as read WIP: use a timeout of 45 seconds for git operations WIP: proxy experiment WIP: enable module proxy server WIP: clean up module proxy server internal/exp/service/notification/githubapi: don't show closed event when PR is merged WIP: stream mark-read updates WIP: compute import paths for events TODO: decide on prefixtitle package import path, split away other notifv2 changes, clean up and review, then rebase onto master. WIP: sort diskSegments, add /notificationsv2/status page WIP: parse prefixed titles properly for notifications too WIP: more red logo (add to frontend) WIP: notifv2b WIP: clean up, remove no longer used services WIP: start polling unread notifs render tabnav in /notificationsv2 app WIP: add missing review, event ID, and HTMLURL handling WIP: add Show Notifications checkbox; include import path prefix in v2tov1 adapter WIP: support PR open events in mail.go, support review requested events WIP: add a copy of notificationsapp (v1) WIP: start using internal app/notificationsv2 WIP: render tabnav in /notifications app too WIP: rename app/notificationsv2 -> app/notifications factor out FS watchers out of main.go into newDirWatcher notificationsv2.go only show the tabnav with the streaming tab to site admins while notificationsv2 is still WIP implement notification/httpclient CountNotifications method internal/exp/service/activity/github: handle invalid Message-ID values WIP: move notificationsv2 app into internal/exp/app/notifications move WebAssembly compilation into virtual filesystem replace notificationsapp.BaseURIContextKey with constructor parameter have notificationsApp.ServeHTTP return error, check if os.IsPermission(err) is true, etc. WIP: integrate stream page into notifications app v2 more WIP: merge StreamHeadPre into HeadPre; use HTML5 doctype HTML5 doctype is to fix table not inheriting font-family from body by default. Remove code { font-family: "Go Mono"; } for now, because it doesn't seem to be used at this time. Can be re-added when needed. add notificationfs.DevNull, delete commented out MarkAllNotificationsRead WIP: merge StreamBaseURL into BaseURL clean up notifications app v2 a bit WIP: unify notifs app v2 frontend WIP: make progress on unifying notifs app v2 WIP: wrap up unifying notifs app v2 for now WIP: consolidate more notifs app v2 code, avoid use of dom/v2 for now because binary too big WIP: clean up and unify frontend client WIP: take care of XXXs, catch panics, review permission checks final pass over documentation and other finishing touches ```

The goal of this PR is to clean up this work and merge it into master branch, so I don't have to keep carrying the feature branch. Code that isn't fully finished is placed in exp directories, which are for packages that are "experimental or under development."

Yay, finally! 🎉 This is good enough that I can leave it as is and work on other things too. ✅