ShishKabab
commented
6 years ago I've been doing some research on the Direct Linking feature and the code that already exists. So far we wave the server created by @blahah to be found in https://github.com/WorldBrain/direct-linking-server and a direct linking client cloned and modified from the Apache incubator-annotator repo.
The current direct linking server consists of a Node.js/Express server that seems to be meant to be directly exposed to the internet. It stores and retrieves annotations from a LevelDB database, without validation as a PoC. Upon access of a direct link, it's supposed to fetch the page the original page, inject some JS code to highlight the necessary text (package for injection is present, but not implemented yet), and return this to the user.
This approach seems to be the most practical for now, since the alternatives I can think of have issues: 1) Embed external content in iFrame: not possible, since security policies would prevent us from modifying the target's HTML 2) Retrieve page client-side via XHR: would run into trouble with CORS policies.
However, this approach still has some issues to be resolved: 1) The content is served from our domain, meaning that linked resources like images, JS and CSS need to be fixed 2) Scripts may want to commmunicate with their server, which might mean we need to replace URLs in scripts 3) Back-ends of pages should be configured to allow any requests from other sites, which will break pages that are 'too dynamic'
One solution would be to serve a stripped, read-friendly version of the page, linking to the original website. However, this could introduce other problems: 1) The content owners would probably not be happy with us storing their content on our servers (see Google News problems in Spain.) 2) If content goes viral, content owners want ad revenue which requires users to directly land on their pages
The current direct linking server could be relatively easily be adapted to run on AWS EB + S3 + Lambda + API Gateway to be easily deployed and dynamically scaled to handle peak traffic. Current things to do there would be: 1) Write script to be injected 1a) Write highlighting code to be shared between front- and back-end (#301) 1b) UI to encourage user towards Memex 2) Refactor to use pluggable back-end 3) Implement S3 backend for retrieving annotations 4) Write Lambda function to validate and upload annotation 5) Write deployment procedures
Before going further into the technical details however, let's discuss here how to solve these bigger design challenges :)
blackforestboi
In that context i thought about how we can rebrand "direct links". I find them not very descriptive.
How could we call it? Options: 
digi0ps
tilgovi
As discussed with @blahah and Charlie, we will pull forward the direct linking feature. It will enable users to link to specific quotes of an article, instead of just the url. It will also provide us with an ideal viral loop to aquire users.
From my understanding today, there are 3 parts to this:
1. Selection and anchoring
Similar to how hypothes.is does it. we would have a modal popping up, whenever a user selects a piece of text. It would show > "highlight" & "link to quote".
Also we could put these options in the context menu:
Technical details
@tilgovi,@bigbluehat & @Treora rebuilt the annotator library that powers the Hypothes.is annotation software. The code will enable us to create an anchor to a piece of text. It still awaits to become an npm package, but AFAIK it is functional. @Treora also built this nifty demo to showcase its capabilities.
2. Proxy server
The way I understood it, the proxy servers role is to inject the JS code into the loaded website a user is about to visit, so that the hightlights can be displayed. Our proxy would host a DB with all the quotes that have been made. Possibly already as a dat node we can replicate among several instances..
Important here is that the link created shows the social preview data of the link destination, not the proxy. To get what I mean with that, try creating a post on social media with this link: https://hyp.is/d6o9PM67EeeGWFs6SrbDHg/www.nytimes.com/2011/10/19/nyregion/new-york-planting-a-million-treestoo-many-some-say.html
At Hypothes.is will show preview images of hypothes.is and the resulting quote, but nothing about the nytimes article.
Earlier in the product life cycle of hypothes.is this routing to the information about the original article was still in place. Now it shows the red preview of hypothes.is. I find this undesirable, as the preview on social media gives important information about the article. We already will inject some "ad like" thing (see point 3) with the proxy. No need to annoy users even more or hinder their UX.
Technical details:
This is the library built at Hypothes.is that facilitated the proxy: https://github.com/hypothesis/via Might offer some inspiration.
3. Interruption for new users to expose them to direct link features
Whenever a user clicks the links for direct linking, there is a loading screen (also hypothes.is has it). it is an ideal place to expose users to the features they could have by installing the Memex. We can use it to notify non-memex-users that they just received a direct link and that they can do this as well, and also have other features, by downloading the memex extension. Existing Memex users should not have this in-between screen anymore and instead directly see the highlight.
Design of this screen is still TBD, most importantly, we need the functionality to work.
4. Optional for later: Search highlights, show them in results list. @poltak
It would be useful to be able to search for the words of the highlights a person makes and rank the words higher. If we don't get this in in the first iteration, we should definitely store the quotes in raw somewhere, so we can re-index them later and also display them alongside the pages in the results overview.
Known challenges: