Implement library support for link shims and shorteners

[jonathanmayer]

Probably the easiest way to do this is create a new module (e.g., WebScience.Utilities.LinkResolution) that can check whether a link needs resolution (i.e., matches a known shim or shortener) and can resolve a link (either by parsing URL parameters or making a request to the URL and checking the redirect.

Some additional thoughts:

• We'll need a good dataset of link shims and shorteners. I've added some starting points as comments and shared a few relevant GitHub repos with @PranayAnchuri. Kevin Borgolte is going to try to find Bitly domains using our DNS data access.
• For redirect checking, a good option would be the Fetch API with the redirect option set to manual. That way we only load the resolved URL, not the entire page.
• We should support at least two levels of recursion, since usage of third-party link shorteners remains common on Twitter.

[PranayAnchuri]

First version of short urls using fetch api : https://github.com/citp/web-science/commit/13bde320dae763b814dcad16e65ea7b7b1b8ea6c

raw url is sent from the corresponding content script via message passing

PS: Response headers are not set to get the "Location" of redirected url when using manual mode in fetch API.

https://fetch.spec.whatwg.org/#atomic-http-redirect-handling

[jonathanmayer]

First version of short urls using fetch api : 13bde32

Please implement this in a new utility module, because we'll want to use the functionality in other study modules (e.g., SocialMediaSharing).

PS: Response headers are not set to get the "Location" of redirected url when using manual mode in fetch API.

https://fetch.spec.whatwg.org/#atomic-http-redirect-handling

A possible workaround: use the Fetch API to fire off the initial request, then use the WebRequest API to read the Location header in the response.

[jonathanmayer]

One more thought on this: we should make sure to gracefully handle the situation where the resolution requires an HTTP(S) request, and that request doesn't get a response.

[jonathanmayer]

Mulled over a bit more how to accomplish link resolution with the limited WebExtension APIs. I think the way to go is something like…

• Make the entry point for link resolution a function that returns a Promise.
• When a link resolution request comes in, remember the URL and the Promise resolve and reject functions in a key-value data structure where the URL is the key (e.g., an Object or a Map) and the value contains a collection of resolve and reject functions (since there might be multiple resolution requests for the same URL), plus a recursion depth (since we might need to recursively resolve).
• If this is a URL we aren't already resolving, use the Fetch API to fire off an HTTP(S) request. Set the manual redirect flag so that the browser won’t follow redirects, and use an AbortController to set a timeout.
  • If the Fetch call results in a redirect, ignore it, since we don't need to follow it.
  • If the Fetch call results in an error, invoke the relevant Promise reject function(s) and forget the resolution data.
• Use the WebRequest API to watch for background page HTTP(S) requests to known link shorteners / shims. On a matching request, check the Location response header to get the resolved URL.
  • If the Location does not match a URL shortener or shim, invoke the relevant Promise resolve function(s) and forget the resolution data.
  • If the Location matches a URL shortener or shim, and we haven't hit the recursion limit, recursively resolve the new URL.
  • Otherwise, invoke the relevant promise reject function(s) and forget the resolution data.

See https://github.com/citp/web-science/issues/20#issuecomment-557696819 for thoughts on lists of URL shorteners/shims.

[PranayAnchuri]

Implemented a utility module 'LinkResolution' that resolves a url.