Integrate URL shim and shortener resolution into LinkExposure studies

[jonathanmayer]

One possible path forward...

• Implement detection for URL shims and shorteners in the linkExposure.js content script.
• If there's a match, pass the URL to a background script for resolution.
• Resolve the URL in the background script.
• Pass the resolved URL back to the content script.

[PranayAnchuri]

First three steps are done : https://github.com/citp/web-science/commits/temp Short urls are obtained from - https://github.com/specious/check-bitly/blob/master/test/bitly-domains.txt and enriched with twitter and google see : https://github.com/citp/web-science/blob/temp/study/domains.js

[jonathanmayer]

see : https://github.com/citp/web-science/blob/temp/study/domains.js

Please put these shims and shorteners in a separate file associated with the new utility module.

Short urls are obtained from - https://github.com/specious/check-bitly/blob/master/test/bitly-domains.txt and enriched with twitter and google

There are a number of additional link shorteners to integrate. Please include as many as you can find. Some good starting points:

• Apache SpamAssassin
  • https://github.com/apache/spamassassin/blob/trunk/rulesrc/sandbox/pds/20_urlshort.cf
• HTTPS Everywhere
  • https://github.com/https-everywhere-bot/https-everywhere/blob/master/src/chrome/content/rules/Bitly_branded_short_domains.xml
• NYU SMaPP Lab
  • https://github.com/SMAPPNYU/urlExpander/blob/master/urlexpander/core/constants.py
• MISP
  • https://github.com/MISP/misp-warninglists/blob/master/lists/url-shortener/list.json
• LongUrl.org (now defunct)
  • https://web.archive.org/web/20160317000701/longurl.org/services
• Miscellaneous GitHub repositories
  • https://github.com/lrei/canonical_urls/blob/master/shorteners.txt
  • https://github.com/boutetnico/url-shorteners/blob/master/list.txt

[PranayAnchuri]

Done. LinkExposure module is integrated with LinkResolution.