normalize / canonicalize URLs to detect duplicates

[pdehaan]

I've briefly played with normalize-url and it seemed cool (although the logic seems simple enough for us to do ourselves).

We'd still need to run it through some filter that removes all those utm_* parameters (and any other parameters we may not want). I haven't found a node module which does that yet -- but it'd be nice to build and maintain for other unwanted tracking codes. Not sure if @nchapman was mentioning some Ruby module which did that so we can see what they filter against.

[nchapman]

https://github.com/postrank-labs/postrank-uri

[pdehaan]

Wow, that seems way better than my garbage module: https://github.com/pdehaan/strip-utm Basically all strip-utm does is parse a URL string, remove any utm_ query params and then call normalize-url so sort the remaining query params, and return the URL as a string.