Open sylvinus opened 8 years ago
There are many simple things we could do to improve our normalized URLs and avoid duplicates:
PHPSESSID
utm_*
Some good ideas there: https://github.com/iipc/webarchive-commons/tree/master/src/main/java/org/archive/url https://github.com/rajbot/surt/tree/master/surt
Code for this should be done in https://github.com/commonsearch/cosr-back/blob/master/cosrlib/url.py. Exhaustive unit tests would be great!
There are many simple things we could do to improve our normalized URLs and avoid duplicates:
PHPSESSID
,utm_*
, ..)Some good ideas there: https://github.com/iipc/webarchive-commons/tree/master/src/main/java/org/archive/url https://github.com/rajbot/surt/tree/master/surt
Code for this should be done in https://github.com/commonsearch/cosr-back/blob/master/cosrlib/url.py. Exhaustive unit tests would be great!