Closed alexhaydock closed 5 years ago
We'd already deduplicated lots of case-different domains.
Behind the scenes, input URLs do get case-folded. The scheme and hostname parts of the URL are case-insensitive, but the path and query string are not.
I forgot to filter out the old case variants when I was populating the lists.
I've eliminated a lot of the case-duplicates from the lists. There were 500 or so across all of the lists, so I don't think it will have had a huge statistical impact.
It will take a bit more time to get it completely cleaned up and to make sure that we have test results for all the canonical lower-case URLs.
That should have sorted it. Looks like we're due for another de-duping session in the main URLs list.
I've eliminated a lot of the case-duplicates from the lists. There were 500 or so across all of the lists, so I don't think it will have had a huge statistical impact.
Thanks Dan. I'll make sure the stats in the report get updated appropriately anyway, just in case.
Can we close this now?
I believe this can be closed, yes.
Some sites seem to appear more than once (particularly in lists) with capital letters in them:
https://www.blocked.org.uk/site/http://gay.com https://www.blocked.org.uk/site/http://Gay.com
Judging by the fact that the "Last Checked" date of these two entries available on the frontend is in sync with each other, it seems like the actual backend entry for the site in the database might be the same one, so I'm not sure if these "ghost" sites with odd capitalisation that appear on the frontend are actually affecting our statistics in the backend, but it's worth getting rid of them anyway.
I suppose since we're dealing with domains it makes sense to force everything to be lowercase?