search

ooni / probe

OONI Probe network measurement tool for detecting internet censorship
https://ooni.org/install
BSD 3-Clause "New" or "Revised" License
751 stars 142 forks source link

websites: 301-based websites blocking #2079

Open yanmaani opened 2 years ago

yanmaani commented 2 years ago

(I'm not sure if this is an issue with the lists or with the explorer, but this MAY be a duplicate of citizenlab/test-lists#940.)

If a domain is accessed over HTTP, and it responds by 301'ing to a different domain, that seems like some indication of blocking, or at least of the site owner wanting to fake it (see: Daily Mail incident).

I think it may be prudent to detect cases where a domain, accessed over HTTP, responds by a 301/302 to a different domain that isn't a subdomain of the initial origin, and flag those in some way:

Here's two examples: NL - GET http://rt.com -> GET https://freedom.nl/landingpage/russia-today-en-sputnik-zijn-geblokkeerd-op-last-van-het-om UA - GET http://rt.com -> GET https://cdn.kyivstar.ua/sites/default/files/lock_sites/index.html

Neither freedom.nl nor cdn.kyivstar.ua are subdomains of rt.com.

bassosimone commented 2 years ago

Thank you for letting us know about this! I am going to flag this issue as a data quality issue and add it to the list of actions to be considered while redesigning our network measurement methodology.