net4people / bbs

Forum for discussing Internet censorship circumvention
3.44k stars 81 forks source link

List of copyright-related DNS blocks in Germany (CUII Lists) #387

Open wkrp opened 2 months ago

wkrp commented 2 months ago

The site https://cuiiliste.de/ claims to catalog a list of domain names blocked by DNS by certain ISPs in Germany at the behest of CUII (Clearingstelle Urheberrecht im Internet / Clearing Body for Copyright on the Internet). My understanding is that these domains are not blocked by law; rather CUII is an independent organization with which some ISPs have a private agreement (archive, English, English archive).

The methodology of obtaining the list is unclear. CUII publishes a list of its blocking recommendations, but the list is only website names or base domains, not the kind of list an ISP would need to implement the block technically. German Wikipedia says the source of the list of domains is a "leak" (with more discussion at the talk page). A TorrentFreak article (archive) implies it was created through trial DNS resolutions:

While there haven’t been any obvious errors that we’re aware of, access to information related to blocking would provide much needed transparency. With no information available from official sources, Damian, a 17-year-old German student, got together with some friends and embarked on a mission to fill in the blanks.

After sifting through the data and running domains though extensive DNS resolver tests, Damian launched CUIIliste.de, effectively lifting the blocking veil by exposing all URLs without redactions.

The CUII FAQ and Code of Conduct (1(b)) says the only technical means of blocking used is DNS blocking, so trial DNS resolution should be sufficient as a test. The self-test on the website actually just looks up the ASN of the web browser client and checks it against a list of known-affected ASNs.

There's a paginated search interface at https://cuiiliste.de/domains, but there is also a JSON API that is probably more convenient.

https://api.cuiiliste.de/blocked_domains api.cuiiliste.de_blocked_domains_20240823.json

Tabular list of domains from /blocked_domains API |first_blocked_on|domain| |---|--:| |2024-07-21|`astrotheque.net`| |2024-08-21|`www.astrotheque.net`| |2021-09-06|`bs.to`| |2024-08-21|`www.bs.to`| |2024-01-08|`buffsports.me`| |2024-08-21|`www.buffsports.me`| |2024-01-08|`buffstreams.sx`| |2024-08-21|`www.buffstreams.sx`| |2021-09-13|`burningseries.ac`| |2024-08-21|`www.burningseries.ac`| |2024-08-21|`www.burningseries.co`| |2024-08-21|`www.burningseries.sx`| |2024-07-21|`burningseries.tw`| |2024-08-21|`www.burningseries.tw`| |2024-08-21|`www.burningseries.vc`| |2024-07-21|`canna-power.to`| |2021-04-14|`canna.to`| |2021-05-10|`uu.canna.to`| |2022-06-07|`cine.to`| |2024-08-21|`www.cine.to`| |2023-08-07|`filmfans.org`| |2024-08-21|`www.filmfans.org`| |2024-03-04|`filmpalast.to`| |2024-08-21|`www.filmpalast.to`| |2024-07-21|`harleyquinnwidget.com`| |2024-07-21|`harleyquinnwidget.live`| |2024-07-21|`harleyquinnwidget.net`| |2024-08-21|`www.harleyquinnwidget.com`| |2023-01-23|`israbox.com`| |2024-07-21|`israbox-music.com`| |2024-07-21|`israbox-music.org`| |2024-07-21|`isrbx.com`| |2023-01-23|`isrbx.me`| |2024-07-21|`isrbx.net`| |2024-07-21|`american-football.jokerguide.com`| |2024-07-21|`athletics.jokerguide.com`| |2024-07-21|`aussie-rules.jokerguide.com`| |2024-07-21|`badminton.jokerguide.com`| |2024-07-21|`basketball1.jokerguide.com`| |2024-07-21|`basketball.jokerguide.com`| |2024-07-21|`beach-volley.jokerguide.com`| |2024-07-21|`boxing1.jokerguide.com`| |2024-07-21|`cycling.jokerguide.com`| |2024-07-21|`darts1.jokerguide.com`| |2024-07-21|`f1.jokerguide.com`| |2024-07-21|`football1.jokerguide.com`| |2024-07-21|`futsal.jokerguide.com`| |2024-07-21|`golf.jokerguide.com`| |2024-07-21|`handball.jokerguide.com`| |2024-07-21|`ice-hockey.jokerguide.com`| |2024-07-21|`jokerguide.com`| |2024-07-21|`mlb1.jokerguide.com`| |2024-07-21|`motorsport.jokerguide.com`| |2024-07-21|`nba3.jokerguide.com`| |2024-07-21|`nba.jokerguide.com`| |2024-07-21|`ncaab1.jokerguide.com`| |2024-07-21|`ncaaf.jokerguide.com`| |2024-07-21|`nfl2.jokerguide.com`| |2024-07-21|`nfl.jokerguide.com`| |2024-07-21|`nhl4.jokerguide.com`| |2024-07-21|`reddit.jokerguide.com`| |2024-07-21|`rugby2.jokerguide.com`| |2024-07-21|`snooker2.jokerguide.com`| |2024-07-21|`table-tennis.jokerguide.com`| |2024-07-21|`tennis.jokerguide.com`| |2024-07-21|`ufc1.jokerguide.com`| |2024-07-21|`volleyball2.jokerguide.com`| |2024-07-21|`waterpolo.jokerguide.com`| |2024-08-21|`www.jokerguide.com`| |2024-07-21|`jokerlivestream.net`| |2024-07-21|`jokerlivestream.org`| |2024-08-21|`www.jokerlivestream.org`| |2024-07-21|`jokerlivestream.vip`| |2024-08-21|`www.jokerlivestream.vip`| |2022-03-15|`kinos.to`| |2024-08-21|`ww4.kinos.to`| |2024-08-21|`ww15.kinos.to`| |2024-08-21|`ww17.kinos.to`| |2024-08-21|`ww19.kinos.to`| |2024-08-21|`www8.kinos.to`| |2024-08-21|`www12.kinos.to`| |2024-08-21|`www13.kinos.to`| |2024-08-21|`www14.kinos.to`| |2024-08-21|`www15.kinos.to`| |2024-08-21|`www17.kinos.to`| |2024-08-21|`www.kinos.to`| |2024-08-21|`www.kinox.am`| |2024-08-21|`www.kinox.bz`| |2024-08-21|`www.kinox.click`| |2024-08-21|`www3.kinox.click`| |2024-08-21|`www3.kinox.cloud`| |2024-08-21|`www.kinox.cloud`| |2024-08-21|`www.kinox.club`| |2024-07-21|`kinox.digital`| |2024-08-21|`www.kinox.digital`| |2024-08-21|`www3.kinox.digital`| |2024-07-21|`kinox.direct`| |2024-08-21|`www.kinox.direct`| |2024-07-21|`kinox.express`| |2024-08-21|`www.kinox.express`| |2024-08-21|`www3.kinox.express`| |2024-07-21|`kinox.fun`| |2024-08-21|`www.kinox.fun`| |2024-08-21|`www3.kinox.fun`| |2024-08-21|`www3.kinox.fyi`| |2024-08-21|`www.kinox.fyi`| |2024-07-21|`kinox.gratis`| |2024-08-21|`www.kinox.gratis`| |2024-08-21|`www3.kinox.gratis`| |2024-07-21|`kinox.io`| |2024-08-21|`www.kinox.io`| |2024-08-21|`www3.kinox.io`| |2024-07-21|`kinox.lol`| |2024-08-21|`www.kinox.lol`| |2024-08-21|`www3.kinox.lol`| |2024-07-21|`kinox.me`| |2024-08-21|`www.kinox.me`| |2024-08-21|`www3.kinox.me`| |2024-07-21|`kinox.mobi`| |2024-08-21|`www.kinox.mobi`| |2024-08-21|`www3.kinox.mobi`| |2024-07-21|`kinox.pub`| |2024-08-21|`www.kinox.pub`| |2024-08-21|`www3.kinox.pub`| |2024-08-21|`www3.kinox.sh`| |2024-08-21|`www.kinox.sh`| |2024-08-21|`www.kinox.space`| |2024-07-21|`kinox.sx`| |2024-08-21|`www.kinox.sx`| |2024-08-21|`www3.kinox.sx`| |2021-11-01|`kinox.to`| |2024-08-21|`ww4.kinox.to`| |2024-08-21|`ww7.kinox.to`| |2024-08-21|`ww8.kinox.to`| |2024-08-21|`ww11.kinox.to`| |2024-08-21|`ww15.kinox.to`| |2024-08-21|`ww16.kinox.to`| |2024-08-21|`ww17.kinox.to`| |2024-08-21|`ww18.kinox.to`| |2024-08-21|`ww19.kinox.to`| |2024-08-21|`www1.kinox.to`| |2024-08-21|`www2.kinox.to`| |2024-08-21|`www4.kinox.to`| |2024-08-21|`www5.kinox.to`| |2024-08-21|`www6.kinox.to`| |2024-08-21|`www7.kinox.to`| |2024-08-21|`www8.kinox.to`| |2024-08-21|`www9.kinox.to`| |2024-08-21|`www12.kinox.to`| |2024-08-21|`www13.kinox.to`| |2024-08-21|`www15.kinox.to`| |2024-08-21|`www16.kinox.to`| |2024-08-21|`www17.kinox.to`| |2024-08-21|`www18.kinox.to`| |2024-08-21|`www.kinox.to`| |2024-08-21|`www.kinox.tube`| |2024-07-21|`kinox.tv`| |2024-08-21|`www.kinox.tv`| |2024-08-21|`www3.kinox.tv`| |2024-07-21|`kinox.wtf`| |2024-08-21|`www.kinox.wtf`| |2024-08-21|`www.kinoz.co`| |2022-03-16|`kinoz.to`| |2024-08-21|`ww14.kinoz.to`| |2024-08-21|`ww15.kinoz.to`| |2024-08-21|`ww17.kinoz.to`| |2024-08-21|`ww18.kinoz.to`| |2024-08-21|`ww19.kinoz.to`| |2024-08-21|`www3.kinoz.to`| |2024-08-21|`www4.kinoz.to`| |2024-08-21|`www5.kinoz.to`| |2024-08-21|`www8.kinoz.to`| |2024-08-21|`www12.kinoz.to`| |2024-08-21|`www13.kinoz.to`| |2024-08-21|`www14.kinoz.to`| |2024-08-21|`www15.kinoz.to`| |2024-08-21|`www16.kinoz.to`| |2024-08-21|`www17.kinoz.to`| |2024-08-21|`www18.kinoz.to`| |2024-08-21|`www.kinoz.to`| |2024-07-21|`megakino.biz`| |2024-08-21|`www.megakino.biz`| |2024-07-21|`megakino.cab`| |2024-08-21|`www.megakino.cab`| |2024-02-27|`megakino.co`| |2024-08-21|`www.megakino.co`| |2024-07-21|`megakino.ink`| |2024-08-21|`www.megakino.ink`| |2024-07-21|`megakino.men`| |2024-08-21|`www.megakino.men`| |2024-07-24|`megakino.vin`| |2024-08-21|`www.megakino.vin`| |2024-07-21|`megakino.ws`| |2024-08-21|`www.megakino.ws`| |2021-07-27|`newalbumreleases.net`| |2024-07-21|`newerastreams.com`| |2024-08-21|`www.newerastreams.com`| |2021-05-31|`nsw2u.com`| |2024-08-21|`www.nsw2u.com`| |2024-07-21|`nsw2u.in`| |2024-07-21|`nsw2u.net`| |2024-08-21|`www.nsw2u.net`| |2024-08-21|`www.nsw2u.org`| |2021-11-12|`nsw2u.xyz`| |2024-03-01|`nswgame.com`| |2024-08-21|`www.nswgame.com`| |2024-03-04|`romslab.com`| |2024-08-21|`www.romslab.com`| |2024-01-08|`sci-hub.ru`| |2024-08-21|`www.sci-hub.ru`| |2024-01-08|`sci-hub.se`| |2024-08-21|`www.sci-hub.se`| |2024-01-08|`sci-hub.st`| |2024-08-21|`www.sci-hub.st`| |2023-08-07|`serienfans.org`| |2024-08-21|`www.serienfans.org`| |2024-07-21|`serienjunkies.biz`| |2024-08-21|`www.serienjunkies.biz`| |2024-07-21|`serienjunkies.eu`| |2024-08-21|`www.serienjunkies.eu`| |2024-07-21|`serienjunkies.info`| |2024-08-21|`www.serienjunkies.info`| |2022-04-04|`serienjunkies.org`| |2024-07-21|`new.serienjunkies.org`| |2024-08-21|`www.serienjunkies.org`| |2024-07-21|`serienjunkies.us`| |2024-08-21|`www.serienjunkies.us`| |2021-03-30|`serienstream.to`| |2024-08-21|`www.serienstream.to`| |2021-03-30|`serien.sx`| |2021-03-30|`s.to`| |2024-08-21|`www.s.to`| |2024-07-21|`streamkiste.club`| |2024-07-21|`streamkiste.fun`| |2024-07-21|`streamkiste.me`| |2024-07-21|`streamkiste.net`| |2024-07-21|`streamkiste.pro`| |2024-07-21|`streamkiste.pw`| |2024-07-21|`streamkiste.site`| |2024-07-21|`streamkiste.space`| |2021-09-09|`streamkiste.tv`| |2024-08-21|`www.streamkiste.tv`| |2022-12-05|`taodung.com`| |2024-07-21|`tazz.tv`| |2024-07-21|`tennis.stream`| |2024-07-21|`kinox.unblockit.black`| |2024-07-21|`kinox.unblockit.day`| |2024-07-21|`kinox.unblockit.llc`| |2024-07-21|`kinox.unblockit.mov`| |2024-07-21|`kinox.unblockit.ong`| |2024-07-21|`newalbumreleases.unblocked.co`| |2024-07-21|`newalbumreleases.unblockit.app`| |2024-07-21|`newalbumreleases.unblockit.bet`| |2024-07-21|`newalbumreleases.unblockit.blue`| |2024-07-21|`newalbumreleases.unblockit.buzz`| |2024-07-21|`newalbumreleases.unblockit.cam`| |2024-07-21|`newalbumreleases.unblockit.cat`| |2024-07-21|`newalbumreleases.unblockit.ch`| |2024-07-21|`newalbumreleases.unblockit.club`| |2024-07-21|`newalbumreleases.unblockit.day`| |2024-07-21|`newalbumreleases.unblockit.dev`| |2024-07-21|`newalbumreleases.unblockit.how`| |2024-07-21|`newalbumreleases.unblockit.ink`| |2024-07-21|`newalbumreleases.unblockit.ist`| |2024-07-21|`newalbumreleases.unblockit.kim`| |2024-07-21|`newalbumreleases.unblockit.li`| |2024-07-21|`newalbumreleases.unblockit.link`| |2024-07-21|`newalbumreleases.unblockit.ltd`| |2024-07-21|`newalbumreleases.unblockit.me`| |2024-07-21|`newalbumreleases.unblockit.name`| |2024-07-21|`newalbumreleases.unblockit.nz`| |2024-07-21|`newalbumreleases.unblockit.onl`| |2024-07-21|`newalbumreleases.unblockit.uno`| |2024-04-02|`ziperto.com`| |2024-08-21|`www.ziperto.com`|

The list is a bit strange: for example, it contains www1.kinox.to, www2.kinox.to, and www4.kinox.to, but not www3.kinox.to.

wkrp commented 2 months ago

Does anyone know what one of the DNS blocks looks like? Is the returned IP address one of the ISP's own, an IP address shared by all member ISPs, or a useless one like 127.0.0.1? Do you get 2 DNS responses (the one containing the block address and a real one) or just 1 (the one containing a block address). In other words, is it DNS response injection, or did the ISPs program their own resolvers to respond differently to queries for certain names? The provided circumvention instructions seem to recommend just turning on DNS over HTTPS.

mmmray commented 2 months ago

Except for things like captive portals, most DNS blocks in central and western europe are performed without any response injection. It's not just DoH that can bypass those (mostly copyright-related) restrictions but any change in DNS servers at all. In Germany specifically, ISPs face a surprising amount of liability for individual cases of copyright infringement or any misbehavior of their users. I am not sure if that might explain their behavior in the case of website blocking.

DAMcraft commented 2 months ago

hey, owner of cuiiliste.de here a CUII blocked site returns a CNAME going to notice.cuii.info grafik

interestingly enough, we observed that the biggest german ISP, Telekom, recently stopped returning that CNAME and now just returns NXDOMAIN. try it yourself: dig kinox.to @dns.telekom.de

info: the DNS resolver in the screenshot can only be accessed if 1&1 is your ISP.

DAMcraft commented 2 months ago

The list is a bit strange: for example, it contains www1.kinox.to, www2.kinox.to, and www4.kinox.to, but not www3.kinox.to

hey, it's an original list from the cuii, i have no clue why they're doing it like that. also, in some cases only www subdomains are blocked (like www.kinox.fyi, kinox.fyi itself isn't blocked) you can test a domain here: https://cuiiliste.de/probe?domain=kinox.to

since the cuii isn't blocking wildcard records, but instead are blocking all subdomains manually, we decided to include the www3 and other subdomains.

wkrp commented 2 months ago

a CUII blocked site returns a CNAME going to notice.cuii.info

Great, thanks for that information. With this, I was able to find examples of such DNS blocks in OONI explorer, like this one in Deutsche Telekom. It seems OONI doesn't record the CNAME record, but it does have the A record of 167.233.14.14. As expected, it leads to an eventual ssl_invalid_hostname error as the notice.cuii.info server doesn't have a certificate for sci-hub.se.

https://explorer.ooni.org/m/20240819220406.466321_DE_webconnectivity_1e0e3b6a896d55ff

{
  "input": "https://sci-hub.se/",
  "measurement_start_time": "2024-08-19 22:04:05",
  "probe_asn": "AS3320",
  "probe_cc": "DE",
  "probe_network_name": "Deutsche Telekom AG",
  "report_id": "20240819T220216Z_webconnectivity_DE_3320_n1_7rgS9FQZEtLxh5WU",
  "resolver_asn": "AS3320",
  "resolver_ip": "217.237.150.54",
  "resolver_network_name": "Deutsche Telekom AG",
  "test_keys": {
    "queries": [
      {
        "answers": [
          {
            "asn": 24940,
            "as_org_name": "Hetzner Online GmbH",
            "answer_type": "A",
            "ipv4": "167.233.14.14",
            "ttl": null
          }
        ],
        "engine": "system",
        "failure": null,
        "hostname": "sci-hub.se",
        "query_type": "A",
        "resolver_hostname": null,
        "resolver_port": null,
        "resolver_address": "",
        "t": 0.019840731,
        "tags": null
      }
    ],
    "dns_experiment_failure": null,
    "dns_consistency": "inconsistent",
    "http_experiment_failure": "ssl_invalid_hostname",
  },
  "test_name": "web_connectivity",
}

Looking at the past month of measurements, about half are showing as Anomaly.

https://explorer.ooni.org/chart/mat?test_name=web_connectivity&axis_x=measurement_start_day&since=2024-07-25&until=2024-08-25&time_grain=day&probe_cc=DE&domain=sci-hub.se

Web Connectivity Test, sci-hub.se, Germany, from 2024-07-25 to 2024-08-23. There are between 30 and 80 measurements per day. About half are OK in green and half Anomaly in orange, with a small number of Failure in gray.