search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.42k stars 10.04k forks source link

NRK gives Unable to download JSON metadata: <urlopen error [Errno 51] Network is unreachable> #29896

Closed NoLooseEnds closed 2 years ago

NoLooseEnds commented 3 years ago

Checklist

Verbose log

youtube-dl -v -s --no-geo-bypass -i https://tv.nrk.no/serie/fantus-og-maskinene/sesong/1
youtube-dl
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '-s', '--no-geo-bypass', '-i', 'https://tv.nrk.no/serie/fantus-og-maskinene/sesong/1']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.06.06
[debug] Python version 3.9.6 (CPython) - macOS-11.5.2-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.4, ffprobe 4.4, rtmpdump 2.4
[debug] Proxy map: {}
[NRKTVSeason] fantus-og-maskinene/1: Downloading season JSON
ERROR: Unable to download JSON metadata: <urlopen error [Errno 51] Network is unreachable> (caused by URLError(OSError(51, 'Network is unreachable')))
  File "/usr/local/Cellar/youtube-dl/2021.6.6/libexec/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/Cellar/youtube-dl/2021.6.6/libexec/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 2288, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 555, in error
    result = self._call_chain(*args)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 747, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 517, in open
    response = self._open(req, data)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/youtube-dl/2021.6.6/libexec/lib/python3.9/site-packages/youtube_dl/utils.py", line 2735, in https_open
    return self.do_open(functools.partial(
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 1349, in do_open
    raise URLError(err)

Description

I've had this issue for months now, and I can't seem to figure out what the issue is. I've tried different machines (macos and Linux), with and without VPN. Different DNS servers (mostly using my own pihole).

Download specific episodes work ie. youtube-dl --no-geo-bypass -i https://tv.nrk.no/serie/fantus-og-maskinene/sesong/1/episode/1 – so the issue is downloading whole season json.

(Tried with and without --no-geo-bypass flag)

Seems like there could be an issue with me having lyse fiber? I see that pinging tv.nrk.no gives this result:

ping tv.nrk.no
PING a1931.dscr.akamai.net (81.166.122.16) 56(84) bytes of data.
64 bytes from 16.81-166-122.customer.lyse.net (81.166.122.16): icmp_seq=1 ttl=59 time=9.48 ms
64 bytes from 16.81-166-122.customer.lyse.net (81.166.122.16): icmp_seq=2 ttl=59 time=9.62 ms
^C
--- a1931.dscr.akamai.net ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 9.482/9.548/9.615/0.066 ms

Running 

 dig @8.8.8.8 tv.nrk.no

; <<>> DiG 9.16.1-Ubuntu <<>> @8.8.8.8 tv.nrk.no
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33476
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;tv.nrk.no.                     IN      A

;; ANSWER SECTION:
tv.nrk.no.              1       IN      CNAME   tv.nrk.no.edgesuite.net.
tv.nrk.no.edgesuite.net. 21591  IN      CNAME   a1931.dscr.akamai.net.
a1931.dscr.akamai.net.  11      IN      A       23.36.76.89
a1931.dscr.akamai.net.  11      IN      A       23.36.76.210

;; Query time: 4 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Sep 01 11:13:08 CEST 2021
;; MSG SIZE  rcvd: 142
dig @1.1.1.1 tv.nrk.no

; <<>> DiG 9.16.1-Ubuntu <<>> @1.1.1.1 tv.nrk.no
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31268
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;tv.nrk.no.                     IN      A

;; ANSWER SECTION:
tv.nrk.no.              1       IN      CNAME   tv.nrk.no.edgesuite.net.
tv.nrk.no.edgesuite.net. 14825  IN      CNAME   a1931.dscr.akamai.net.
a1931.dscr.akamai.net.  11      IN      A       81.166.122.16
a1931.dscr.akamai.net.  11      IN      A       81.166.122.35

;; Query time: 4 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Wed Sep 01 11:12:59 CEST 2021
;; MSG SIZE  rcvd: 139
dig tv.nrk.no

; <<>> DiG 9.16.1-Ubuntu <<>> tv.nrk.no
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26750
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;tv.nrk.no.                     IN      A

;; ANSWER SECTION:
tv.nrk.no.              5       IN      CNAME   tv.nrk.no.edgesuite.net.
tv.nrk.no.edgesuite.net. 14746  IN      CNAME   a1931.dscr.akamai.net.
a1931.dscr.akamai.net.  4       IN      A       81.166.122.35
a1931.dscr.akamai.net.  4       IN      A       81.166.122.16

;; Query time: 4 msec
;; SERVER: 10.0.0.1#53(10.0.0.1)
;; WHEN: Wed Sep 01 11:14:18 CEST 2021
;; MSG SIZE  rcvd: 139

Looks like google DNS resolves differently than cloudflare dns (and my pihole - but that uses cloudflare as upstream dns). I did try to manually set the DNS to 8.8.8.8, flush dns, verified it resolved to the 23.36.76.89, but got the same result.

Any input?

ghost commented 3 years ago

This is resolved in git master, or 57b9a4b4c6cf2580b5007db78bd333a9a237fd47.