search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.03k stars 10.01k forks source link

DRTV not working #30530

Open Chemical-Nils opened 2 years ago

Chemical-Nils commented 2 years ago

Checklist

Verbose log

[~] # youtube-dl --version
2021.12.17
[~] # youtube-dl --verbose https://www.dr.dk/drtv/episode/hitlers-danske-soldater_208769
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--verbose', u'https://www.dr.dk/drtv/episode/hitlers-danske-soldater_208769']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 2.7.13 (CPython) - Linux-5.10.60-qnap-x86_64-with-glibc2.2.5
[debug] exe versions: ffmpeg 3.3.6, ffprobe 3.4.2-static
[debug] Proxy map: {}
[drtv] hitlers-danske-soldater_208769: Downloading webpage
[drtv] 00922016210: Downloading video JSON
[drtv] 00922016210: Downloading m3u8 information
[drtv] 00922016210: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://drod05t-vh.akamaihd.net/p/all/token/download/3d/5fe9c10eaa5a612714959f3d/Hitlers-danske-soldate_d196482fdd2b432da4deb5862dd037bc_4300.mp4'
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1976, in process_info
    success = dl(filename, info_dict)
  File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1915, in dl
    return fd.download(name, info)
  File "/usr/bin/youtube-dl/youtube_dl/downloader/common.py", line 366, in download
    return self.real_download(filename, info_dict)
  File "/usr/bin/youtube-dl/youtube_dl/downloader/http.py", line 351, in real_download
    establish_connection()
  File "/usr/bin/youtube-dl/youtube_dl/downloader/http.py", line 116, in establish_connection
    raise err
HTTPError: HTTP Error 403: Forbidden

Description

WRITE DESCRIPTION HERE Has been working excellent for DRTV untill now. Hope that it is a simple issue to solve.

dirkf commented 2 years ago

The page offers formats like Download-n, which give the above result for me, and also HLS-n, which can be downloaded successfully (also -SpokenSubtitles- versions of both).

Possibly DRTV wants you to have logged in for the first type: more research needed from DRTV users.

You can select only HLS formats using -f '(bestvideo+bestaudio/best)[format_id^=?HLS]', although for this page only combined formats are offered so -f 'best[format_id^=HLS]' would do. (Windows: '' -> "")

kgbfriis commented 2 years ago

I am confirming this bug. This site - dr.dk - does not require login. However, some videos cannot be downloaded.

ERROR: Unsupported URL: https://www.dr.dk/drtv/saeson/pigerne-fra-berlin_258101

Commend line used to reproduce error:

youtube-dl --verbose -f 'best[format_id^=HLS]' https://www.dr.dk/drtv/saeson/pigerne-fra-berlin_258101

Would appreciate any help.

dirkf commented 2 years ago

That URL format isn't known to the extractor, though. You're asking for the series page (saeson) instead of the show page(se), although the series page also contains a player for the active episode (S1E1 when I looked). Either https://www.dr.dk/drtv/se/pigerne-fra-berlin_258108 or https://www.dr.dk/drtv/episode/pigerne-fra-berlin_258108 are valid URLs for S1E1. But those Berlin gals are only for Denmark (and Greenland).

The extractor could be extended to get a playlist from the series page, or the active video if --no-playlist is given.

For OP's page, possibly DRTV is geo-restricting the full downloads but not the adaptive streaming media? That would explain why I see the same from UK.

kgbfriis commented 2 years ago

@dirkf Thank you for the quick response. I am not so familiar with how youtube-dl works technically so I am not sure what you are saying exactly.

Does it mean that youtube-dl will never be able to download this show (and many others from dr.dk since this was just one example), or did I do anything wrong?

Thanks for your help.

dirkf commented 2 years ago

As long as you're in the DRTV area, just use the individual episode (/episode/) or player (/se/) pages, as in the examples I quoted for S1E1. There are only 9 for Pigerne.

Eventually I may be able to roll out my modified DRTV extractor that understands /saeson/ pages.

Chemical-Nils commented 2 years ago

I have looked a little into this using Fiddler & Chrome: The main url https://www.dr.dk/drtv/episode/hitlers-danske-soldater_208769 ==> causes an getting an url to a master playlist

https://drod05t-vh.akamaihd.net/i/all/clear/download/3d/5fe9c10eaa5a612714959f3d/Hitlers-danske-soldate_d196482fdd2b432da4deb5862dd037bc_,300,512,1000,2200,3000,4300,.mp4.csmil/master.m3u8?cc1=name=Fremmedsprogstekster~default=yes~forced=no~lang=da~uri=https://drod05t-vh.akamaihd.net/p/allx/clear/download/3d/5fe9c10eaa5a612714959f3d/subtitles/Foreign-17913845-20c1094f-1cdb-4f61-8208-155e9c768888/playlist.m3u8&cc2=name=Dansk~default=no~forced=no~lang=da~uri=https://drod05t-vh.akamaihd.net/p/allx/clear/download/3d/5fe9c10eaa5a612714959f3d/subtitles/Foreign_HardOfHearing-17913845-20c1094f-1cdb-4f61-8208-155e9c768888/playlist.m3u8 ==> download of a master.m3u8 playlist

This url can be truncated and still download the master.m3u8: https://drod05t-vh.akamaihd.net/i/all/clear/download/3d/5fe9c10eaa5a612714959f3d/Hitlers-danske-soldate_d196482fdd2b432da4deb5862dd037bc_,300,512,1000,2200,3000,4300,.mp4.csmil/master.m3u8

and can be truncated even further: https://drod05t-vh.akamaihd.net/i/all/clear/download/3d/5fe9c10eaa5a612714959f3d/Hitlers-danske-soldate_d196482fdd2b432da4deb5862dd037bc_4300.mp4.csmil/master.m3u8

master.m3u8 appear to be a list of bandwidth dependant url's to different index_X_av.m3u8 playlists which can be played in vlc player

In conclusion i guess the ".csmil/master.m3u8" should be appended to the download url and the resulting master.m3u8 used for the download of the .ts files.

Hope this helps in resolving the issue Nils

dirkf commented 2 years ago

Your issue was that download URLs like this, whose id is Download-4300,

https://drod05t-vh.akamaihd.net/p/all/token/download/3d/5fe9c10eaa5a612714959f3d/Hitlers-danske-soldate_d196482fdd2b432da4deb5862dd037bc_4300.mp4

gave 403.

The M3U8 playlists can be extracted OK and give the HLS streams, which can be downloaded.

dirkf commented 2 years ago

The Download-... formats give me 403 all the time. If that also happens in-region, we must be missing some additional header, cookie, or similar that's required to use them.

If these formats never work, we should strip them out.

If they sometimes work, we can check them when extracting, but it adds some time to each extraction.

DjRicko commented 2 years ago

It seems to be able to download from dr.dk now, but it only downloads partial episodes in some cases: https://www.dr.dk/drtv/episode/dolph-and-wulff_150435 (20 sec long clip and not the full length episode)