[rtbf.be] New videos not available for download

sebastienadam commented 2 years ago

Checklist

[X] I'm reporting a broken site support
[X] I've verified that I'm running youtube-dl version 2021.12.17
[X] I've checked that all provided URLs are alive and playable in a browser
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones

Verbose log

$ youtube-dl --verbose "https://www.rtbf.be/auvio/detail_la-belgique-criminelle?id=2920492"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--verbose', u'https://www.rtbf.be/auvio/detail_la-belgique-criminelle?id=2920492']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 2.7.18 (CPython) - Linux-5.4.0-122-generic-x86_64-with-Ubuntu-20.04-focal
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7
[debug] Proxy map: {}
[RTBF] 2920492: Downloading webpage
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/rtbf.py", line 139, in _real_extract
    self._sort_formats(formats)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 1374, in _sort_formats
    raise ExtractorError('No video formats found')
ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

RTBF site seems to migrate to new streaming platform. It's no more possible to download new videos with youtube-dl (old ones are still available).

Some video examples that are not available for download:

PS: there are still open tickets related with RTBF web site, but they aren't related with this problem (they are older than this problem).

DimitriHautot commented 2 years ago

They are moving from freecaster.net to ericsson.net (for example: c9851ec-az-westeurope.fsly.cdn.ebsd.ericsson.net)

Media-data (JSON), and in particular URLs, are available for https://www.rtbf.be/auvio/detail_la-treve?id=2917912 , but not anymore for https://www.rtbf.be/auvio/detail_la-treve?id=2920003 .

DimitriHautot commented 2 years ago

New streaming service is using DASH / MPD technology. I'll try to understand how it works with other media providers. However, any guidance is more than welcome.

dirkf commented 2 years ago

From @PDupont815:

D:\youtube\Test>youtube-dl --verbose --no-check-certificate https://www.rtbf.be/auvio/detail_sting-l-electron-libre?id=2929134
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', '--no-check-certificate', 'https://www.rtbf.be/auvio/detail_sting-l-electron-libre?id=2929134']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: ffmpeg git-2020-02-20-56df829, ffprobe git-2020-02-20-56df829
[debug] Proxy map: {}
[RTBF] 2929134: Downloading webpage
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\YoutubeDL.py", line 815, in wrapper
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\YoutubeDL.py", line 836, in __extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\common.py", line 534, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\rtbf.py", line 139, in _real_extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\common.py", line 1374, in _sort_formats
youtube_dl.utils.ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

I used to download video from rtbf.be web site with youtube-dl. When I failed, usually I downloaded a new version, which solved the issue. Here it appears that I already have the latest. I you require credential to connect, don't hesitate to mail me. (I'm not sur if it's required). thank for you help. Pierre

dirkf commented 2 years ago

Please try yt-dlp with your account details. The yt-dlp extractor was updated in https://github.com/yt-dlp/yt-dlp/pull/4479 and has further proposed changes in https://github.com/yt-dlp/yt-dlp/pull/4671.

If yt-dlp works, we can back-port the new redbee.py.

sebastienadam commented 2 years ago

I have tried to download a video with yt-dlp, but I received an HTTP 403 error:

$ ./yt-dlp_linux --all-subs --write-all-thumbnails -u *** -p *** --no-progress -o "Zookeeper - Le Héros des animaux" -v "https://www.rtbf.be/auvio/detail_zookeeper-le-heros-des-animaux?id=2918869"
[debug] Command-line config: ['--all-subs', '--write-all-thumbnails', '-u', 'PRIVATE', '-p', 'PRIVATE', '--no-progress', '-o', 'Zookeeper - Le Héros des animaux', '-v', 'https://www.rtbf.be/auvio/detail_zookeeper-le-heros-des-animaux?id=2918869']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.08.14 [5593720] (linux_exe)
[debug] Python 3.10.5 (CPython 64bit) - Linux-5.4.0-122-generic-x86_64-with-glibc2.31 (glibc 2.31)
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, secretstorage-3.3.3, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[RTBF] Downloading JSON metadata
[debug] [RTBF] Extracting URL: https://www.rtbf.be/auvio/detail_zookeeper-le-heros-des-animaux?id=2918869
[RTBF] 2918869: Downloading webpage
[RTBF] 2918869: Downloading m3u8 information
[debug] Sort order given by extractor: res, proto
[debug] Formats sorted by: hasvid, ie_pref, res, proto, lang, quality, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] 2918869: Downloading 1 format(s): hls-4228
Deleting existing file Zookeeper - Le Héros des animaux.jpg
[info] Downloading video thumbnail 0 ...
[info] Writing video thumbnail 0 to: Zookeeper - Le Héros des animaux.jpg
[debug] Invoking hlsnative downloader on "https://rtbf-vod.fl.freecaster.net/vod/rtbf/geo/drm/R/R0a9He0BCL-aes/R0a9He0BCL-aes-audio_fre=128000-video_eng=3860000.m3u8"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 733
[download] Destination: Zookeeper - Le Héros des animaux
[download] Zookeeper - Le Héros des animaux.part-Frag1 has already been downloaded
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "yt_dlp/YoutubeDL.py", line 3169, in process_info
  File "yt_dlp/YoutubeDL.py", line 2893, in dl
  File "yt_dlp/downloader/common.py", line 442, in download
  File "yt_dlp/downloader/hls.py", line 354, in real_download
  File "yt_dlp/downloader/fragment.py", line 519, in download_and_append_fragments
  File "yt_dlp/downloader/fragment.py", line 355, in decrypt_fragment
  File "yt_dlp/downloader/fragment.py", line 345, in _get_key
  File "yt_dlp/YoutubeDL.py", line 3647, in urlopen
  File "urllib/request.py", line 525, in open
  File "urllib/request.py", line 634, in http_response
  File "urllib/request.py", line 563, in error
  File "urllib/request.py", line 496, in _call_chain
  File "urllib/request.py", line 643, in http_error_default
urllib.error.HTTPError: HTTP Error 403: Forbidden

elyse0 commented 2 years ago

@sebastienadam Can you play the video in the browser?

[debug] Command-line config: ['https://www.rtbf.be/auvio/detail_zookeeper-le-heros-des-animaux?id=2918869', '-v', '--cookies-from-browser', 'firefox']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.08.14 [55937202b]
[debug] Lazy loading extractors is disabled
[debug] Python 3.8.10 (CPython 64bit) - Linux-5.4.0-124-generic-x86_64-with-glibc2.29 (glibc 2.31)
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg n5.0.1-8-g11eff7739a-20220712 (setts), ffprobe n5.0.1-8-g11eff7739a-20220712, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, secretstorage-2.3.1, sqlite3-2.6.0, websockets-10.3
[Cookies] Extracting cookies from firefox
[debug] Extracting cookies from: "/home/amish/.mozilla/firefox/ckczlffq.default-release/cookies.sqlite"
[Cookies] Extracted 96 cookies from firefox
[debug] Proxy map: {}
[debug] [RTBF] Extracting URL: https://www.rtbf.be/auvio/detail_zookeeper-le-heros-des-animaux?id=2918869
[RTBF] 2918869: Downloading webpage
[RTBF] 2918869: Downloading m3u8 information
[debug] Sort order given by extractor: res, proto
[debug] Formats sorted by: hasvid, ie_pref, res, proto, lang, quality, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] 2918869: Downloading 1 format(s): hls-4228
[debug] Invoking hlsnative downloader on "https://rtbf-vod.fl.freecaster.net/vod/rtbf/geo/drm/R/R0a9He0BCL-aes/R0a9He0BCL-aes-audio_fre=128000-video_eng=3860000.m3u8"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 733
[download] Destination: Une comédie avec Kevin James [2918869].mp4
[download]   1.0% of ~2.84GiB at  122.66KiB/s ETA 30:08 (frag 7/733)

sebastienadam commented 2 years ago

I did some tests with the suggested commands, but without more success. Then, I realized that my VPN was not configured on Belgium. By selecting Belgium in my VPN, it worked (even without taking the browser cookies). It seems that the site restricts the download to Belgium only (even if, by being authenticated, you can watch videos via the browser from outside Belgium).

Thanks for your help.

dirkf commented 2 years ago

What would be the difference between playing a video in the browser and downloading it as far as the site's servers are concerned?

This observation is consistent with this hypothesis:

access is granted if the server thinks the client is in BE
otherwise the client has to be logged in
the server is not accepting the logged-in session cookies from the browser as evidence of yt-dl being logged-in, perhaps because the cookie value(s) are linked to other characteristics of the browser, such as UA.

elyse0 commented 2 years ago

I think what happened is that in yt-dlp there are two apis, Freecaster that doesn't need cookies (old videos still use this api) and Redbee that always needs them because it needs to get a token

The latest videos only use Redbee, i.e. https://www.rtbf.be/auvio/detail_jt-19h30?id=2930042 In both apis some videos are geo restricted but others not

In this case specifically, sebastienadam was trying to download an old video that uses the old api, that's why he didn't need cookies

dirkf commented 5 months ago

However:

3052 commented 3 weeks ago

its possible to download from RTBF:

msg=POST URL=https://login.auvio.rtbf.be/accounts.getJWT
msg=POST URL=https://exposure.api.redbee.live/v2/customer/RTBF/businessunit/Auvio/auth/gigyaLogin
msg=GET URL=https://bff-service.rtbf.be/auvio/v1.23/pages/emission/dans-les-yeux-d-arno-21024
msg=GET URL=https://exposure.api.redbee.live/v2/customer/RTBF/businessunit/Auvio/entitlement/2888868_6BA97Bb/play
msg=GET URL=https://C9851Ec-az-westeurope-fsly.cdn.redbee.live/rtbf/auvio/assets/2888868_6BA97Bb/materials/wIdpTS0ifA_6BA97Bb/vod-idx.ism/.mpd
msg=GET URL="https://C9851Ec-az-westeurope-fsly.cdn.redbee.live/rtbf/auvio/assets/2888868_6BA97Bb/materials/wIdpTS0ifA_6BA97Bb/vod-idx.ism/vod-idx-video=4000000.dash"
msg="0.21 %" size="1.76 megabyte" rate="1.76 megabyte/s"

dirkf commented 3 weeks ago

Similar effect without downloading at all: dd if=/dev/random of=rtbf_show.mp4 bs=1M count=838 ?

3052 commented 3 weeks ago

doubt it

Untitled

ytdl-org / youtube-dl