DavidCWGA
commented
6 years ago The Firefox extension "Video DownloadHelper" can successfully download these, if that helps at all.
Open MobyGamer opened 6 years ago
DavidCWGA
commented
6 years ago The Firefox extension "Video DownloadHelper" can successfully download these, if that helps at all.
RavenRabbit
commented
6 years ago I'd also appreciate support for that site in youtube-dl.
Video downloadhelper, might be able to download it, but does require you to install some kind of standalone application to enable the download.
Vangelis66
commented
6 years ago They are using AppleHLS streams for these archive programmes; you can use your browser's web console (actual name may vary depending on the browser brand, usually to be found inside Developer Tools) and filter for ".m3u8" (without the quotes); then use the master playlist URI to dump with yt-dl.
https://computer-literacy-project.pilots.bbcconnectedstudio.co.uk/2264a5e86a4aa2e8b36ed6acee9f6462
yields
https://computer-literacy-project.pilots.bbcconnectedstudio.co.uk/asset/video/2264a5e86a4aa2e8b36ed6acee9f6462/index.m3u8 hence
youtube-dl --console-title "https://computer-literacy-project.pilots.bbcconnectedstudio.co.uk/asset/video/2264a5e86a4aa2e8b36ed6acee9f6462/index.m3u8" --hls-prefer-native --hls-use-mpegts --no-part --fixup never -o "TheSiliconFactor_Se1Ep1_SoWhatsItAllAbout_198003191940.ts" =>
[generic] index: Requesting header
[generic] index: Downloading m3u8 information
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 237
[download] Destination: TheSiliconFactor_Se1Ep1_SoWhatsItAllAbout_198003191940.ts
[download] 1.7% of ~835.82MiB at 263.13KiB/s ETA 15:30(of course, you can amend cmd arguments to your liking...)
Enjoy 😄
RavenRabbit
commented
6 years ago Thanks for the hint that does indeed look very promising. The only issue I've got is that at least the one resulting file I tested so far seems to have some hiccups. It may be the player though.
I tried
EDIT: Nevermind that was probably due to the .ts file. I got better results using mkv (but that may just be my player):
youtube-dl --console-title "https://computer-literacy-project.pilots.bbcconnectedstudio.co.uk/asset/video/8000f792d8200143cf49695f14aeec57/index.m3u8" --hls-prefer-native --hls-use-mpegts --no-part --fixup never --prefer-ffmpeg --merge-output-format mkv -o "MakingtheMostoftheMicro_Se1Ep10_AttheEndoftheLine.mkv"
MobyGamer
commented
6 years ago Much thanks to those who discovered a fail-safe solution. I'm also finding that--merge-output-format mkv is working best for me (no stuttering in playback, seek working correctly). I'm not using prefer-hls and letting ffmpeg be the downloader.
Unfortunately, without proper URL support, there is zero metadata available to rename the result (they're all "index-index.*"). It would take many hours, likely riddled with human error, to rename all of the files to something meaningful. Therefore, I hope my original request will still be handled.
Vangelis66
commented
6 years ago Thanks for the hint that does indeed look very promising
Much thanks to those who discovered a fail-safe solution
You are both welcome @RavenRabbit & @MobyGamer :smile: ; it is unrealistic to expect the yt-dl devs to fully support all old and new websites with media content, so some DIY job is the approach I take most of the time; in their defence, they do support most of current streaming techniques: HTTP progressive, AdobeHDS, AppleHLS (AES-128, too!), MPEG-DASH (without encryption); all it takes is some detective work to identify .f4m/.m3u8/.mpd URIs...
List containing all the 267 videos/audios, if someone's interested and I didn't make any errors.
Much obliged for your hard work 👍 ; as you've discovered, there exists a simple pattern to formulate the master playlist URI for each programme: grab the episode's page URL and append /index.m3u8
that was probably due to the .ts file. I got better results using mkv
I'm also finding that--merge-output-format mkv is working best for me (no stuttering in playback, seek working correctly).
On my posted yt-dl cmd I've purposely chosen --hls-native and the MPEG-TS container for two reasons:
(and 3. My SmartTV can perfectly handle .ts files 😜 )
... But I did say:
(of course, you can amend cmd arguments to your liking...)
so with FFmpeg you can opt for (MP4 | Matroska) containers and have the AV file properly indexed, the reason behind
seek working correctly
Unfortunately, without proper URL support, there is zero metadata available to rename the result (they're all "index-index.*"). It would take many hours, likely riddled with human error, to rename all of the files to something meaningful.
... Such is life; you win some, you lose some... 😉
RavenRabbit
commented
6 years ago I did some more testing with the parameters and noticed that in some cases I got a warning during the download reading WARNING: index: malformed AAC bitstream detected.. By supplying --fixup detect_or_warn that got fixed and the videos got a few seconds longer. The shorter videos files also seemed to have issues when skipping to a certain location. In case of the audiofiles I noticed the file getting a lot longer. However, with the short tracks the playback continued at the "end".
I'm not sure which parameters are the best (and this results in only 4 unique files, because some combinations result in the exact same file)
youtube-dl --console-title "https://computer-literacy-project.pilots.bbcconnectedstudio.co.uk/asset/video/53007070caed58649b2eaec53e441326/index.m3u8" --hls-prefer-ffmpeg --no-part --fixup warn --prefer-ffmpeg --merge-output-format mkv -o "Timeshift S1E1 - Dial "B" for Britain- The Story of The Landline_ffmpeg_NO_mpegts_warn.mkv"
youtube-dl --console-title "https://computer-literacy-project.pilots.bbcconnectedstudio.co.uk/asset/video/53007070caed58649b2eaec53e441326/index.m3u8" --hls-prefer-ffmpeg --no-part --fixup detect_or_warn --prefer-ffmpeg --merge-output-format mkv -o "Timeshift S1E1 - Dial "B" for Britain- The Story of The Landline_ffmpeg_NO_mpegts_detect.mkv"
youtube-dl --console-title "https://computer-literacy-project.pilots.bbcconnectedstudio.co.uk/asset/video/53007070caed58649b2eaec53e441326/index.m3u8" --hls-prefer-native --hls-use-mpegts --no-part --fixup detect_or_warn --prefer-ffmpeg --merge-output-format mkv -o "Timeshift S1E1 - Dial "B" for Britain- The Story of The Landline_native_mpegts_detect.mkv"
youtube-dl --console-title "https://computer-literacy-project.pilots.bbcconnectedstudio.co.uk/asset/video/53007070caed58649b2eaec53e441326/index.m3u8" --hls-prefer-native --no-part --fixup detect_or_warn --prefer-ffmpeg --merge-output-format mkv -o "Timeshift S1E1 - Dial "B" for Britain- The Story of The Landline_native_NO_mpegts_detect.mkv"
# shorter results:
youtube-dl --console-title "https://computer-literacy-project.pilots.bbcconnectedstudio.co.uk/asset/video/53007070caed58649b2eaec53e441326/index.m3u8" --hls-prefer-ffmpeg --hls-use-mpegts --no-part --fixup detect_or_warn --prefer-ffmpeg --merge-output-format mkv -o "Timeshift S1E1 - Dial "B" for Britain- The Story of The Landline_ffmpeg_mpegts_detect.mkv"
youtube-dl --console-title "https://computer-literacy-project.pilots.bbcconnectedstudio.co.uk/asset/video/53007070caed58649b2eaec53e441326/index.m3u8" --hls-prefer-ffmpeg --hls-use-mpegts --no-part --fixup warn --prefer-ffmpeg --merge-output-format mkv -o "Timeshift S1E1 - Dial "B" for Britain- The Story of The Landline_ffmpeg_mpegts_warn.mkv"
youtube-dl --console-title "https://computer-literacy-project.pilots.bbcconnectedstudio.co.uk/asset/video/53007070caed58649b2eaec53e441326/index.m3u8" --hls-prefer-native --hls-use-mpegts --no-part --fixup warn --prefer-ffmpeg --merge-output-format mkv -o "Timeshift S1E1 - Dial "B" for Britain- The Story of The Landline_native_mpegts_warn.mkv"
youtube-dl --console-title "https://computer-literacy-project.pilots.bbcconnectedstudio.co.uk/asset/video/53007070caed58649b2eaec53e441326/index.m3u8" --hls-prefer-native --no-part --fixup warn --prefer-ffmpeg --merge-output-format mkv -o "Timeshift S1E1 - Dial "B" for Britain- The Story of The Landline_native_NO_mpegts_warn.mkv"At the moment I'm tempted to go for the --hls-prefer-ffmpeg --no-part --fixup warn --prefer-ffmpeg --merge-output-format mkv -o
But I'm not quite sure which one's the best...
MobyGamer
commented
6 years ago While I was able to retrieve what I needed, the actual uploaded video is, disappointingly, deinterlaced to 25p in some cases, and (confusingly) to 15 fps in others. It's a case of setting the wrong settings, and sadly they likely won't pull the tapes out all over again to do it correctly. What a shame.
What is the purpose of your issue?
This is a site request. The BBC has released all of their computing history television programs at https://computer-literacy-project.pilots.bbcconnectedstudio.co.uk/programmes but none of the playlist pages, nor the individual videos, are able to be grabbed with youtube-dl 2018.06.25. The error for all attempts is "Unsupported URL". The website is not geolocation-restricted (site is in the UK, I'm in the USA, video plays in the browser just fine).
Sample URLs:
Main page: https://computer-literacy-project.pilots.bbcconnectedstudio.co.uk/programmes
Show/playlist page for "The Silicon Factor": https://computer-literacy-project.pilots.bbcconnectedstudio.co.uk/a1293cad8666ac0cddec7bf5530c4f30
Individual show page (The Silicon Factor, "1. So What's It All about? BBC 2 Series 1 Episode 1") which plays a video perfectly in a browser: https://computer-literacy-project.pilots.bbcconnectedstudio.co.uk/2264a5e86a4aa2e8b36ed6acee9f6462
An attempt at the individual show page above results in:
e:>youtube-dl -v https://computer-literacy-project.pilots.bbcconnectedstudio.co.uk/2264a5e86a4aa2e8b36ed6acee9f6462 [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-v', 'https://computer-literacy-project.pilots.bbcconnectedstudio.co.uk/2264a5e86a4aa2e8b36ed6acee9f6462'] [debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252 [debug] youtube-dl version 2018.06.25 [debug] Python version 3.4.4 (CPython) - Windows-10-10.0.17134 [debug] exe versions: ffmpeg N-89931-g42323c3e3a, ffprobe N-83280-gcba4f0e [debug] Proxy map: {} [generic] 2264a5e86a4aa2e8b36ed6acee9f6462: Requesting header WARNING: Falling back on generic information extractor. [generic] 2264a5e86a4aa2e8b36ed6acee9f6462: Downloading webpage [generic] 2264a5e86a4aa2e8b36ed6acee9f6462: Extracting information [generic] 2264a5e86a4aa2e8b36ed6acee9f6462: Requesting header WARNING: Falling back on generic information extractor. [generic] 2264a5e86a4aa2e8b36ed6acee9f6462: Downloading webpage [generic] 2264a5e86a4aa2e8b36ed6acee9f6462: Extracting information ERROR: Unsupported URL: https://computer-literacy-project.pilots.bbcconnectedstudio.co.uk/embed/2264a5e86a4aa2e8b36ed6acee9f6462 Traceback (most recent call last): File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpkc931nsv\build\youtube_dl\YoutubeDL.py", line 792, in extract_info File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpkc931nsv\build\youtube_dl\extractor\common.py", line 501, in extract File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpkc931nsv\build\youtube_dl\extractor\generic.py", line 3269, in _real_extract youtube_dl.utils.UnsupportedError: Unsupported URL: https://computer-literacy-project.pilots.bbcconnectedstudio.co.uk/embed/2264a5e86a4aa2e8b36ed6acee9f6462
Also, not to put any pressure on anyone, but according to the BBC's "taster" program, these videos will only be available for 3 months, so getting a fix for this before September 27th 2018 would be humbly appreciated.