Unsupported URL : theweathernetwork.com

Ramblin commented 5 years ago

Checklist

[ ] I'm reporting a broken site support issue
[ x] I've verified that I'm running youtube-dl version 2019.05.20
[x ] I've checked that all provided URLs are alive and playable in a browser
[ x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x ] I've searched the bugtracker for similar bug reports including closed ones
[ x] I've read bugs section in FAQ

Verbose log

richard@LinuxMint2019:~/Downloads$ youtube-dl -v --list-formats https://www.theweathernetwork.com/ca/videos/gallery/timelapse-over-ontarios-georgian-bay-doesnt-look-real-see-it/sharevideo/6040324276001/most_popular
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'--list-formats', u'https://www.theweathernetwork.com/ca/videos/gallery/timelapse-over-ontarios-georgian-bay-doesnt-look-real-see-it/sharevideo/6040324276001/most_popular']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.05.20
[debug] Python version 2.7.15rc1 (CPython) - Linux-4.15.0-50-generic-x86_64-with-LinuxMint-19.1-tessa
[debug] exe versions: ffmpeg 3.4.6, ffprobe 3.4.6
[debug] Proxy map: {}
[generic] most_popular: Requesting header
WARNING: Falling back on generic information extractor.
[generic] most_popular: Downloading webpage
[generic] most_popular: Extracting information
ERROR: Unsupported URL: https://www.theweathernetwork.com/ca/videos/gallery/timelapse-over-ontarios-georgian-bay-doesnt-look-real-see-it/sharevideo/6040324276001/most_popular
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/generic.py", line 2340, in _real_extract
    doc = compat_etree_fromstring(webpage.encode('utf-8'))
  File "/usr/local/bin/youtube-dl/youtube_dl/compat.py", line 2551, in compat_etree_fromstring
    doc = _XML(text, parser=etree.XMLParser(target=_TreeBuilder(element_factory=_element_factory)))
  File "/usr/local/bin/youtube-dl/youtube_dl/compat.py", line 2540, in _XML
    parser.feed(text)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1659, in feed
    self._raiseerror(v)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1523, in _raiseerror
    raise err
ParseError: not well-formed (invalid token): line 10, column 69
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 796, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 529, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/generic.py", line 3329, in _real_extract
    raise UnsupportedError(url)
UnsupportedError: Unsupported URL: https://www.theweathernetwork.com/ca/videos/gallery/timelapse-over-ontarios-georgian-bay-doesnt-look-real-see-it/sharevideo/6040324276001/most_popular

Description

Tried to download video(s) from theweathernetwork.com and get the "Unsupported URL" error message.

The video is shown in a frame on that page so I am not sure of that is part of the issue or if the site simply blocks youtube-dl download attempts.

exekutive commented 2 years ago

I would also like to use youtube-dl with this website. Does anybody know some good hacks? The browser developer panel shows the video is a brightcove videojs segmented mpeg video stream.

My log:

youtube-dl -v --list-formats --ignore-config  "https://www.theweathernetwork.com/ca/news/article/winter-tires-should-they-go-on-the-back-or" 
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '--list-formats', '--ignore-config', 'https://www.theweathernetwork.com/ca/news/article/winter-tires-should-they-go-on-the-back-or']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.06.06
[debug] Python version 3.9.6 (CPython) - macOS-11.6-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.4, ffprobe 4.4, rtmpdump 2.4
[debug] Proxy map: {}
[generic] winter-tires-should-they-go-on-the-back-or: Requesting header
WARNING: Falling back on generic information extractor.
[generic] winter-tires-should-they-go-on-the-back-or: Downloading webpage
[generic] winter-tires-should-they-go-on-the-back-or: Extracting information
ERROR: Unsupported URL: https://www.theweathernetwork.com/ca/news/article/winter-tires-should-they-go-on-the-back-or
Traceback (most recent call last):
  File "/usr/local/Cellar/youtube-dl/2021.6.6/libexec/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/Cellar/youtube-dl/2021.6.6/libexec/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/Cellar/youtube-dl/2021.6.6/libexec/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/Cellar/youtube-dl/2021.6.6/libexec/lib/python3.9/site-packages/youtube_dl/extractor/generic.py", line 3520, in _real_extract
    raise UnsupportedError(url)
youtube_dl.utils.UnsupportedError: Unsupported URL: https://www.theweathernetwork.com/ca/news/article/winter-tires-should-they-go-on-the-back-or

ghost commented 2 years ago

Find "master.m3u8" url from browser's dev tools and pass to youtube-dl, if you don't mind the hassle.

exekutive commented 2 years ago

@kikuyan that doesn't work

youtube-dl "https://manifest.prod.boltdns.net/manifest/v1/hls/v4/clear/1942203455001/e8ff5aad-d125-476c-8eeb-2f46f81a3260/10s/master.m3u8?fastly_token=NjE5ODY3YzRfZmM2MmE5YjU4ZmJlM2FlZjI3OTdmNGQ3OTQxNjY5YjdhZjliMGY5ZWNiOTMzMmIwY2ZmZDcwNTQ1NzgzNmVkZQ%3D%3D"
[generic] master: Requesting header
[generic] master: Downloading m3u8 information
ERROR: requested format not available

youtube-dl "https://manifest.prod.boltdns.net/manifest/v1/hls/v4/clear/1942203455001/e8ff5aad-d125-476c-8eeb-2f46f81a3260/10s/master.m3u8"
[generic] master: Requesting header
WARNING: Could not send HEAD request to https://manifest.prod.boltdns.net/manifest/v1/hls/v4/clear/1942203455001/e8ff5aad-d125-476c-8eeb-2f46f81a3260/10s/master.m3u8: HTTP Error 403: Forbidden
[generic] master: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>);

ghost commented 2 years ago

$ youtube-dl "https://manifest.prod.boltdns.net/manifest/v1/hls/v4/clear/1942203455001/d97dae86-b821-4831-bf66-8d353ce10373/10s/master.m3u8?fastly_token=NjE5ODhhNGJfMzUzMTJhNmFmMWU5ZWUwZjQzNTg2NGZkZTg2NjUxNmVjZGMwMjZlODAwNGVkMzgwOTkzODAwNjg2NmJkZDVmMA%3D%3D" \
> --ignore-config --hls-prefer-native        
[generic] master: Requesting header
[generic] master: Downloading m3u8 information
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 7
[download] Destination: master-master.f1673.mp4
[download] 100% of 10.74MiB in 00:06
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 7
[download] Destination: master-master.faudio-2-en__Main_.mp4
[download] 100% of 1.04MiB in 00:01
[ffmpeg] Merging formats into "master-master.mp4"
Deleting original file master-master.f1673.mp4 (pass -k to keep)
Deleting original file master-master.faudio-2-en__Main_.mp4 (pass -k to keep)
$ ffprobe master-master.mp4 
ffprobe version 4.4.1 Copyright (c) 2007-2021 the FFmpeg developers
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'master-master.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.76.100
  Duration: 00:01:00.63, start: 0.000000, bitrate: 1529 kb/s
  Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 1426 kb/s, 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
  Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 95 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]

exekutive commented 2 years ago

Adding --hls-prefer-native worked. Thank you.

Ramblin commented 2 years ago

I have just tested this solution and it still works:

Click on the "Share" icon (bottom right of the video wondow) you will see the URL and the embed code fields copy the code shown in the "embed code" field it will look something like

Take the URL from that code: 
    //players.brightcove.net/1942203455001/SkHslaO1M_default/index.html?videoId=6285468143001

Use that URL in youtube-dl (adding the http: as a prefix) You will get a long list of URL options For hi-def video, take the one that has "hls-{???}-0" (without the quotes and replacing the brackets and the content in the brackets) For good audio, take the one that has "dash-{long_UUID}-0" (without the quotes and replacing the brackets and the content in the brackets)

Use youtube-dl --list formats to see the list Use youtube-dl -f {video_file},{audio_file} to get hte two files Use ffmpeg to merge the two files ffmpeg -i "{video file}.mp4" -i "{audio file}.m4a" -c copy -map 0:v:0 -map 1:a:0 "{combined output file}.mp4"

Works for me

ytdl-org / youtube-dl