search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
131.3k stars 9.95k forks source link

ABC iView HTTP Error 404: Not found #24960

Closed xvids121 closed 3 years ago

xvids121 commented 4 years ago

youtube-dl reports HTTP:404 errors on iView video URLS. other sites work fine.

Tried all the options I can think of in a number of combinations.

--no c:\users\user1.cache dir found, even hidden or system. --ignore-config --deleted c:\users\user1\youtube-dl.conf --no-cache and --rm-cache-dir --config-location as a freshly created zero-sized file.

-U update switch

--no-geo-bypass --no-cache-dir

--cookies file --disabled firewall and tested. socket error reported so this s a different error and ∴ probably not related to firewall.

This used to work seamlessly with iView but something has changed. Is there a way to get more verbosity so I can troubleshoot which (if any settings have changed)?

C:\TEMP\aBC-IVIEW>youtube-dl --no-check-certificate --no-geo-bypass --ignore-config --no-cache-dir --verbose https://iview.abc.net.au/show/catalyst/series/20/video/SC1802H004S00 [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['--no-check-certificate', '--no-geo-bypass', '--ignore-config', '--no-cache-dir', '--verbose', 'https://iview.abc.net.au/show/catalyst/series/20/video/SC1802H004S00'] [debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252 [debug] youtube-dl version 2020.03.24 [debug] Python version 3.4.4 (CPython) - Windows-10-10.0.18362 [debug] exe versions: none [debug] Proxy map: {} [abc.net.au:iview] SC1802H004S00: Downloading JSON metadata [abc.net.au:iview] SC1802H004S00: Downloading webpage ERROR: Unable to download webpage: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\extractor\common.py", line 627, in _request_webpage File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\YoutubeDL.py", line 2238, in urlopen File "C:\Python\Python34\lib\urllib\request.py", line 470, in open File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response File "C:\Python\Python34\lib\urllib\request.py", line 508, in error File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default

ffmpeg version git-2020-04-22-2e38c63 Copyright (c) 2000-2020 the FFmpeg developers built with gcc 9.3.1 (GCC) 20200328 configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libsrt --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --disable-w32threads --enable-libmfx --enable-ffnvcodec --enable-cuda-llvm --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt --enable-amf libavutil 56. 42.102 / 56. 42.102 libavcodec 58. 80.100 / 58. 80.100 libavformat 58. 42.101 / 58. 42.101 libavdevice 58. 9.103 / 58. 9.103 libavfilter 7. 79.100 / 7. 79.100 libswscale 5. 6.101 / 5. 6.101 libswresample 3. 6.100 / 3. 6.100 libpostproc 55. 6.100 / 55. 6.100 Hyper fast Audio and Video encoder usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Use -h to get full help or, even better, run 'man ffmpeg'

ilike2burnthing commented 4 years ago

Remove --no-geo-bypass

Working here with:

ilike2burnthing commented 4 years ago

Similar issue https://github.com/ytdl-org/youtube-dl/issues/24680 - may be related to Python being on the system, given that's where the error appears in the log

xvids121 commented 4 years ago

Similar issue #24680 - may be related to Python being on the system, given that's where the error appears in the log

Disabled the windows python.exe stub in settings->apps->app execution aliases and uninstalled python. Same issue.

Also tried both --no-geo-bypass and normal mode. Perhaps this is related to SSL or some other settings? Is there a way to get more verbosity/information on specifically what youtube-dl is communicating to see why it is not found?

ilike2burnthing commented 4 years ago

--print-traffic, --write-pages, and -j would be your best options: https://github.com/ytdl-org/youtube-dl#verbosity--simulation-options

xvids121 commented 4 years ago

Looks like the main links are not working. I tried using with and without -g options for some of the .m3u8 files but am not sure which .m3u8 files to use.

End up getting an error. I have attached 2 files.
See SC1802H008S00https-_iview.abc.net.au_api_programs_catalyst_SC1802H008S00.dump for the main file.

See additional_information for output of what happens when trying grab video using .m3u8 files.

Not sure what else I can try at this point or what else I can test. Any ideas?

additional_information.txt

SC1802H008S00https-_iview.abc.net.au_api_programs_catalyst_SC1802H008S00.dump.txt

ilike2burnthing commented 4 years ago

I get the same errors using https://iview-vod.akamaized.net/prod/cla/classification_PG_2019/20191126150417/fairplay/master.m3u8

Available streams for https://iview.abc.net.au/show/catalyst/series/20/video/SC1802H004S00:

streams: {mpegdash: {,…}, fairplay: {,…}, hls: {,…}}
fairplay: {,…}
protected: true
sd: "https://iview-vod.akamaized.net/prod/sc1/SC1802H004S00CE1D1/20190219155946/fairplay/master.m3u8"
sd-low: "https://iview-vod.akamaized.net/prod/sc1/SC1802H004S00CE1D1/20190219155946/fairplay/master.m3u8"
hls: {,…}
protected: false
sd: "https://iviewhlsvod-vh.akamaihd.net/i/playback/_definst_/_video/sc1/SC1802H004S00CE1D1_20190219155946_,1500000,1000000,650000,500000,220000,.mp4.csmil/master.m3u8?subtitle_file_pattern=identifier_langcode&subtitle_identifier=SC1802H004S00CE1D1&subtitle_location=%2Fplayback%2Fcaptions%2Fprod%2Fsc"
sd-low: "https://iviewhlsvod-vh.akamaihd.net/i/playback/_definst_/_video/sc1/SC1802H004S00CE1D1_20190219155946_,650000,500000,220000,.mp4.csmil/master.m3u8?subtitle_file_pattern=identifier_langcode&subtitle_identifier=SC1802H004S00CE1D1&subtitle_location=%2Fplayback%2Fcaptions%2Fprod%2Fsc"
mpegdash: {,…}
protected: true
sd: "https://iview-vod.akamaized.net/prod/sc1/SC1802H004S00CE1D1/20190219155946/mpegdash/master.mpd"
sd-low: "https://iview-vod.akamaized.net/prod/sc1/SC1802H004S00CE1D1/20190219155946/mpegdash/master.mpd"

YTDL can only download the unprotected streams. For me, it uses https://iviewhlsvod-vh.akamaihd.net/i/playback/_definst_/_video/sc1/SC1802H004S00CE1D1_20190219155946_,1500000,1000000,650000,500000,220000,.mp4.csmil/master.m3u8 but then adds a different id string each time (reusing an old one fails), e.g.:

https://iviewhlsvod-vh.akamaihd.net/i/playback/_definst_/_video/sc1/SC1802H004S00CE1D1_20190219155946_,1500000,1000000,650000,500000,220000,.mp4.csmil/master.m3u8?null=0&id=AgBSDPQXAhVLHaj5ol7Orzjb1cu1pG7mZHsP7qtbKUoK+NjWQ7CBVnKOsO9VULx6phyj926EFEz+rw%3d%3d

For me this is entirely automated, so I'm not sure why it doesn't do that for you, or how to work around it.

Have you tried - https://github.com/ytdl-org/youtube-dl#developer-instructions Download, unzip, open cmd in folder, run python -m youtube_dl https://iview.abc.net.au/show/catalyst/series/20/video/SC1802H004S00

Do you have another computer you can try it on? One that hasn't had Python or youtube-dl on it? Or can you use Windows Sandbox?

xvids121 commented 4 years ago

Strangely I seem to get two errors and one warning, indicated in bold below:

Is the failed HEAD request causing the 405 and/or 403 errors? Is there a different mode for this? Will try a fresh install of Windows 7 in a VM.

C:\TEMP\aBC-IVIEW>youtube-dl --rm-cache-dir --ignore-config --no-geo-bypass https://iviewhlsvod-vh.akamaihd.net/i/playback/_definst_/_video/sc1/SC1802H004S00CE1D1_20190219155946_,1500000,1000000,650000,500000,220000,.mp4.csmil/master.m3u8 Removing cache dir C:\Users\bates/.cache\youtube-dl .. [generic] master: Requesting header WARNING: Could not send HEAD request to https://iviewhlsvod-vh.akamaihd.net/i/playback/_definst_/_video/sc1/SC1802H004S00CE1D1_20190219155946_,1500000,1000000,650000,500000,220000,.mp4.csmil/master.m3u8: HTTP Error 405: Method Not Allowed [generic] master: Downloading webpage ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

ilike2burnthing commented 4 years ago

I get the same warning and errors if I just use https://iviewhlsvod-vh.akamaihd.net/i/playback/_definst_/_video/sc1/SC1802H004S00CE1D1_20190219155946_,1500000,1000000,650000,500000,220000,.mp4.csmil/master.m3u8, it needs the id string after it.

I tried --add-header but nothing I used worked. Also played around with using ffmpeg directly, but to no avail.

Fingers crossed for your VM.

Drseevee commented 3 years ago

I don't seem to have any trouble with the command as written. youtube-dl version 2020.12.26 on Win 10 machine. Can you confirm that the issue still exists?

PS H:\Downloads> & '.\youtube-dl(1).exe' --no-check-certificate --no-geo-bypass --ignore-config --no-cache-dir --verbose https://iview.abc.net.au/show/catalyst/series/20/video/SC1802H004S00 [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['--no-check-certificate', '--no-geo-bypass', '--ignore-config', '--no-cache-dir', '--verbose', 'https://iview.abc.net.au/show/catalyst/series/20/video/SC1802H004S00'] [debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252 [debug] youtube-dl version 2020.12.26 [debug] Python version 3.4.4 (CPython) - Windows-10-10.0.18362 [debug] exe versions: ffmpeg git-2020-01-08-5bd0010, ffprobe git-2020-01-08-5bd0010 [debug] Proxy map: {} [abc.net.au:iview] SC1802H004S00: Downloading JSON metadata [abc.net.au:iview] SC1802H004S00: Downloading webpage [abc.net.au:iview] SC1802H004S00: Downloading m3u8 information [debug] Default format spec: bestvideo+bestaudio/best [debug] Invoking downloader on 'http://iviewhls-i.akamaihd.net/i/playback/_definst_/_video/sc1/SC1802H004S00CE1D1_20190219155946_,1500000,1000000,650000,500000,220000,.mp4.csmil/index_0_av.m3u8?null=0&id=AgByS2dGF8BstSqS6V+IrDoWNl9z6duGdxXAGsd0KRnECpOEwl9FQ75or2YYfkDo5DdOritYZ7Y8%2fw%3d%3d' [hlsnative] Downloading m3u8 manifest [hlsnative] Total fragments: 351 [download] Destination: Series 20 The Mindfulness Experiment-SC1802H004S00.mp4 [download] 100% of 519.98MiB in 03:25 [debug] ffmpeg command line: ffprobe -show_streams "file:Series 20 The Mindfulness Experiment-SC1802H004S00.mp4" [ffmpeg] Fixing malformed AAC bitstream in "Series 20 The Mindfulness Experiment-SC1802H004S00.mp4" [debug] ffmpeg command line: ffmpeg -y -loglevel "repeat+info" -i "file:Series 20 The Mindfulness Experiment-SC1802H004S00.mp4" -c copy -f mp4 "-bsf:a" aac_adtstoasc "file:Series 20 The Mindfulness Experiment-SC1802H004S00.temp.mp4"