404 error on certain imgur URLs

groszdaniel commented 3 years ago

Checklist

[x] I'm reporting a broken site support
[x] I've verified that I'm running youtube-dl version 2021.06.06
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar issues including closed ones

Verbose log

[debug] System config: [] [debug] User config: ['--default-search', 'auto_warning', '--ignore-errors', '--add-metadata', '--all-subs', '--convert-subs=srt', '--embed-subs'] [debug] Custom config: [] [debug] Command-line args: ['-v', 'https://imgur.com/a/XlRUIpM'] [debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8 [debug] youtube-dl version 2021.06.06 [debug] Python version 3.8.10 (CPython) - Linux-5.12.10-1-default-x86_64-with-glibc2.2.5 [debug] exe versions: ffmpeg 4.4, ffprobe 4.4, rtmpdump 2.4 [debug] Proxy map: {} [imgur:album] XlRUIpM: Downloading JSON metadata ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. File "/bin/youtube-dl/youtube_dl/extractor/common.py", line 634, in _request_webpage return self._downloader.urlopen(url_or_request) File "/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2288, in urlopen return self._opener.open(req, timeout=self._socket_timeout) File "/usr/lib64/python3.8/urllib/request.py", line 531, in open response = meth(req, response) File "/usr/lib64/python3.8/urllib/request.py", line 640, in http_response response = self.parent.error( File "/usr/lib64/python3.8/urllib/request.py", line 569, in error return self._call_chain(args) File "/usr/lib64/python3.8/urllib/request.py", line 502, in _call_chain result = func(args) File "/usr/lib64/python3.8/urllib/request.py", line 649, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp)

Description

Certain imgur URLs give a 404 error when trying to download them, while they work in Firefox. They are generally URLs beginning with https://imgur.com/a/ , referring to a single video. The example video is NSFW.

dirkf commented 1 year ago

OP example is 404.

groszdaniel commented 1 year ago

@dirkf This video still exists, but gives the same error with youtube-dl 2021.12.17: https://imgur.com/a/j1ltixx

dirkf commented 1 year ago

Indeed:

$ ./youtube-dl-20230718 'https://imgur.com/a/j1ltixx'
[imgur:album] j1ltixx: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
$

This is actually, or at least obscured by, #25351, where the album API URL in the extractor is no longer valid:

         data = self._download_json(
+            'https://api.imgur.com/post/v1/albums/%s?client_id=%s&include=media,account' % (client_id, gallery_id),
+            gallery_id, fatal=False)
-            'https://imgur.com/gallery/%s.json' % gallery_id,
-            gallery_id)['data']['image']

The new replacement API URL works in the browser but gives 401 or 429 in yt-dl, despite apparently the same headers.

dirkf commented 1 year ago

Ah, let's try this instead!

         data = self._download_json(
+            'https://api.imgur.com/post/v1/albums/%s?client_id=%s&include=media,account' % (gallery_id, client_id),
+            gallery_id, fatal=False)
-            'https://imgur.com/gallery/%s.json' % gallery_id,
-            gallery_id)['data']['image']

Then, hurrah!

$ $ python3 -m youtube_dl -v -F 'https://imgur.com/a/j1ltixx'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '-F', 'https://imgur.com/a/j1ltixx']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 5e51cb0aa
[debug] Python 3.11.2 (CPython i686 32bit) - Linux-6.1.0-12-686-pae-i686-with-glibc2.36 - OpenSSL 3.0.9 30 May 2023 - glibc 2.36
[debug] exe versions: ffmpeg 5.1.3-1, ffprobe 5.1.3-1
[debug] Proxy map: {}
[imgur:album] j1ltixx: Downloading JSON metadata
[download] Downloading playlist: Racchetta Zanzare YPD comprato da YASTA
[imgur:album] playlist Racchetta Zanzare YPD comprato da YASTA: Collected 1 video ids (downloading 1 of them)
[download] Downloading video 1 of 1
[Imgur] h4Yc6Cp: Downloading webpage
[info] Available formats for h4Yc6Cp:
format code  extension  resolution note
gif          gif        960x1706   gif container, gif, video only, 3.04MiB
mp4          mp4        960x1706   (best)
[download] Finished downloading playlist: Racchetta Zanzare YPD comprato da YASTA
$

ytdl-org / youtube-dl