search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.2k stars 913 forks source link

reddit Failed to parse JSON (caused by JSONDecodeError("Expecting value in '': line 1 column 1 (char 0)")) #5411

Open mo-han opened 5 months ago

mo-han commented 5 months ago

r/Tongactie the whole subreddit has this issue. BTW this subreddit is kind of wierd: i cannot cross post from it.

[gallery-dl][debug] Version 1.26.9
[gallery-dl][debug] Python 3.8.10 - Windows-10-10.0.22631-SP0
[gallery-dl][debug] requests 2.31.0 - urllib3 2.1.0
[gallery-dl][debug] Configuration Files ['C:\\Users\\mo-han\\locallib\\usr\\etc\\gallery-dl.json']
[gallery-dl][debug] Starting DownloadJob for 'https://www.reddit.com/r/Tongactie/hot'
[reddit][debug] Using RedditSubredditExtractor for 'https://www.reddit.com/r/Tongactie/hot'
[cookies][debug] Extracting cookies from C:\Users\mo-han\AppData\Roaming\Mozilla\Firefox\Profiles\4n321zy5.default-release\cookies.sqlite
[cookies][info] Extracted 153 cookies from Firefox
[reddit][debug] Using custom API credentials (client-id Qd-9E*****************)
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): oauth.reddit.com:443
[urllib3.connectionpool][debug] https://oauth.reddit.com:443 "GET /r/Tongactie/hot/.json?limit=100&raw_json=1 HTTP/1.1" 200 96089
[reddit][debug] Active postprocessor modules: [ExecPP]
[downloader.ytdl][debug] [Reddit] Extracting URL: https://www.reddit.com/r/Tongactie/comments/11si4bd/pink/
[downloader.ytdl][debug] [Reddit] 11si4bd: Downloading JSON metadata
[downloader.ytdl][warning] [Reddit] 11si4bd: Failed to parse JSON: Expecting value in '': line 1 column 1 (char 0)
[downloader.ytdl][debug] [Reddit] www.reddit.com request failed, retrying with old.reddit.com
[downloader.ytdl][debug] [Reddit] 11si4bd: Downloading JSON metadata
[downloader.ytdl][error] ERROR: [Reddit] 11si4bd: 11si4bd: Failed to parse JSON (caused by JSONDecodeError("Expecting value in '': line 1 column 1 (char 0)")); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
[download][error] Failed to download ytdl:https://www.reddit.com/r/Tongactie/comments/11si4bd/pink/
[downloader.ytdl][debug] [Reddit] Extracting URL: https://www.reddit.com/r/Tongactie/comments/10gwkab/twins/
[downloader.ytdl][debug] [Reddit] 10gwkab: Downloading JSON metadata
[downloader.ytdl][warning] [Reddit] 10gwkab: Failed to parse JSON: Expecting value in '': line 1 column 1 (char 0)
[downloader.ytdl][debug] [Reddit] www.reddit.com request failed, retrying with old.reddit.com
[downloader.ytdl][debug] [Reddit] 10gwkab: Downloading JSON metadata
[downloader.ytdl][error] ERROR: [Reddit] 10gwkab: 10gwkab: Failed to parse JSON (caused by JSONDecodeError("Expecting value in '': line 1 column 1 (char 0)")); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
[download][error] Failed to download ytdl:https://www.reddit.com/r/Tongactie/comments/10gwkab/twins/
[downloader.ytdl][debug] [Reddit] Extracting URL: https://www.reddit.com/r/Tongactie/comments/zhznex/combo/
[downloader.ytdl][debug] [Reddit] zhznex: Downloading JSON metadata
[downloader.ytdl][warning] [Reddit] zhznex: Failed to parse JSON: Expecting value in '': line 1 column 1 (char 0)
[downloader.ytdl][debug] [Reddit] www.reddit.com request failed, retrying with old.reddit.com
[downloader.ytdl][debug] [Reddit] zhznex: Downloading JSON metadata
[downloader.ytdl][error] ERROR: [Reddit] zhznex: zhznex: Failed to parse JSON (caused by JSONDecodeError("Expecting value in '': line 1 column 1 (char 0)")); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
[download][error] Failed to download ytdl:https://www.reddit.com/r/Tongactie/comments/zhznex/combo/
[downloader.ytdl][debug] [Reddit] Extracting URL: https://www.reddit.com/r/Tongactie/comments/z0azs1/pink_dominating/
[downloader.ytdl][debug] [Reddit] z0azs1: Downloading JSON metadata
[downloader.ytdl][warning] [Reddit] z0azs1: Failed to parse JSON: Expecting value in '': line 1 column 1 (char 0)
[downloader.ytdl][debug] [Reddit] www.reddit.com request failed, retrying with old.reddit.com
[downloader.ytdl][debug] [Reddit] z0azs1: Downloading JSON metadata
[downloader.ytdl][error] ERROR: [Reddit] z0azs1: z0azs1: Failed to parse JSON (caused by JSONDecodeError("Expecting value in '': line 1 column 1 (char 0)")); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
[download][error] Failed to download ytdl:https://www.reddit.com/r/Tongactie/comments/z0azs1/pink_dominating/
[downloader.ytdl][debug] [Reddit] Extracting URL: https://www.reddit.com/r/Tongactie/comments/z0b6ix/long_brazil_kiss/
[downloader.ytdl][debug] [Reddit] z0b6ix: Downloading JSON metadata
[downloader.ytdl][warning] [Reddit] z0b6ix: Failed to parse JSON: Expecting value in '': line 1 column 1 (char 0)
[downloader.ytdl][debug] [Reddit] www.reddit.com request failed, retrying with old.reddit.com
[downloader.ytdl][debug] [Reddit] z0b6ix: Downloading JSON metadata
mikf commented 5 months ago

Try using a different reddit.videos setting. "dash" shouldn't have any JSON problems.

mo-han commented 5 months ago

yes, videos=dash works for this sub's posts. ytdl get error, and true is auto ytdl and also error. would be better if gldl could auto detect such error and switch to try dash.

BTW, { "extractor": { "reddit": { "videos": "dash", in the config file not working, -o videos=dash is working. i'm sure the config is active since the reddit client-id is in it and used.

cheese529 commented 5 months ago

@mo-han what error is yt-dlp getting? can you please post a verbose log for it?

mo-han commented 5 months ago

@mo-han what error is yt-dlp getting? can you please post a verbose log for it?

i thought the above is the verbose log, it isn't?

Hrxn commented 5 months ago

BTW this subreddit is kind of wierd: i cannot cross post from it.

This is probably by design, a setting on reddit's side.

Can you try with a single submission URL, just to be sure?

Because I'm using "videos": "ytdl" in my config for reddit, and it was working for me..

FWIW, I've tested these URLs

python.exe 'C:\Sources\gallery-dl-master\gallery_dl' 'https://www.reddit.com/r/Tongactie/comments/11si4bd/pink/'
python.exe 'C:\Sources\gallery-dl-master\gallery_dl' 'https://www.reddit.com/r/Tongactie/comments/10gwkab/twins/'

BTW, { "extractor": { "reddit": { "videos": "dash", in the config file not working, -o videos=dash is working. i'm sure the config is active since the reddit client-id is in it and used.

This does not make any sense. Settings should work the same in any cases.

Hrxn commented 5 months ago

One more thing, do you have "module": "yt_dlp" in both "extractor.ytdl" and "downloader.ytdl"?

Also maybe try "forward-cookies": true, in "downloader.ytdl", I have that one as well. Just to be sure.

cheese529 commented 5 months ago

@mo-han what error is yt-dlp getting? can you please post a verbose log for it?

i thought the above is the verbose log, it isn't?

I meant a verbose log for yt-dlp, you must follow what @Hrxn has mentioned first though as that is how to utilize yt-dlp within gallery-dl

mo-han commented 5 months ago

@Hrxn @cheese529 Ok I'll try these things later and update if finding something.

C:\Users\mo-han>gallery-dl %url% -vv
[gallery-dl][debug] Version 1.26.9
[gallery-dl][debug] Python 3.8.10 - Windows-10-10.0.22631-SP0
[gallery-dl][debug] requests 2.31.0 - urllib3 2.1.0
[gallery-dl][debug] Configuration Files []
[gallery-dl][debug] Starting DownloadJob for 'https://www.reddit.com/r/Tongactie/comments/11si4bd/pink/'
[reddit][debug] Using RedditSubmissionExtractor for 'https://www.reddit.com/r/Tongactie/comments/11si4bd/pink/'
[reddit][debug] Using default API credentials (client-id 6N9uN0krSDE-ig)
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): oauth.reddit.com:443
[urllib3.connectionpool][debug] https://oauth.reddit.com:443 "GET /comments/11si4bd/.json?limit=0&raw_json=1 HTTP/1.1" 429 None
[reddit][info] Register your own OAuth application and use its credentials to prevent this error: https://github.com/mikf/gallery-dl/blob/master/docs/configuration.rst#extractorredditclient-id--user-agent
[reddit][info] Waiting until 15:30:00 for rate limit reset.
[urllib3.connectionpool][debug] Resetting dropped connection: oauth.reddit.com
[urllib3.connectionpool][debug] https://oauth.reddit.com:443 "GET /comments/11si4bd/.json?limit=0&raw_json=1 HTTP/1.1" 200 2697
[downloader.ytdl][debug] [generic] Extracting URL: https://v.redd.it/3wlxt3lzm0oa1
[downloader.ytdl][debug] [generic] 3wlxt3lzm0oa1: Downloading webpage
[downloader.ytdl][error] ERROR: [generic] Unable to download webpage: HTTP Error 403: Blocked (caused by <HTTPError 403: Blocked>)
[download][error] Failed to download ytdl:https://v.redd.it/3wlxt3lzm0oa1

C:\Users\mo-han>gallery-dl %url% -vv
[gallery-dl][debug] Version 1.26.9
[gallery-dl][debug] Python 3.8.10 - Windows-10-10.0.22631-SP0
[gallery-dl][debug] requests 2.31.0 - urllib3 2.1.0
[gallery-dl][debug] Configuration Files []
[gallery-dl][debug] Starting DownloadJob for 'https://www.reddit.com/r/Tongactie/comments/11si4bd/pink/'
[reddit][debug] Using RedditSubmissionExtractor for 'https://www.reddit.com/r/Tongactie/comments/11si4bd/pink/'
[reddit][debug] Using default API credentials (client-id 6N9uN0krSDE-ig)
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): oauth.reddit.com:443
[urllib3.connectionpool][debug] https://oauth.reddit.com:443 "GET /comments/11si4bd/.json?limit=0&raw_json=1 HTTP/1.1" 429 None
[reddit][info] Register your own OAuth application and use its credentials to prevent this error: https://github.com/mikf/gallery-dl/blob/master/docs/configuration.rst#extractorredditclient-id--user-agent
[reddit][info] Waiting until 15:40:00 for rate limit reset.
[urllib3.connectionpool][debug] Resetting dropped connection: oauth.reddit.com
[urllib3.connectionpool][debug] https://oauth.reddit.com:443 "GET /comments/11si4bd/.json?limit=0&raw_json=1 HTTP/1.1" 200 2693
[downloader.ytdl][debug] [generic] Extracting URL: https://v.redd.it/3wlxt3lzm0oa1
[downloader.ytdl][debug] [generic] 3wlxt3lzm0oa1: Downloading webpage
[downloader.ytdl][debug] [redirect] Following redirect to https://www.reddit.com/r/Tongactie/comments/11si4bd/pink/?rdt=41090
[downloader.ytdl][debug] [Reddit] Extracting URL: https://www.reddit.com/r/Tongactie/comments/11si4bd/pink/?rdt=41090
[downloader.ytdl][debug] [Reddit] 11si4bd: Downloading JSON metadata
[downloader.ytdl][debug] [Reddit] 11si4bd: Downloading m3u8 information
[downloader.ytdl][debug] [Reddit] 11si4bd: Downloading MPD manifest
  .\gallery-dl\reddit\Tongactie\11si4bd pink.mp4[downloader.ytdl][debug] [download] Destination: \\?\C:\Users\mo-han\gallery-dl\reddit\Tongactie\11si4bd pink.ffallback.mp4
[downloader.ytdl][debug] [download]   0.0% of   14.28MiB at  949.15KiB/s ETA 00:15
[downloader.ytdl][debug] [download]   0.0% of   14.28MiB at    1.39MiB/s ETA 00:10
[downloader.ytdl][debug] [download]   0.0% of   14.28MiB at  143.00KiB/s ETA 01:42
[downloader.ytdl][debug] [download]   0.1% of   14.28MiB at  299.88KiB/s ETA 00:48
[downloader.ytdl][debug] [download]   0.2% of   14.28MiB at  606.87KiB/s ETA 00:24
[downloader.ytdl][debug] [download]   0.4% of   14.28MiB at  646.18KiB/s ETA 00:22
[downloader.ytdl][debug] [download]   0.9% of   14.28MiB at  437.59KiB/s ETA 00:33
[downloader.ytdl][debug] [download]   1.7% of   14.28MiB at  598.70KiB/s ETA 00:24
[downloader.ytdl][debug] [download]   3.5% of   14.28MiB at  731.42KiB/s ETA 00:19
[downloader.ytdl][debug] [download]   7.0% of   14.28MiB at    1.03MiB/s ETA 00:12
[downloader.ytdl][debug] [download]  14.0% of   14.28MiB at    1.60MiB/s ETA 00:07
[downloader.ytdl][debug] [download]  28.0% of   14.28MiB at    2.50MiB/s ETA 00:04
[downloader.ytdl][debug] [download]  56.0% of   14.28MiB at    4.17MiB/s ETA 00:01
[downloader.ytdl][debug] [download]  84.0% of   14.28MiB at    5.34MiB/s ETA 00:00
[downloader.ytdl][debug] [download] 100.0% of   14.28MiB at    6.14MiB/s ETA 00:00
[downloader.ytdl][debug] [download] 100% of   14.28MiB in 00:00:02 at 5.34MiB/s
[downloader.ytdl][debug] [download] Destination: \\?\C:\Users\mo-han\gallery-dl\reddit\Tongactie\11si4bd pink.fdash-audio_0_131460.m4a
[downloader.ytdl][debug] [download]   0.2% of  464.53KiB at  Unknown B/s ETA Unknown
[downloader.ytdl][debug] [download]   0.6% of  464.53KiB at    1.90MiB/s ETA 00:00
[downloader.ytdl][debug] [download]   1.5% of  464.53KiB at    2.70MiB/s ETA 00:00
[downloader.ytdl][debug] [download]   3.2% of  464.53KiB at    5.78MiB/s ETA 00:00
[downloader.ytdl][debug] [download]   6.7% of  464.53KiB at    8.56MiB/s ETA 00:00
[downloader.ytdl][debug] [download]  13.6% of  464.53KiB at   15.20MiB/s ETA 00:00
[downloader.ytdl][debug] [download]  27.3% of  464.53KiB at   30.64MiB/s ETA 00:00
[downloader.ytdl][debug] [download]  54.9% of  464.53KiB at   27.81MiB/s ETA 00:00
[downloader.ytdl][debug] [download] 100.0% of  464.53KiB at   23.48MiB/s ETA 00:00
[downloader.ytdl][debug] [download] 100% of  464.53KiB in 00:00:00 at 1.22MiB/s
[downloader.ytdl][debug] [Merger] Merging formats into "\\?\C:\Users\mo-han\gallery-dl\reddit\Tongactie\11si4bd pink.mp4"
[downloader.ytdl][debug] Deleting original file \\?\C:\Users\mo-han\gallery-dl\reddit\Tongactie\11si4bd pink.ffallback.mp4 (pass -k to keep)
[downloader.ytdl][debug] Deleting original file \\?\C:\Users\mo-han\gallery-dl\reddit\Tongactie\11si4bd pink.fdash-audio_0_131460.m4a (pass -k to keep)
* .\gallery-dl\reddit\Tongactie\11si4bd pink.mp4

Last try:

So it's a matter of proxy server, should've thought of that when saw the 403 blocked message. The ytdl.module is not needed in config as the default is doing things correctly. It's still a mystery the videos value in my config.json takes no effect, but it's not big deal.