search

yt-dlp / yt-dlp

A feature-rich command-line audio/video downloader
https://discord.gg/H5MNcFW63r
The Unlicense
91.13k stars 7.08k forks source link

Dropout.tv write-info-json #2360

Open schmuelio opened 2 years ago

schmuelio commented 2 years ago

Checklist

Region

UK

Example URLs

https://www.dropout.tv/dimension-20/season:1/videos/fantasy-high-live-at-rtx-austin-1 https://www.dropout.tv/dimension-20/season:1/videos/prompocalypse-pt-2 https://www.dropout.tv/dimension-20

Description

I'm attempting to use yt-dlp to search for and download episodes from a site using sonarr_yt-dlp and came across this error.

I've narrowed it down to a failure to extract playlist information from specific videos within a playlist on dropout.tv, returning error 406.

Downloading the whole playlist as-is works correctly, but this would cause a lot of wasted download time and effort when a single episode is needed.

I'm afraid I don't have any suggestions on how to fix, but I'm happy to work with and provide any additional information as needed.

Any workarounds would also be welcomed. Thanks.

Verbose log

$ yt-dlp "https://www.dropout.tv/dimension-20/season:1/videos/fantasy-high-live-at-rtx-austin-1" --username PRIVATE --password PRIVATE --write-info-json --no-download -Uv
[debug] Command-line config: ['https://www.dropout.tv/dimension-20/season:1/videos/fantasy-high-live-at-rtx-austin-1', '--username', 'PRIVATE', '--password', 'PRIVATE', '--write-info-json', '--no-download', '-Uv']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, err utf-8, pref UTF-8
[debug] yt-dlp version 2021.12.27 [6223f67a8]
[debug] Python version 3.10.1 (CPython 64bit) - Linux-5.16.0-arch1-1-x86_64-with-glibc2.33
[debug] exe versions: ffmpeg 4.4.1 (setts), ffprobe 4.4.1, rtmpdump 2.4
[debug] Optional libraries: Cryptodome, secretstorage, mutagen, sqlite, websockets
[debug] Proxy map: {}
Latest version: 2021.12.27, Current version: 2021.12.27
yt-dlp is up to date (2021.12.27)
[debug] [Dropout] Extracting URL: https://www.dropout.tv/dimension-20/season:1/videos/fantasy-high-live-at-rtx-austin-1
[Dropout] fantasy-high-live-at-rtx-austin-1: Getting authenticity token
[Dropout] fantasy-high-live-at-rtx-austin-1: Logging out
ERROR: [Dropout] fantasy-high-live-at-rtx-austin-1: Unable to download webpage: HTTP Error 406: Not Acceptable (caused by <HTTPError 406: 'Not Acceptable'>); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output. (caused by <HTTPError 406: 'Not Acceptable'>); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
  File "/usr/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 717, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 3458, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.10/urllib/request.py", line 525, in open
    response = meth(req, response)
  File "/usr/lib/python3.10/urllib/request.py", line 634, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.10/urllib/request.py", line 563, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
schmuelio commented 2 years ago

On further inspection I think this is a rate-limiting mechanism of some kind.

It looked like it worked correctly during an attempt this morning, but after several queries it began returning Error 429: Too Many Requests, further attempts resulted in 406 again.

$ CONFIGPATH="./config.yml" python3 sonarr_youtubedl.py
...
2022-01-17 15:16:14,748 - sonarr_youtubedl - INFO -   Dimension 20:
WARNING: [vhx:embed] Ignoring subtitle tracks found in the HLS manifest; if any subtitle tracks are missing, please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
... [Repeated several times]
WARNING: [vhx:embed] Ignoring subtitle tracks found in the HLS manifest; if any subtitle tracks are missing, please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
2022-01-17 15:17:45,153 - sonarr_youtubedl - INFO -     4: Found - The Beginning Begins:
2022-01-17 15:17:45,170 - sonarr_youtubedl - INFO -       Downloaded - The Beginning Begins
WARNING: [vhx:embed] Ignoring subtitle tracks found in the HLS manifest; if any subtitle tracks are missing, please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
WARNING: [vhx:embed] Ignoring subtitle tracks found in the HLS manifest; if any subtitle tracks are missing, please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
ERROR: [Dropout] hall-of-heroes: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by <HTTPError 429: 'Too Many Requests'>); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output. (caused by <HTTPError 429: 'Too Many Requests'>); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
... [Repeated several times]
ERROR: [Dropout] home-for-the-holidays: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by <HTTPError 429: 'Too Many Requests'>); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output. (caused by <HTTPError 429: 'Too Many Requests'>); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.

$ CONFIGPATH="./config.yml" python3 sonarr_youtubedl.py
...
2022-01-17 15:19:21,461 - sonarr_youtubedl - INFO -   Dimension 20:
ERROR: [Dropout] times-squaremageddon-pt-2: Unable to download webpage: HTTP Error 406: Not Acceptable (caused by <HTTPError 406: 'Not Acceptable'>); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output. (caused by <HTTPError 406: 'Not Acceptable'>); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
...

Verified by attempting with yt-dlp directly:

yt-dlp https://www.dropout.tv/dimension-20 --username PRIVATE --password PRIVATE
[DropoutSeason] dimension-20: Downloading webpage
[download] Downloading playlist: Dimension 20 - Fantasy High
[DropoutSeason] playlist Dimension 20 - Fantasy High: Collected 20 videos; downloading 20 of them
[download] Downloading video 1 of 20
[Dropout] episode-1: Getting authenticity token
[Dropout] episode-1: Logging in
[Dropout] episode-1: Logging out
ERROR: [Dropout] episode-1: Unable to download webpage: HTTP Error 406: Not Acceptable (caused by <HTTPError 406: 'Not Acceptable'>); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output. (caused by <HTTPError 406: 'Not Acceptable'>); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
[download] Downloading video 2 of 20
[Dropout] clash-of-the-corn-cuties: Getting authenticity token
[Dropout] clash-of-the-corn-cuties: Logging in
[Dropout] clash-of-the-corn-cuties: Logging out
ERROR: [Dropout] clash-of-the-corn-cuties: Unable to download webpage: HTTP Error 406: Not Acceptable (caused by <HTTPError 406: 'Not Acceptable'>); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output. (caused by <HTTPError 406: 'Not Acceptable'>); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
...
DoNotResuscitate commented 2 years ago

@schmuelio how long did it take for the rate limiting to go away?

schmuelio commented 2 years ago

From memory on the order of an hour or two I think.

pukkandan commented 2 years ago

I just noticed that OP's error is happening during logout. Logout should actually be non-fatal

cc @KathrynElrod

DoNotResuscitate commented 2 years ago

From memory on the order of an hour or two I think.

Hmm, it's been over 4 hours for me and I still can't login on the website, keeps throwing the 406 error. I made a ticket with Dropout about it.

schmuelio commented 2 years ago

It could be closer to 4 hours, I haven't tried again for a little while.

On Fri, 11 Mar 2022, 8:31 pm Alex G, @.***> wrote:

From memory on the order of an hour or two I think.

Hmm, it's been over 4 hours for me and I still can't login on the website, keeps throwing the 406 error. I made a ticket with Dropout about it.

— Reply to this email directly, view it on GitHub https://github.com/yt-dlp/yt-dlp/issues/2360#issuecomment-1065494729, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWYFYVAWOF3SBUJD56RJB53U7OUQVANCNFSM5MDBEVYA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

DeftNerd commented 2 years ago

In a separate issue ( #4035 ), I referenced something that may be related to this problem.

I'm pretty sure the rate limitation is being triggered by the number of authentications. The yt-dlp extractor for dropout.tv is pretty casual about logging out and logging in.

If you point yt-dlp at a URL representing a whole season, the extractor logs out and logs back in for each video in the season. If the extractor can maintain the session for the whole download process, it'll likely reduce the risk of being rate limited.