search

tubearchivist / tubearchivist

Your self hosted YouTube media server
https://www.tubearchivist.com
GNU General Public License v3.0
5.27k stars 229 forks source link

[Bug]: KeyError 'channel' in playlist.py line 96, tubearchivist stopped downloading new videos / all playlists deactivated #835

Open gebi opened 6 days ago

gebi commented 6 days ago

I've read the documentation

Operating System

Ubuntu 22.04 with docker compose

Your Bug Report

Describe the bug

Adding playlist PLbFspXL_XHc0Vk1MoObY9XLa1i6c-L1Pb on 0.4.11 creates a deactived playlist (see debug output) When the playlist was initially synced with 0.4.10, both 0.4.10 and 0.4.11 stop downloading new videso completely.

Synced with tubearchivist 0.4.10 created a playlist that is shown as containing no videos, but has the thumbnail of the first video shown as playlist thumbnail. after playlist -> delete -> all, and resync with 0.4.11 playlist is shown as 'deactivated' (deleting only metadata was not enough, interestingly enough, all videos are downloaded, just not shown in the playlist)

logs for both problems below 1) tubearchivist stops downloading new videos completely (when playlist is synced with 0.4.10 2) playlist is deactivated when synced with 0.4.11, but download of everything else still works, actually currently all playlists are shown as 'deactivated', and all playlist log the same error as the problematic playlist

Steps To Reproduce

adding playlist and trying to sync it

Expected behavior

not stopping the whole tubearchivist instance to stop syncing (for playlist initially added with 0.4.10) and syncing the playlist when playlist is added on 0.4.11

Relevant log output

# ad 1 - tubearchivist stops downloading new videos completely (when playlist is synced with `0.4.10`

tubearchivist    | [2024-11-16 18:06:29,655: ERROR/ForkPoolWorker-16] Task update_subscribed[e718e3c8-4add-4e2b-8d87-85cb556361b9] raised unexpected: KeyError('channel')
tubearchivist    | Traceback (most recent call last):
tubearchivist    |   File "/root/.local/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
tubearchivist    |     R = retval = fun(*args, **kwargs)
tubearchivist    |                  ^^^^^^^^^^^^^^^^^^^^
tubearchivist    |   File "/root/.local/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
tubearchivist    |     return self.run(*args, **kwargs)
tubearchivist    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
tubearchivist    |   File "/app/home/tasks.py", line 107, in update_subscribed
tubearchivist    |     missing_videos = handler.scan()
tubearchivist    |                      ^^^^^^^^^^^^^^
tubearchivist    |   File "/app/home/src/download/subscriptions.py", line 340, in scan
tubearchivist    |     self.scan_playlists()
tubearchivist    |   File "/app/home/src/download/subscriptions.py", line 359, in scan_playlists
tubearchivist    |     missing = playlist_handler.find_missing()
tubearchivist    |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tubearchivist    |   File "/app/home/src/download/subscriptions.py", line 291, in find_missing
tubearchivist    |     is_active = playlist.update_playlist()
tubearchivist    |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
tubearchivist    |   File "/app/home/src/index/playlist.py", line 173, in update_playlist
tubearchivist    |     self.build_json(scrape=True)
tubearchivist    |   File "/app/home/src/index/playlist.py", line 50, in build_json
tubearchivist    |     self.get_entries(ids_found)
tubearchivist    |   File "/app/home/src/index/playlist.py", line 96, in get_entries
tubearchivist    |     if not entry["channel"]:
tubearchivist    |            ~~~~~^^^^^^^^^^^
tubearchivist    | KeyError: 'channel'
tubearchivist    | [2024-11-16 18:06:29,656: WARNING/ForkPoolWorker-16] e718e3c8-4add-4e2b-8d87-85cb556361b9 return callback

# ad 2 - playlist is deactivated when synced with `0.4.11`, but download of everything else still works

tubearchivist    | [2024-11-16 18:33:02,777: WARNING/ForkPoolWorker-16] PLbFspXL_XHc0Vk1MoObY9XLa1i6c-L1Pb: get metadata from es
tubearchivist    | [2024-11-16 18:33:02,781: WARNING/ForkPoolWorker-16] PLbFspXL_XHc0Vk1MoObY9XLa1i6c-L1Pb: get metadata from youtube
tubearchivist    | [2024-11-16 18:33:03,348: WARNING/ForkPoolWorker-16] WARNING: [youtube:tab] unable to extract yt initial data; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
tubearchivist    | [2024-11-16 18:33:03,349: WARNING/ForkPoolWorker-16] WARNING: [youtube:tab] Incomplete yt initial data received. Retrying (1/3)...
tubearchivist    | [2024-11-16 18:33:03,569: WARNING/ForkPoolWorker-16] WARNING: [youtube:tab] unable to extract yt initial data; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
tubearchivist    | [2024-11-16 18:33:03,569: WARNING/ForkPoolWorker-16] WARNING: [youtube:tab] Incomplete yt initial data received. Retrying (2/3)...
tubearchivist    | [2024-11-16 18:33:03,798: WARNING/ForkPoolWorker-16] WARNING: [youtube:tab] unable to extract yt initial data; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
tubearchivist    | [2024-11-16 18:33:03,798: WARNING/ForkPoolWorker-16] WARNING: [youtube:tab] Incomplete yt initial data received. Retrying (3/3)...
tubearchivist    | [2024-11-16 18:33:04,033: WARNING/ForkPoolWorker-16] WARNING: [youtube:tab] unable to extract yt initial data; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
tubearchivist    | [2024-11-16 18:33:04,033: WARNING/ForkPoolWorker-16] WARNING: [youtube:tab] Incomplete yt initial data received; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest
version using  yt-dlp -U. Giving up after 3 retries
tubearchivist    | [2024-11-16 18:33:04,035: WARNING/ForkPoolWorker-16] ERROR: [youtube:tab] PLbFspXL_XHc0Vk1MoObY9XLa1i6c-L1Pb: Playlists that require authentication may not extract correctly without a successful webpage download. If you are not downloading private content, or your cookies are only for the first account and channel, pass "--extractor-args youtubetab:skip=authcheck" to skip this check
tubearchivist    | [2024-11-16 18:33:04,035: WARNING/ForkPoolWorker-16] https://www.youtube.com/playlist?list=PLbFspXL_XHc0Vk1MoObY9XLa1i6c-L1Pb: failed to get info from youtube with message ERROR: [youtube:tab] PLbFspXL_XHc0Vk1MoObY9XLa1i6c-L1Pb: Playlists that require authentication may not extract correctly without a successful webpage download. If you are not downloading private content, or your cookies are only
for the first account and channel, pass "--extractor-args youtubetab:skip=authcheck" to skip this check
tubearchivist    | [2024-11-16 18:33:04,035: WARNING/ForkPoolWorker-16] PLbFspXL_XHc0Vk1MoObY9XLa1i6c-L1Pb: deactivate document

Anything else?

https://www.youtube.com/playlist?list=PLbFspXL_XHc0Vk1MoObY9XLa1i6c-L1Pb

if i'd had to guess, this video might be the problem https://www.youtube.com/watch?v=-mTmvZjNPdU

bbilly1 commented 5 days ago

You are getting blocked by YT, see our FAQ with some workarounds. https://docs.tubearchivist.com/faq/#am-i-getting-blockedthrottled

Once you unblock your IP/account this should work again.

gebi commented 4 days ago

oh wow... i've no idea what happened.

playlist always shown as having zero videos, new ip and updated session did not help, BUT after adding one new video to the playlist and deleting the cookie in ta it synced the new video and now the playlist is not shown as empty anymore but contains all videos, no idea how because it definitely did not download the rest of the videos, but works again now.

thx for your patience, and sry for wasting your time :/