Open holta opened 1 month ago
FYI I tried the following:
systemctl stop calibre-web
cd /library/calibre-web
mv xklb-metadata.db xklb-metadata.db.BKP
systemctl start calibre-web
Then I tried "Download to IIAB" with https://youtu.be/zRM2AnwNY20 again.
Result is:
NOTICE the slightly different error message in Tasks view: failed to download: None
New iiab-diagnostics: https://paste.centos.org/view/3eb329f8
@deldesir can you please log in to .0.10 to investigate?
- Is this mysterious msg at the bottom of
/var/log/xklb.log
relevant/meaningful ❓2024-10-17 08:58:49 - [Debug] [best]: Download was marked as deleted 6 minutes and 49 seconds ago. Skipping! 2024-10-17 08:58:49 - [Info] lb-wrapper's xklb command (dl) completed successfully.
@deldesir what are all these best
messages telling us, can you help explain?
root@box:~# grep -n '\[best\]' /var/log/xklb.log
51:2024-10-17 08:52:00 - [Debug] [best]: yt-dlp ERROR: [generic] 'best' is not a valid URL. Set --default-search "ytsearch" (or run yt-dlp "ytsearch:best" ) to search YouTube
52:2024-10-17 08:52:00 - [Debug] [best]: yt-dlp returned no info
53:2024-10-17 08:52:00 - [Debug] [best]: Unrecoverable error matched. ERROR: [generic] 'best' is not a valid URL. Set --default-search "ytsearch" (or run yt-dlp "ytsearch:best" ) to search YouTube
79:2024-10-17 08:54:44 - [Debug] [best]: Download was marked as deleted 2 minutes and 45 seconds ago. Skipping!
104:2024-10-17 08:58:49 - [Debug] [best]: Download was marked as deleted 6 minutes and 49 seconds ago. Skipping!
129:2024-10-17 11:59:45 - [Debug] [best]: Download was marked as deleted 3 hours and 8 minutes ago. Skipping!
182:2024-10-18 09:05:50 - [Debug] [best]: yt-dlp ERROR: [generic] 'best' is not a valid URL. Set --default-search "ytsearch" (or run yt-dlp "ytsearch:best" ) to search YouTube
183:2024-10-18 09:05:50 - [Debug] [best]: yt-dlp returned no info
184:2024-10-18 09:05:50 - [Debug] [best]: Unrecoverable error matched. ERROR: [generic] 'best' is not a valid URL. Set --default-search "ytsearch" (or run yt-dlp "ytsearch:best" ) to search YouTube
All 30 mentions of best
in /var/log/xklb.log are hereunder if it helps!
root@box:~# grep -n 'best' /var/log/xklb.log
43:2024-10-17 08:51:59 - [Info] Running xklb command: lb dl '/library/calibre-web/xklb-metadata.db' --video --search 'https://www.youtube.com/watch?v=M_Zt5K057xs' --format best --format-sort 'tbr~1000' --write-thumbnail --subs --live --live-from-start -o '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
45:2024-10-17 08:52:00 - [Debug] ['/usr/local/bin/lb', 'dl', '/library/calibre-web/xklb-metadata.db', '--video', '--search', 'https://www.youtube.com/watch?v=M_Zt5K057xs', '--format', 'best', '--format-sort', 'tbr~1000', '--write-thumbnail', '--subs', '--live', '--live-from-start', '-o', '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s', '-vv']
46:2024-10-17 08:52:00 - [Debug] {'include': ['https://www.youtube.com/watch?v=M_Zt5K057xs'], 'subs': True, 'live': True, 'paths': ['best']}
48:2024-10-17 08:52:00 - [Debug] {'path': 'best'}
49:2024-10-17 08:52:00 - [Debug] {'noprogress': True, 'lazy_playlist': True, 'noplaylist': True, 'no_check_certificate': True, 'ignore_no_formats_error': True, 'skip_playlist_after_errors': 21, 'playlistend': 20000, 'extractor_args': {'youtube': {'skip': ['authcheck']}}, 'check_formats': 'selected', 'logger': <xklb.createdb.tube_backend.download.<locals>.DictLogger object at 0x719a57287080>, 'postprocessors': [{'key': 'FFmpegMetadata'}, {'key': 'FFmpegEmbedSubtitle'}], 'playlist_items': '1', 'extractor_retries': 3, 'retries': 12, 'retry_sleep_functions': {'extractor': <function download.<locals>.<lambda> at 0x719a5729f880>, 'http': <function download.<locals>.<lambda> at 0x719a5729f920>, 'fragment': <function download.<locals>.<lambda> at 0x719a5729f9c0>}, 'outtmpl': {'default': '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s'}, 'progress_hooks': [<function download.<locals>.<lambda> at 0x719a5729fa60>], 'subtitlesformat': 'srt/best', 'writesubtitles': True, 'writeautomaticsub': True, 'format': '--format-sort', 'writethumbnail': True, 'live_from_start': True, 'include': ['https://www.youtube.com/watch?v=M_Zt5K057xs'], 'subs': True, 'live': True}
50:2024-10-17 08:52:00 - [Debug] [yt-dlp]: Downloading best
51:2024-10-17 08:52:00 - [Debug] [best]: yt-dlp ERROR: [generic] 'best' is not a valid URL. Set --default-search "ytsearch" (or run yt-dlp "ytsearch:best" ) to search YouTube
52:2024-10-17 08:52:00 - [Debug] [best]: yt-dlp returned no info
53:2024-10-17 08:52:00 - [Debug] [best]: Unrecoverable error matched. ERROR: [generic] 'best' is not a valid URL. Set --default-search "ytsearch" (or run yt-dlp "ytsearch:best" ) to search YouTube
74:2024-10-17 08:54:43 - [Info] Running xklb command: lb dl '/library/calibre-web/xklb-metadata.db' --video --search 'https://www.youtube.com/watch?v=M_Zt5K057xs' --format best --format-sort 'tbr~1000' --write-thumbnail --subs --live --live-from-start -o '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
76:2024-10-17 08:54:44 - [Debug] ['/usr/local/bin/lb', 'dl', '/library/calibre-web/xklb-metadata.db', '--video', '--search', 'https://www.youtube.com/watch?v=M_Zt5K057xs', '--format', 'best', '--format-sort', 'tbr~1000', '--write-thumbnail', '--subs', '--live', '--live-from-start', '-o', '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s', '-vv']
77:2024-10-17 08:54:44 - [Debug] {'include': ['https://www.youtube.com/watch?v=M_Zt5K057xs'], 'subs': True, 'live': True, 'paths': ['best']}
79:2024-10-17 08:54:44 - [Debug] [best]: Download was marked as deleted 2 minutes and 45 seconds ago. Skipping!
99:2024-10-17 08:58:48 - [Info] Running xklb command: lb dl '/library/calibre-web/xklb-metadata.db' --video --search 'https://www.youtube.com/watch?v=zRM2AnwNY20' --format best --format-sort 'tbr~1000' --write-thumbnail --subs --live --live-from-start -o '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
101:2024-10-17 08:58:48 - [Debug] ['/usr/local/bin/lb', 'dl', '/library/calibre-web/xklb-metadata.db', '--video', '--search', 'https://www.youtube.com/watch?v=zRM2AnwNY20', '--format', 'best', '--format-sort', 'tbr~1000', '--write-thumbnail', '--subs', '--live', '--live-from-start', '-o', '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s', '-vv']
102:2024-10-17 08:58:48 - [Debug] {'include': ['https://www.youtube.com/watch?v=zRM2AnwNY20'], 'subs': True, 'live': True, 'paths': ['best']}
104:2024-10-17 08:58:49 - [Debug] [best]: Download was marked as deleted 6 minutes and 49 seconds ago. Skipping!
124:2024-10-17 11:59:44 - [Info] Running xklb command: lb dl '/library/calibre-web/xklb-metadata.db' --video --search 'https://www.youtube.com/watch?v=zRM2AnwNY20' --format best --format-sort 'tbr~1000' --write-thumbnail --subs --live --live-from-start -o '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
126:2024-10-17 11:59:45 - [Debug] ['/usr/local/bin/lb', 'dl', '/library/calibre-web/xklb-metadata.db', '--video', '--search', 'https://www.youtube.com/watch?v=zRM2AnwNY20', '--format', 'best', '--format-sort', 'tbr~1000', '--write-thumbnail', '--subs', '--live', '--live-from-start', '-o', '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s', '-vv']
127:2024-10-17 11:59:45 - [Debug] {'include': ['https://www.youtube.com/watch?v=zRM2AnwNY20'], 'subs': True, 'live': True, 'paths': ['best']}
129:2024-10-17 11:59:45 - [Debug] [best]: Download was marked as deleted 3 hours and 8 minutes ago. Skipping!
174:2024-10-18 09:05:49 - [Info] Running xklb command: lb dl '/library/calibre-web/xklb-metadata.db' --video --search 'https://www.youtube.com/watch?v=zRM2AnwNY20' --format best --format-sort 'tbr~1000' --write-thumbnail --subs --live --live-from-start -o '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
176:2024-10-18 09:05:49 - [Debug] ['/usr/local/bin/lb', 'dl', '/library/calibre-web/xklb-metadata.db', '--video', '--search', 'https://www.youtube.com/watch?v=zRM2AnwNY20', '--format', 'best', '--format-sort', 'tbr~1000', '--write-thumbnail', '--subs', '--live', '--live-from-start', '-o', '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s', '-vv']
177:2024-10-18 09:05:49 - [Debug] {'include': ['https://www.youtube.com/watch?v=zRM2AnwNY20'], 'subs': True, 'live': True, 'paths': ['best']}
179:2024-10-18 09:05:49 - [Debug] {'path': 'best'}
180:2024-10-18 09:05:49 - [Debug] {'noprogress': True, 'lazy_playlist': True, 'noplaylist': True, 'no_check_certificate': True, 'ignore_no_formats_error': True, 'skip_playlist_after_errors': 21, 'playlistend': 20000, 'extractor_args': {'youtube': {'skip': ['authcheck']}}, 'check_formats': 'selected', 'logger': <xklb.createdb.tube_backend.download.<locals>.DictLogger object at 0x7a1068560aa0>, 'postprocessors': [{'key': 'FFmpegMetadata'}, {'key': 'FFmpegEmbedSubtitle'}], 'playlist_items': '1', 'extractor_retries': 3, 'retries': 12, 'retry_sleep_functions': {'extractor': <function download.<locals>.<lambda> at 0x7a106856dbc0>, 'http': <function download.<locals>.<lambda> at 0x7a106856dc60>, 'fragment': <function download.<locals>.<lambda> at 0x7a106856dd00>}, 'outtmpl': {'default': '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s'}, 'progress_hooks': [<function download.<locals>.<lambda> at 0x7a106856dda0>], 'subtitlesformat': 'srt/best', 'writesubtitles': True, 'writeautomaticsub': True, 'format': '--format-sort', 'writethumbnail': True, 'live_from_start': True, 'include': ['https://www.youtube.com/watch?v=zRM2AnwNY20'], 'subs': True, 'live': True}
181:2024-10-18 09:05:49 - [Debug] [yt-dlp]: Downloading best
182:2024-10-18 09:05:50 - [Debug] [best]: yt-dlp ERROR: [generic] 'best' is not a valid URL. Set --default-search "ytsearch" (or run yt-dlp "ytsearch:best" ) to search YouTube
183:2024-10-18 09:05:50 - [Debug] [best]: yt-dlp returned no info
184:2024-10-18 09:05:50 - [Debug] [best]: Unrecoverable error matched. ERROR: [generic] 'best' is not a valid URL. Set --default-search "ytsearch" (or run yt-dlp "ytsearch:best" ) to search YouTube
I tried again with longer-form (canonical) URL https://www.youtube.com/watch?v=zRM2AnwNY20 for the same video:
After running this:
systemctl stop calibre-web
cd /library/calibre-web
mv xklb-metadata.db xklb-metadata.db.BKP2
systemctl start calibre-web
The result is the same (failed to download: None
) as seen below:
New VM's built with Debian 12 and Ubuntu 24.04 do NOT have this problem!
Strangely, the problem appears on two Ubuntu 24.10 VM's only 🤔
Debian 13 should be tested next, to see what happens there?!
- Debian 13 should be tested next, to see what happens there?!
Verdict / FYI: Video downloads work on a fresh & fully-updated Debian 13 Trixie VM install❗
@deldesir and I were wondering if Ubuntu 24.10's bash version might be causing this new problem. But that would appear not to be true, as Debian 13 (works!) and Ubuntu 24.10 (doesn't work!) are running the exact same version of bash:
root@box:~# bash --version
GNU bash, version 5.2.32(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Whereas Ubuntu 24.04 shows...
root@u2404-desk:~# bash --version
GNU bash, version 5.2.21(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
@deldesir please test IIAB Calibre-Web on Ubuntu 25.04 (Plucky Puffin) pre-releases next week when that's made available via Multipass, and report back, Thanks!
Clarif For All: Multipass makes IIAB Calibre-Web testing nearly instantaneous — follow the instructions at https://github.com/iiab/calibre-web/wiki#wrench-installation
@deldesir please test IIAB Calibre-Web on Ubuntu 25.04 (Plucky Puffin) pre-releases next week when that's made available via Multipass, and report back, Thanks!
Ubuntu 25.04 Release Schedule & (Pre-)Release Notes:
While I thought this was obvious (many fresh installs of Ubuntu 24.10 by at least 3 different people, were never able to "Download to IIAB" over the past 10+ days, which by definition means an empty database /library/calibre-web/xklb-metadata.db
and no prior "Download to IIAB" attempts) this is important to state more prominently, to be crystal clear! 🙏
Another example, just to confirm this problem is still happening:
https://www.youtube.com/watch?v=AAyCh3IWlN4 ("MI PRIMERA VEZ EN ECUADOR!")
iiab-diagnostics: https://paste.centos.org/view/d9b61464
The issue seems to be related to Bash and it affects only Ubuntu 24.10.
Other issues about Bash on Ubuntu 24.10 (may or may not be related): https://github.com/jellyfin/jellyfin-repo-helper-scripts/issues/5 https://askubuntu.com/questions/1531490/printf-builtin-prints-nan-for-floating-point-numbers-in-ubuntu-24-10
Tests should now be run on Ubuntu 25.04 pre-releases like this one:
https://cloud-images.ubuntu.com/plucky/
(...to confirm that IIAB Calibre-Web urgently needs to be fixed on both OS's, Ubuntu 24.10 and 25.04!)
Looks like the same exact problem on Ubuntu 25.04 pre-releases:
Test URL: https://www.youtube.com/watch?v=tmbZVmXyOXM
iiab-diagnostics: https://paste.centos.org/view/0cfde31d
New error with Bash 5.3.0(1)-beta (devel): Test made on Ubuntu 24.10 with and without the test plan.
root@bm2410b:~# tail -f /var/log/xklb.log
2024-11-12 11:25:03 - [Info] Using yt-dlp 2024.11.04
2024-11-12 11:25:03 - [Info] Running xklb command: lb tubeadd '/library/calibre-web/xklb-metadata.db' 'https://www.youtube.com/watch?v=Wver27ebRWU' --force -vv
2024-11-12 11:25:03 - [Debug] library v3.0.024 :: /root/.local/share/pipx/venvs/xklb/bin
2024-11-12 11:25:03 - [Debug] ['/usr/local/bin/lb', 'tubeadd', '/library/calibre-web/xklb-metadata.db', 'https://www.youtube.com/watch?v=Wver27ebRWU', '--force', '-vv']
2024-11-12 11:25:03 - [Debug] {'force': True, 'paths': ['https://www.youtube.com/watch?v=Wver27ebRWU']}
2024-11-12 11:25:03 - [Debug] {'noprogress': True, 'skip_download': True, 'lazy_playlist': True, 'extract_flat': True, 'no_check_certificate': True, 'ignore_no_formats_error': True, 'skip_playlist_after_errors': 21, 'playlistend': 20000, 'force': True}
2024-11-12 11:25:03 - [Debug] yt-dlp initialized 0.0813
2024-11-12 11:25:03 - [Info] [youtube] Extracting URL: https://www.youtube.com/watch?v=Wver27ebRWU
2024-11-12 11:25:03 - [Info] [youtube] Wver27ebRWU: Downloading webpage
2024-11-12 11:25:04 - [Info] [youtube] Wver27ebRWU: Downloading ios player API JSON
2024-11-12 11:25:04 - [Info] [youtube] Wver27ebRWU: Downloading mweb player API JSON
2024-11-12 11:25:04 - [Info] [youtube] Wver27ebRWU: Downloading player 6f0e2ed6
2024-11-12 11:25:06 - [Debug] no such table: media
2024-11-12 11:25:06 - [Debug] Importing playlist-less media https://www.youtube.com/watch?v=Wver27ebRWU
2024-11-12 11:25:06 - [Debug] media.playlist_media_add 2.5361
2024-11-12 11:25:07 - [Debug] ydl.extract_info done 0.7216
2024-11-12 11:25:07 - [Debug]
2024-11-12 11:25:07 - [Debug] Optimizing database
2024-11-12 11:25:07 - [Debug] Processing table: media
2024-11-12 11:25:07 - [Debug] Transforming column order: ['id', 'size', 'duration', 'time_uploaded', 'time_created', 'time_modified', 'time_deleted', 'time_downloaded', 'fps', 'view_count', 'path', 'webpath', 'extractor_id']
2024-11-12 11:25:07 - [Debug] Creating index: id
2024-11-12 11:25:07 - [Debug] Creating index: size
2024-11-12 11:25:07 - [Debug] Creating index: duration
2024-11-12 11:25:07 - [Debug] Creating index: time_uploaded
2024-11-12 11:25:07 - [Debug] Creating index: time_created
2024-11-12 11:25:07 - [Debug] Creating index: time_modified
2024-11-12 11:25:07 - [Debug] Creating index: time_deleted
2024-11-12 11:25:07 - [Debug] Creating index: time_downloaded
2024-11-12 11:25:07 - [Debug] Creating index: fps
2024-11-12 11:25:07 - [Debug] Creating index: view_count
2024-11-12 11:25:07 - [Debug] Creating index: live_status
2024-11-12 11:25:07 - [Debug] Creating index: uploader
2024-11-12 11:25:07 - [Debug] Creating index: path
2024-11-12 11:25:07 - [Debug] Creating fts index: ['path', 'webpath', 'title']
2024-11-12 11:25:07 - [Debug] Processing table: captions
2024-11-12 11:25:07 - [Debug] Creating index: media_id
2024-11-12 11:25:07 - [Debug] Creating index: time
2024-11-12 11:25:07 - [Debug] Creating fts index: ['text']
2024-11-12 11:25:07 - [Debug] Running VACUUM
2024-11-12 11:25:07 - [Debug] Running ANALYZE
2024-11-12 11:25:07 - [Info]
2024-11-12 11:25:07 - [Info] lb-wrapper's xklb command (tubeadd) completed successfully.
2024-11-12 11:25:07 - [Info] Using yt-dlp 2024.11.04
2024-11-12 11:25:07 - [Info] Running xklb command: lb dl '/library/calibre-web/xklb-metadata.db' --video --search 'https://www.youtube.com/watch?v=Wver27ebRWU' --format best --format-sort 'tbr~1000' --write-thumbnail --subs --live --live-from-start -o '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
2024-11-12 11:25:08 - [Debug] library v3.0.024 :: /root/.local/share/pipx/venvs/xklb/bin
2024-11-12 11:25:08 - [Debug] ['/usr/local/bin/lb', 'dl', '/library/calibre-web/xklb-metadata.db', '--video', '--search', 'https://www.youtube.com/watch?v=Wver27ebRWU', '--format', 'best', '--format-sort', 'tbr~1000', '--write-thumbnail', '--subs', '--live', '--live-from-start', '-o', '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s', '-vv']
2024-11-12 11:25:08 - [Debug] {'include': ['https://www.youtube.com/watch?v=Wver27ebRWU'], 'subs': True, 'live': True, 'paths': ['best']}
2024-11-12 11:25:08 - [Debug] Skipping non-existent file best
2024-11-12 11:25:08 - [Debug] {'time_modified': 0, 'time_deleted': 0, 'download_attempts': 0}
2024-11-12 11:25:08 - [Debug] {'id': 1, 'path': 'https://www.youtube.com/watch?v=Wver27ebRWU', 'title': '3 reasons why medications are so expensive in the US - Kiah Williams', 'duration': 320, 'time_created': 1731428706, 'size': 0, 'time_modified': 0, 'time_downloaded': 0, 'time_deleted': 0, 'error': None, 'extractor_key': 'Playlist-less media'}
2024-11-12 11:25:08 - [Debug] {'noprogress': True, 'lazy_playlist': True, 'noplaylist': True, 'no_check_certificate': True, 'ignore_no_formats_error': True, 'skip_playlist_after_errors': 21, 'playlistend': 20000, 'extractor_args': {'youtube': {'skip': ['authcheck']}}, 'check_formats': 'selected', 'logger': <xklb.createdb.tube_backend.download.<locals>.DictLogger object at 0x770995dd9ac0>, 'postprocessors': [{'key': 'FFmpegMetadata'}, {'key': 'FFmpegEmbedSubtitle'}], 'playlist_items': '1', 'extractor_retries': 3, 'retries': 12, 'retry_sleep_functions': {'extractor': <function download.<locals>.<lambda> at 0x770995de9940>, 'http': <function download.<locals>.<lambda> at 0x770995de99e0>, 'fragment': <function download.<locals>.<lambda> at 0x770995de9a80>}, 'outtmpl': {'default': '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s'}, 'progress_hooks': [<function download.<locals>.<lambda> at 0x770995de9b20>], 'subtitlesformat': 'srt/best', 'writesubtitles': True, 'writeautomaticsub': True, 'format': '--format-sort', 'writethumbnail': True, 'live_from_start': True, 'include': ['https://www.youtube.com/watch?v=Wver27ebRWU'], 'subs': True, 'live': True}
2024-11-12 11:25:08 - [Debug] [yt-dlp]: Downloading https://www.youtube.com/watch?v=Wver27ebRWU
2024-11-12 11:25:11 - [Debug] [https://www.youtube.com/watch?v=Wver27ebRWU]: yt-dlp ERROR: [youtube] Wver27ebRWU: No video formats found!
2024-11-12 11:25:11 - [Debug] [https://www.youtube.com/watch?v=Wver27ebRWU]: yt-dlp returned no info
2024-11-12 11:25:11 - [Debug] [https://www.youtube.com/watch?v=Wver27ebRWU]: Unrecoverable error matched. ERROR: [youtube] Wver27ebRWU: No video formats found!
2024-11-12 11:25:11 - [Debug] Extra media data {'time_modified': 0, 'time_downloaded': 0, 'time_deleted': 0}
Why this matters? That means the command was parsed correctly. The above error is yt-dlp related. I am going to test with the version shipped with Ubuntu 24.10 now.
Update: Same error message encountered on Bash 5.2.32(1)-release.
Good news at last. Single video downloaded successfully on Ubuntu 24.10 when option --search
is omitted in the download command (lb dl '/library/calibre-web/xklb-metadata.db' --video 'https://www.youtube.com/watch?v=Wver27ebRWU' --format best --format-sort 'tbr~1000' --write-thumbnail --subs --live --live-from-start -o '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
)
What next?
--search
option was introduced in https://github.com/iiab/calibre-web/pull/124/commits/64257d2743be95b4576f1230288f89a8b2fcd5b8 because without it all failed videos would be added to the list of requested videos. I am going to check whether leaving it out or not will make a difference.
Good news at last. Single video downloaded successfully on Ubuntu 24.10 when option --search is omitted in the download command
Great the root cause is starting to emerge!
Ubuntu 24.10 was released more than a month ago so hopefully this'll be solved soon!
Just another test to confirm this is still failing on the latest (and fully apt-updated) Ubuntu 25.04 pre-release:
Test URL: https://www.youtube.com/shorts/dZXJnmhGd2k
iiab-diagnostics https://paste.centos.org/view/18f85c43
What next?
--search
option was introduced in 64257d2 because without it all failed videos would be added to the list of requested videos. I am going to check whether leaving it out or not will make a difference.
Videos will download when --search option is not used, but this makes all failed videos from previous runs eligible for current downloading session.
Let's consider this database: https://dbhub.io/deldesir/xklb-metadata-test.db. Some videos are not downloaded. They are recognized by their path
values in table media
being URLs instead of actual paths.
Now, when the playlist of 4 videos https://www.youtube.com/playlist?list=PL1mGU3fBqrV9yoNCE4YNJFsDzBg8Np68A is submitted for download, notice how the database changed: https://dbhub.io/deldesir/xklb-metadata-test2.db. What happened is 14 videos were downloaded instead of the 4 requested. Here is the log:
Hi Blondel,
From the log it seems like the wrapper script is calling lb dl
14 times:
2024-11-14 12:41:54 - [Info] Running xklb command: lb tubeadd '/library/calibre-web/xklb-metadata.db' 'https://www.youtube.com/playlist?list=PL1mGU3fBqrV9yoNCE4YNJFsDzBg8Np68A' --force -vv
2024-11-14 12:41:56 - [Info] Running xklb command: lb tubeadd '/library/calibre-web/xklb-metadata.db' 'https://www.youtube.com/watch?v=tmwIhOCXLrQ' --force -vv
2024-11-14 12:41:59 - [Info] Running xklb command: lb tubeadd '/library/calibre-web/xklb-metadata.db' 'https://www.youtube.com/watch?v=YcUDBgYodIE' --force -vv
2024-11-14 12:42:02 - [Info] Running xklb command: lb tubeadd '/library/calibre-web/xklb-metadata.db' 'https://www.youtube.com/watch?v=P9VdyPbbzlI' --force -vv
2024-11-14 12:42:05 - [Info] Running xklb command: lb tubeadd '/library/calibre-web/xklb-metadata.db' 'https://www.youtube.com/watch?v=ubR8rEgSZSU' --force -vv
2024-11-14 12:42:08 - [Info] Running xklb command: lb tubeadd '/library/calibre-web/xklb-metadata.db' 'https://www.youtube.com/watch?v=PaxVCsnox_4' --force -vv
2024-11-14 12:42:12 - [Info] Running xklb command: lb tubeadd '/library/calibre-web/xklb-metadata.db' 'https://www.youtube.com/watch?v=9WvoGlQ7zH8' --force -vv
2024-11-14 12:42:15 - [Info] Running xklb command: lb tubeadd '/library/calibre-web/xklb-metadata.db' 'https://www.youtube.com/watch?v=_KhYRqozTDE' --force -vv
2024-11-14 12:42:18 - [Info] Running xklb command: lb tubeadd '/library/calibre-web/xklb-metadata.db' 'https://www.youtube.com/watch?v=2uTq2vtyTA0' --force -vv
2024-11-14 12:42:22 - [Info] Running xklb command: lb tubeadd '/library/calibre-web/xklb-metadata.db' 'https://www.youtube.com/watch?v=zFI-MG5PEtw' --force -vv
2024-11-14 12:42:25 - [Info] Running xklb command: lb tubeadd '/library/calibre-web/xklb-metadata.db' 'https://www.youtube.com/watch?v=rm10ifaufys' --force -vv
2024-11-14 12:42:29 - [Info] Running xklb command: lb tubeadd '/library/calibre-web/xklb-metadata.db' 'https://www.youtube.com/watch?v=O0i7-DsUUm0' --force -vv
2024-11-14 12:42:32 - [Info] Running xklb command: lb tubeadd '/library/calibre-web/xklb-metadata.db' 'https://www.youtube.com/watch?v=EBQJPyB4OVs' --force -vv
2024-11-14 12:42:36 - [Info] Running xklb command: lb tubeadd '/library/calibre-web/xklb-metadata.db' 'https://www.youtube.com/watch?v=nL0ujGxgDTU' --force -vv
2024-11-14 12:42:39 - [Info] Running xklb command: lb tubeadd '/library/calibre-web/xklb-metadata.db' 'https://www.youtube.com/watch?v=Q7Ya9R95HLE' --force -vv
2024-11-14 12:42:42 - [Info] Running xklb command: lb dl '/library/calibre-web/xklb-metadata.db' --video 'https://www.youtube.com/watch?v=2uTq2vtyTA0' --format best --format-sort 'tbr~1000' --write-thumbnail --subs --live --live-from-start -o '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
2024-11-14 12:42:52 - [Info] Running xklb command: lb dl '/library/calibre-web/xklb-metadata.db' --video 'https://www.youtube.com/watch?v=PaxVCsnox_4' --format best --format-sort 'tbr~1000' --write-thumbnail --subs --live --live-from-start -o '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
2024-11-14 12:43:02 - [Info] Running xklb command: lb dl '/library/calibre-web/xklb-metadata.db' --video 'https://www.youtube.com/watch?v=zFI-MG5PEtw' --format best --format-sort 'tbr~1000' --write-thumbnail --subs --live --live-from-start -o '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
2024-11-14 12:43:12 - [Info] Running xklb command: lb dl '/library/calibre-web/xklb-metadata.db' --video 'https://www.youtube.com/watch?v=EBQJPyB4OVs' --format best --format-sort 'tbr~1000' --write-thumbnail --subs --live --live-from-start -o '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
2024-11-14 12:43:25 - [Info] Running xklb command: lb dl '/library/calibre-web/xklb-metadata.db' --video 'https://www.youtube.com/watch?v=O0i7-DsUUm0' --format best --format-sort 'tbr~1000' --write-thumbnail --subs --live --live-from-start -o '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
2024-11-14 12:43:38 - [Info] Running xklb command: lb dl '/library/calibre-web/xklb-metadata.db' --video 'https://www.youtube.com/watch?v=_KhYRqozTDE' --format best --format-sort 'tbr~1000' --write-thumbnail --subs --live --live-from-start -o '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
2024-11-14 12:43:48 - [Info] Running xklb command: lb dl '/library/calibre-web/xklb-metadata.db' --video 'https://www.youtube.com/watch?v=YcUDBgYodIE' --format best --format-sort 'tbr~1000' --write-thumbnail --subs --live --live-from-start -o '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
2024-11-14 12:43:57 - [Info] Running xklb command: lb dl '/library/calibre-web/xklb-metadata.db' --video 'https://www.youtube.com/watch?v=P9VdyPbbzlI' --format best --format-sort 'tbr~1000' --write-thumbnail --subs --live --live-from-start -o '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
2024-11-14 12:44:07 - [Info] Running xklb command: lb dl '/library/calibre-web/xklb-metadata.db' --video 'https://www.youtube.com/watch?v=nL0ujGxgDTU' --format best --format-sort 'tbr~1000' --write-thumbnail --subs --live --live-from-start -o '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
2024-11-14 12:44:17 - [Info] Running xklb command: lb dl '/library/calibre-web/xklb-metadata.db' --video 'https://www.youtube.com/watch?v=Q7Ya9R95HLE' --format best --format-sort 'tbr~1000' --write-thumbnail --subs --live --live-from-start -o '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
2024-11-14 12:44:25 - [Info] Running xklb command: lb dl '/library/calibre-web/xklb-metadata.db' --video 'https://www.youtube.com/watch?v=9WvoGlQ7zH8' --format best --format-sort 'tbr~1000' --write-thumbnail --subs --live --live-from-start -o '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
2024-11-14 12:44:37 - [Info] Running xklb command: lb dl '/library/calibre-web/xklb-metadata.db' --video 'https://www.youtube.com/watch?v=ubR8rEgSZSU' --format best --format-sort 'tbr~1000' --write-thumbnail --subs --live --live-from-start -o '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
2024-11-14 12:44:48 - [Info] Running xklb command: lb dl '/library/calibre-web/xklb-metadata.db' --video 'https://www.youtube.com/watch?v=rm10ifaufys' --format best --format-sort 'tbr~1000' --write-thumbnail --subs --live --live-from-start -o '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
2024-11-14 12:44:55 - [Info] Running xklb command: lb dl '/library/calibre-web/xklb-metadata.db' --video 'https://www.youtube.com/watch?v=tmwIhOCXLrQ' --format best --format-sort 'tbr~1000' --write-thumbnail --subs --live --live-from-start -o '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).170B_%(view_count)3.2D_[%(id).64B].%(ext)s' -vv
Maybe this is the cause of the issue?
This is the way I use library
and the way I recommend running it:
One process adds URLS:
lb tubeadd ~/lb/video.db $URLS
Another process downloads:
lb dl --video ~/lb/video.db --prefix ~/d/dump/video/
You could have lb dl
as a systemd service with Restart=always
and RuntimeMaxSec=6h
. They can be concurrent processes because xklb enables WAL mode
Today I learned. Now the challenge is to get the dynamic polling for every video added. This is reported in a Tasks page in real time.
The current flow is as follow:
Task MetadataExtract is called:
Task Download is called
I expect with the alternative approach of running lb dl as a service, I'll be able to poll xklb log for every video separately. Will do some experiments and let you know how this went.
I don't yet see how a systemd service improves the situation?
Both a YouTube Short video (https://www.youtube.com/shorts/M_Zt5K057xs) and a regular YouTube video https://youtu.be/zRM2AnwNY20 (The Wild Story of the Taum Sauk Dam Failure) failed to download repeatedly, to this Ubuntu 24.10 VM.
Rebooting did not help:
iiab-diagnostics: https://paste.centos.org/view/a53f9107 (that'll disappear in 4 weeks!)
/var/log/calibre-web.log
mean?Full excerpt (Lines 1122-1127) below:
/var/log/xklb.log
relevant/meaningful ❓@deldesir suggests this is what happened:
RECAP QUESTIONS:
@deldesir were you able to learn anything by logging into the
.0.10
VM directly?Is this issue the same as #212 ? Or at least similar?? Or not at all?!
What short-term CLI hack or workaround gets IIAB Calibre-Web machines like this able to download again? So that can quickly be written into https://github.com/iiab/calibre-web/wiki
What long-term fix or reimplementation is most appropriate — so that this stop happening entirely ?
Finally, how can we make real progress in removing errors like
An error occurred during the subprocess execution: 'NoneType' object is not subscriptable
from educators' web UI — in favor of much more specific and human-readable explanation in 2025?202 begins to address this serious lack non-specific and non-readable
Tasks
view messages — but is a new ticket needed to make sure this is solved properly?