Open holta opened 3 weeks ago
Just FYI the "Metadata Fetch" stage has now been running for more than an hour, and its Progress column currently shows 70%.
The "Metadata Fetch" stage failed after ~86 minutes — is this related to an earlier video stuck in some queue possibly?
Possibly related:
Just FYI regardless of the root cause for the (failed) ~86 minute "Metadata Fetch", videos that keep reappearing in "Tasks" view (e.g. https://www.youtube.com/watch?v=n_iRE9alO44) should be handled, to explain what's happening.
FWIW in this one particular case, browsing to https://www.youtube.com/watch?v=n_iRE9alO44 shows "This live stream recording is not available.", "No views" and "Streamed 1 year ago" — error messages that in an ideal world might become more visible via "Tasks" view?
1) @deldesir in the short term, can you suggest a manual hack to clear out problematic videos that are "trapped" and keep reappearing in "Tasks" view? 2) Can we delineate or document what/all kinds of problematic videos keep reappearing in "Tasks" view? (Regardless whether these are merely "visual clutter" — or more serious clogging / blocking of other downloads!) 3) As soon as we can, educators/families will need clear explanations of what went wrong. (With actionable hints/suggestions in "Tasks" view, to shine a bright light on emerging problems in jargon-free language, wherever we can anyway!)
Can I have access to your xklb-metadata? https://www.youtube.com/watch?v=n_iRE9alO44 looks out-of-norm and might not have its error recorded in media.error
.
UPDATE: Please ignore the request. I'll try to download it myself.
suggest a manual hack to clear out problematic videos that are "trapped" and keep reappearing in "Tasks" view?
Just FYI installing the very latest IIAB Calibre-Web onto LRN2 did not help:
Trying to "Download to IIAB" https://www.youtube.com/playlist?list=PLybg94GvOJ9ELZEe9s2NXTKr41Yedbw7M is currently working through a ~100 minute "Metadata Fetch".
root@lrn2:~# tail -700 /var/log/xklb.log | pastebinit -b dpaste.com https://dpaste.com/8C28P5R3V
@deldesir: at first I suspected Calculus playlist https://www.youtube.com/playlist?list=PLybg94GvOJ9ELZEe9s2NXTKr41Yedbw7M might be pulling in more than its official 35 videos?
As if you watch the running of tail -f /var/log/xklb.log
growing, you'll see what appears to be many hundreds of videos being polled (a new one every 5-10 seconds) during "Metadata Fetch" — example:
2024-06-17 18:29:32 - [Debug] yt-dlp initialized 0.2313
2024-06-17 18:29:32 - [Info] [youtube] Extracting URL: https://www.youtube.com/watch?v=45lckolNRm8
2024-06-17 18:29:32 - [Info] [youtube] 45lckolNRm8: Downloading webpage
2024-06-17 18:29:33 - [Info] [youtube] 45lckolNRm8: Downloading ios player API JSON
2024-06-17 18:29:36 - [Debug] Importing playlist-less media https://www.youtube.com/watch?v=45lckolNRm8
2024-06-17 18:29:36 - [Debug] media.playlist_media_add 3.7105
2024-06-17 18:29:36 - [Info]
2024-06-17 18:29:36 - [Debug] ydl.extract_info done 0.0167
2024-06-17 18:29:36 - [Info] lb-wrapper's xklb command (tubeadd) completed successfully.
2024-06-17 18:29:37 - [Info] Using yt-dlp 2024.05.27
2024-06-17 18:29:37 - [Info] Running xklb command: lb tubeadd /library/calibre-web/xklb-metadata.db https://www.youtube.com/watch?v=hUiIkiQGiIM --force -vv
2024-06-17 18:29:38 - [Debug] library v2.8.065
2024-06-17 18:29:38 - [Debug] ['/usr/local/bin/lb', 'tubeadd', '/library/calibre-web/xklb-metadata.db', 'https://www.youtube.com/watch?v=hUiIkiQGiIM', '--force', '-vv']
2024-06-17 18:29:38 - [Debug] {'force': True, 'paths': ['https://www.youtube.com/watch?v=hUiIkiQGiIM']}
2024-06-17 18:29:38 - [Debug] {'noprogress': True, 'skip_download': True, 'lazy_playlist': True, 'extract_flat': True, 'no_check_certificate': True, 'ignore_no_formats_error': True, 'skip_playlist_after_errors': 21, 'playlistend': 20000, 'force': True}
But that theory proved false: as https://www.youtube.com/watch?v=45lckolNRm8 is "PARLONS EN Les études à l’étranger. Expérience des anciens étudiants congolais en Chine" from YouTube channel TOP CONGO FM — it seems these hundreds of videos must somehow be stuck within IIAB Calibre-Web — it keeps trying to reprocess or redownload them somehow...
A quick way to confirm this is to use a clean xklb-metadata.db.
A quick way to confirm this is to use a clean xklb-metadata.db.
Good idea. Feel free to run...
systemctl stop calibre-web
cd /library/calibre-web
mv xklb-metadata.db xklb-metadata.db.OLD2
systemctl start calibre-web
...and then try again to "Download to IIAB" https://www.youtube.com/playlist?list=PLybg94GvOJ9ELZEe9s2NXTKr41Yedbw7M on LRN2?
"Minimal Reproducer" URL?
Any Minimal Reproducer URL that "pollutes" xklb-metadata.db, would be incredibly useful for testing — and to help us all understand this better — if we can identify one or several such URLs! :pray:
I confirm "lb tubeadd" is taking longer to process the videos. I don't see any pollution with irrelevant videos.
Update: pollution happening for me with 3-videos playlist https://www.youtube.com/playlist?list=PLJicmE8fK0Eg8cBExYSr9r3tzg9ZQo3gD. Using a clean db is a temporary measure to get around this.
Trying to download playlist https://www.youtube.com/playlist?list=PLybg94GvOJ9ELZEe9s2NXTKr41Yedbw7M on LRN2 is going extremely slowly. The "Progress" for "Metadata Fetch" has increased by 1 percentage point every minute or so, as if this initial stage alone will take about 100 minutes:
The playlist's 35 videos are each about 5-12min long.
Any idea why the "Metadata Fetch" stage is going so slowly? Hopefully the
/var/log/xklb.log
logs below provide hints to help explain?