search

iiab / calibre-web

:books: Web app for browsing, reading and downloading eBooks stored in a Calibre database
GNU General Public License v3.0
3 stars 4 forks source link

"Download to IIAB" failures must be shown in "Tasks" view with explanation [e.g. "Media check failed (will try again later)" "Failed reading header" with .mhtml] [e.g. `YouTube_Movies_TV` channel encrypted w/ browser-based DRM?] #85

Closed holta closed 2 months ago

holta commented 8 months ago

Summary:

1) Calibre-Web's "Tasks" view must acknowledge all "Download to IIAB" and "Upload" button failures. 2) Regardless what the error appears to be, some version of the error must be shown in "Tasks" view. Calibre-Web's Python needs to be revised to surface (display) the most meaningful errors possible. 3) EXAMPLE: The 97min video below does not currently work with "Download to IIAB": https://youtu.be/HPkDFc8hq5c

root@box:/library/downloads/calibre-web# tree -h
[4.0K]  .
├── [4.0K]  Youtube
│   └── [4.0K]  YouTube_Movies_TV
│       ├── [2.5M]  Mean_Girls_3.10M_[HPkDFc8hq5c].mhtml
│       └── [228K]  Mean_Girls_3.10M_[HPkDFc8hq5c].webp
└── [132K]  survey.db

3 directories, 3 files
root@box:~# tree /library/calibre-web/
/library/calibre-web/
├── config
│   └── app.db
├── metadata.db
└── metadata_db_prefs_backup.json

2 directories, 3 files
root@box:~# lb --version
2.3.001
root@box:~# yt-dlp --version
2023.12.30
root@box:~# hostname -I
10.181.233.173 192.168.0.235 10.8.0.58 fd42:fb6b:9c93:d2f1:5054:ff:fe7f:2f9d
root@box:~# hostname -A
t5.lxd box box t5.lxd
root@box:~# systemctl status calibre-web
● calibre-web.service - Calibre-Web
     Loaded: loaded (/etc/systemd/system/calibre-web.service; enabled; preset: enabled)
     Active: active (running) since Tue 2024-01-02 22:10:33 EST; 4min 56s ago
   Main PID: 3942 (python3)
      Tasks: 4 (limit: 2300)
     Memory: 173.5M
        CPU: 7.367s
     CGroup: /system.slice/calibre-web.service
             └─3942 /usr/local/calibre-web-py3/bin/python3 /usr/local/calibre-web-py3/cps.py -p /library/calibre-web/config/app.db

Jan 02 22:10:33 box systemd[1]: Started calibre-web.service - Calibre-Web.
root@box:~# cat /var/log/calibre-web.log
[2024-01-02 22:10:34,597]  INFO {cps:168} *** "greenlet" version does not meet the requirements. Should: <2.1.0, Found: 3.0.3, please consider installing required version ***
[2024-01-02 22:10:34,597]  INFO {cps:177} Starting Calibre Web...
[2024-01-02 22:10:34,603] DEBUG {cps.services:26} Cannot import goodreads, showing authors-metadata will not work: No module named 'betterreads'
[2024-01-02 22:10:34,605] DEBUG {cps.services:34} Cannot import simpleldap, logging in with ldap will not work: No module named 'flask_simpleldap'
[2024-01-02 22:10:34,637] DEBUG {cps.services:49} Cannot import gmail, sending books via Gmail Oauth2 Verification will not work: No module named 'google_auth_oauthlib'
[2024-01-02 22:10:34,652]  WARN {py.warnings:109} /usr/local/calibre-web-py3/lib/python3.11/site-packages/flask_limiter/extension.py:336: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
  warnings.warn(

[2024-01-02 22:10:34,780] DEBUG {cps.gdriveutils:81} Cannot import pydrive, httplib2, using gdrive will not work: No module named 'pydrive'
[2024-01-02 22:10:35,026]  INFO {apscheduler.scheduler:181} Scheduler started
[2024-01-02 22:10:35,026]  INFO {apscheduler.scheduler:895} Added job "end scheduled task" to job store "default"
[2024-01-02 22:10:35,140] DEBUG {cps.comic:46} Cannot import comicapi, extracting comic metadata will not work: No module named 'comicapi'
[2024-01-02 22:10:35,141] DEBUG {cps.comic:53} Cannot import rarfile, extracting cover files from rar files will not work: No module named 'rarfile'
[2024-01-02 22:10:35,141] DEBUG {cps.comic:59} Cannot import py7zr, extracting cover files from CB7 files will not work: No module named 'py7zr'
[2024-01-02 22:10:35,211] DEBUG {cps.admin:81} Cannot import Flask-Dance, login with Oauth will not work: No module named 'flask_dance'
[2024-01-02 22:10:35,215] DEBUG {cps.gdrive:44} Cannot import googleapiclient, using GDrive will not work: No module named 'googleapiclient'
[2024-01-02 22:10:35,254] DEBUG {cps.search_metadata:62} Import error for metadata source: scholar - No module named 'scholarly'
[2024-01-02 22:10:35,262] DEBUG {cps.search_metadata:62} Import error for metadata source: douban - No module named 'html2text'
[2024-01-02 22:10:35,270] DEBUG {cps.search_metadata:62} Import error for metadata source: lubimyczytac - No module named 'dateutil'
[2024-01-02 22:10:35,271] DEBUG {cps.search_metadata:62} Import error for metadata source: amazon - No module named 'bs4'
[2024-01-02 22:10:35,342]  INFO {cps.server:267} Starting Tornado server on :8083
[2024-01-02 22:12:28,769] DEBUG {cps.web:1410} You are now logged in as: 'Admin'
[2024-01-02 22:12:28,789] DEBUG {cps.ub:82} Login and store session : 8b1f3e026c99c26da2710a803e79fd8c3731814f5a5be1b048a9411edb469756fa361953799483455066116aa7c0ded030ea40dafe80a1633b5985cd2bc19327
[2024-01-02 22:12:28,790] DEBUG {cps.ub:84} Found stored session: 8b1f3e026c99c26da2710a803e79fd8c3731814f5a5be1b048a9411edb469756fa361953799483455066116aa7c0ded030ea40dafe80a1633b5985cd2bc19327
[2024-01-02 22:14:58,237] DEBUG {cps.services.worker:91} Add Task for user: Admin - Download https://www.youtube.com/watch?v=HPkDFc8hq5c
[2024-01-02 22:14:58,241]  INFO {cps.tasks.download:28} Starting download task for URL: https://www.youtube.com/watch?v=HPkDFc8hq5c
[2024-01-02 22:14:58,242]  INFO {cps.tasks.download:37} Subprocess args: ['lb-wrapper', 'https://www.youtube.com/watch?v=HPkDFc8hq5c']
[2024-01-02 22:14:58,263]  INFO {cps.tasks.download:50} /usr/local/bin/lb

[2024-01-02 22:15:07,115]  INFO {cps.editbooks:382} Requested files list: ['https://www.youtube.com/watch?v=HPkDFc8hq5c']
[2024-01-02 22:15:07,115] ERROR {cps:1414} Exception on /meta [GET]
Traceback (most recent call last):
  File "/usr/local/calibre-web-py3/lib/python3.11/site-packages/flask/app.py", line 2190, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/calibre-web-py3/lib/python3.11/site-packages/flask/app.py", line 1486, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/calibre-web-py3/lib/python3.11/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/calibre-web-py3/lib/python3.11/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/calibre-web-py3/cps/editbooks.py", line 471, in meta
    if move_mediafile(requested_files, current_user_name, shelf_title):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/calibre-web-py3/cps/editbooks.py", line 384, in move_mediafile
    requested_file = open(requested_file, "rb")
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'https://www.youtube.com/watch?v=HPkDFc8hq5c'
[2024-01-02 22:15:07,125] ERROR {cps.tasks.download:81} Failed to send the list of requested files to http://192.168.0.235/books/meta
root@box:~# cat /var/log/xklb.log
2024-01-02 22:14:58 - Running command: lb tubeadd /library/downloads/calibre-web/survey.db https://www.youtube.com/watch?v=HPkDFc8hq5c --verbose && lb dl /library/downloads/calibre-web/survey.db --prefix /library/downloads/calibre-web --write-thumbnail --format best --format-sort 'tbr~1000' --video https://www.youtube.com/watch?v=HPkDFc8hq5c --verbose
2024-01-02 22:15:07 - Download completed successfully.
root@box:~# lb dl /library/downloads/calibre-web/survey.db --prefix /library/downloads/calibre-web --write-thumbnail --format best --format-sort 'tbr~1000' --video https://www.youtube.com/watch?v=HPkDFc8hq5c -vv
library v2.3.001
['/usr/local/bin/lb', 'dl', '/library/downloads/calibre-web/survey.db', '--prefix', '/library/downloads/calibre-web', '--write-thumbnail', '--format', 'best', '--format-sort', 'tbr~1000', '--video', 'https://www.youtube.com/watch?v=HPkDFc8hq5c', '-vv']
{'profile': 'video', 'extractor_config': {}, 'extra_media_data': {}, 'extra_playlist_data': {}, 'prefix': '/library/downloads/calibre-web', 'sort': 'm.path like "http%"\n        , m.title IS NOT NULL desc\n        , m.path\n        , random()', 'where': [], 'include': [], 'exclude': [], 'limit': 7200, 'retry_delay': '14 days', 'db': <Database <sqlite3.Connection object at 0x7f70976e63e0>>, 'verbose': 2, 'database': '/library/downloads/calibre-web/survey.db', 'playlists': [], 'defaults': ['sort', 'limit'], 'unk': ['--write-thumbnail', '--format', 'best', '--format-sort', 'tbr~1000', 'https://www.youtube.com/watch?v=HPkDFc8hq5c'], 'action': 'download'}
None
{'path': 'https://www.youtube.com/watch?v=HPkDFc8hq5c', 'title': 'Mean Girls', 'duration': 5810, 'time_created': 1704254287, 'size': 0, 'time_modified': 0, 'time_downloaded': 0, 'time_deleted': 0, 'extractor_key': 'Playlist-less media'}
{'noprogress': True, 'lazy_playlist': True, 'noplaylist': True, 'no_check_certificate': True, 'ignore_no_formats_error': True, 'skip_playlist_after_errors': 21, 'playlistend': 20000, 'extractor_args': {'youtube': {'skip': ['authcheck']}}, 'logger': <xklb.tube_backend.download.<locals>.DictLogger object at 0x7f70975a5610>, 'postprocessors': [{'key': 'FFmpegMetadata'}, {'key': 'FFmpegEmbedSubtitle'}], 'restrictfilenames': True, 'playlist_items': '1', 'extractor_retries': 3, 'retries': 12, 'retry_sleep_functions': {'extractor': <function download.<locals>.<lambda> at 0x7f7096ba5b20>, 'http': <function download.<locals>.<lambda> at 0x7f7096ba5bc0>, 'fragment': <function download.<locals>.<lambda> at 0x7f7096ba5c60>}, 'outtmpl': {'default': '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).200B_%(view_count)3.2D_[%(id).60B].%(ext)s', 'chapter': '/library/downloads/calibre-web/%(extractor_key,extractor)s/%(uploader,uploader_id)s/%(title).200B_%(section_number)03d_%(section_title)s_%(view_count)3.2D_[%(id).60B].%(ext)s'}, 'subtitlesformat': 'srt/best', 'format': 'best', 'format_sort': ['tbr~1000'], 'writethumbnail': True, '_warnings': ['"-f best" selects the best pre-merged format which is often not the best option.\n         To let yt-dlp download and merge the best available formats, simply do not pass any format selection.\n         If you know what you are doing and want only the best pre-merged format, use "-f b" instead to suppress this warning']}
[https://www.youtube.com/watch?v=HPkDFc8hq5c]: Media check failed (will try again later)
Failed reading header. /library/downloads/calibre-web/Youtube/YouTube_Movies_TV/Mean_Girls_3.11M_[HPkDFc8hq5c].mhtml
(b'{\n\n}\n', b'ffprobe version 6.1-5ubuntu1 Copyright (c) 2007-2023 the FFmpeg developers\n  built with gcc 13 (Ubuntu 13.2.0-7ubuntu1)\n  configuration: --prefix=/usr --extra-version=5ubuntu1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libharfbuzz --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librist --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --disable-sndio --enable-libjxl --enable-pocketsphinx --enable-librsvg --enable-libvpl --disable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libsvtav1 --enable-libx264 --enable-libplacebo --enable-librav1e --enable-shared\n  libavutil      58. 29.100 / 58. 29.100\n  libavcodec     60. 31.102 / 60. 31.102\n  libavformat    60. 16.100 / 60. 16.100\n  libavdevice    60.  3.100 / 60.  3.100\n  libavfilter     9. 12.100 /  9. 12.100\n  libswscale      7.  5.100 /  7.  5.100\n  libswresample   4. 12.100 /  4. 12.100\n  libpostproc    57.  3.100 / 57.  3.100\n/library/downloads/calibre-web/Youtube/YouTube_Movies_TV/Mean_Girls_3.11M_[HPkDFc8hq5c].mhtml: Invalid data found when processing input\n')
Extra media data {'rows': 5, 'columns': 5, 'corruption': 50}

Similar "FileNotFoundError: [Errno 2] No such file or directory:" error with Vimeo 9 days ago:

chapmanjacobd commented 8 months ago

The problem here is that YouTube Movies (ie. the YouTube_Movies_TV channel) are encrypted. Normal YouTube videos don't have this DRM

holta commented 8 months ago

The problem here is that YouTube Movies (ie. the YouTube_Movies_TV channel) are encrypted. Normal YouTube videos don't have this DRM

That explains everything!

Thank you @chapmanjacobd !

@deldesir please find a way to surface these kinds of issues to Calibre-Web's "Tasks" view — with a clear explanation (understandable by almost everyone!)

deldesir commented 7 months ago

@deldesir please find a way to surface these kinds of issues to Calibre-Web's "Tasks" view — with a clear explanation (understandable by almost everyone!)

Fixed in #86

holta commented 6 months ago

Progress is significant in January and February! Just one example to make "Tasks" view more readable:

Still, downloads continue to fail without clear explanation — so work continues to bring clarity/sunshine onto which downloads are failing and why — a few recent examples:

deldesir commented 2 months ago

The current state leverages xklb's errors reporting. Error messages are read from xklb-metadata.db and displayed to the user in tasks' message column. Feel free to close as completed.

holta commented 2 months ago

The current state leverages xklb's errors reporting. Error messages are read from xklb-metadata.db and displayed to the user in tasks' message column. Feel free to close as completed.

Tremendous progress:

Great news that users (educators, parents, grassroots communities) can now become more independent!