MP4 files failed to upload [WebM too, for large video files?]

[deldesir]

"Upload" of MP4 files won't work on fresh install (http://sprunge.us/2OkfiI?en) that includes PR #59?

Internet in a Box

500 Internal Server Error

The server encountered an internal error and was unable to complete your request. There is an error in the application.      

Traceback (most recent call last):

  File "/usr/local/calibre-web-py3/lib/python3.11/site-packages/flask/app.py", line 2190, in wsgi_app

    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/calibre-web-py3/lib/python3.11/site-packages/flask/app.py", line 1486, in full_dispatch_request

    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/calibre-web-py3/lib/python3.11/site-packages/flask/app.py", line 1484, in full_dispatch_request

    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/calibre-web-py3/lib/python3.11/site-packages/flask/app.py", line 1469, in dispatch_request

    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/calibre-web-py3/cps/usermanagement.py", line 34, in decorated_view

    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/calibre-web-py3/cps/editbooks.py", line 72, in inner

    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^

  File "/usr/local/calibre-web-py3/cps/editbooks.py", line 263, in upload

    meta, error = file_handling_on_upload(requested_file)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/calibre-web-py3/cps/editbooks.py", line 913, in file_handling_on_upload

    meta = uploader.upload(requested_file, config.config_rarfile_location)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/calibre-web-py3/cps/uploader.py", line 372, in upload

    return process(tmp_file_path, filename_root, file_extension, rar_excecutable)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/calibre-web-py3/cps/uploader.py", line 100, in process

    if not meta.title.strip():
           ^^^^^^^^^^

AttributeError: 'NoneType' object has no attribute 'title'

Please report this issue with all related information:
        Create Issue

        Return to Home

[holta]

@deldesir is MP4 work below possibly helpful here... ?

• PR #65
• PR #69

(e.g. what should @EMG70 look for when retesting of... uploading of MP4's and WebM's?)

[EMG70]

Upload of https://vp.nyt.com/video/2023/10/23/112532_1_opdoc-redears_wg_1080p.mp4 Video 867MB (Red ears NYT) gives error "Request entity too large".Tried with all previous videos deleted and still get same error even with a 300MB MP4 video.

root@box:/# pastebinit -b sprunge.us /var/log/calibre-web.log http://sprunge.us/xmYzLj?en root@box:/# pastebinit -b sprunge.us /var/log/xklb.log http://sprunge.us/pEPBBl?en

[holta]

Upload of MP4 Video 867MB (Red ears NYT) gives error "Request entity too large".

Thanks @EMG70:

Can you paste in the original MP4 URL that you manually downloaded earlier? (To help @deldesir & others fully reproduce the problem, Thanks!)

[EMG70]

Upload of MP4 Video 867MB (Red ears NYT) gives error "Request entity too large".

Thanks @EMG70:

Can you paste in the original MP4 URL that you manually downloaded earlier? (To help @deldesir & others fully reproduce the problem, Thanks!)

This is the original video link https://vp.nyt.com/video/2023/10/23/112532_1_opdoc-redears_wg_1080p.mp4

[deldesir]

Thanks @EMG70. I am working on it. Related to https://github.com/janeczku/calibre-web/issues/1716

[holta]

I am working on it. Related to janeczku#1716

@deldesir can you investigate further, now that PR iiab/iiab#3148 is merged to change NGINX's /etc/nginx/server.conf from client_max_body_size 500M; to client_max_body_size 10000M; ?

I re-ran @EMG70's test with a completely fresh install of everything (uploading the exact same 827MiB = 867MB MP4 to Calibre-Web) and it visually showed a green bar progressing (from left to right) uploading the entire file over ~2 minutes — but then failed immediately after that with the error messages below — "Bad Gateway: File size may be too big" and "INFO {tornado.general:292} Malformed HTTP message from 127.0.0.1: Content-Length too long":

root@box:~# cat /var/log/calibre-web.log
[2023-12-21 12:31:51,493]  INFO {cps:168} *** "greenlet" version does not meet the requirements. Should: <2.1.0, Found: 3.0.2, please consider installing required version ***
[2023-12-21 12:31:51,493]  INFO {cps:177} Starting Calibre Web...
[2023-12-21 12:31:51,499] DEBUG {cps.services:26} Cannot import goodreads, showing authors-metadata will not work: No module named 'betterreads'
[2023-12-21 12:31:51,501] DEBUG {cps.services:34} Cannot import simpleldap, logging in with ldap will not work: No module named 'flask_simpleldap'
[2023-12-21 12:31:51,532] DEBUG {cps.services:49} Cannot import gmail, sending books via Gmail Oauth2 Verification will not work: No module named 'google_auth_oauthlib'
[2023-12-21 12:31:51,552]  WARN {py.warnings:109} /usr/local/calibre-web-py3/lib/python3.11/site-packages/flask_limiter/extension.py:336: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
  warnings.warn(

[2023-12-21 12:31:51,691] DEBUG {cps.gdriveutils:81} Cannot import pydrive, httplib2, using gdrive will not work: No module named 'pydrive'
[2023-12-21 12:31:51,922]  INFO {apscheduler.scheduler:181} Scheduler started
[2023-12-21 12:31:51,923]  INFO {apscheduler.scheduler:895} Added job "end scheduled task" to job store "default"
[2023-12-21 12:31:52,043] DEBUG {cps.comic:46} Cannot import comicapi, extracting comic metadata will not work: No module named 'comicapi'
[2023-12-21 12:31:52,044] DEBUG {cps.comic:53} Cannot import rarfile, extracting cover files from rar files will not work: No module named 'rarfile'
[2023-12-21 12:31:52,044] DEBUG {cps.comic:59} Cannot import py7zr, extracting cover files from CB7 files will not work: No module named 'py7zr'
[2023-12-21 12:31:52,118] DEBUG {cps.admin:81} Cannot import Flask-Dance, login with Oauth will not work: No module named 'flask_dance'
[2023-12-21 12:31:52,122] DEBUG {cps.gdrive:44} Cannot import googleapiclient, using GDrive will not work: No module named 'googleapiclient'
[2023-12-21 12:31:52,160] DEBUG {cps.search_metadata:62} Import error for metadata source: amazon - No module named 'bs4'
[2023-12-21 12:31:52,162] DEBUG {cps.search_metadata:62} Import error for metadata source: scholar - No module named 'scholarly'
[2023-12-21 12:31:52,163] DEBUG {cps.search_metadata:62} Import error for metadata source: douban - No module named 'html2text'
[2023-12-21 12:31:52,187] DEBUG {cps.search_metadata:62} Import error for metadata source: lubimyczytac - No module named 'html2text'
[2023-12-21 12:31:52,278]  INFO {cps.server:267} Starting Tornado server on :8083
[2023-12-21 12:32:53,132] DEBUG {cps.web:1410} You are now logged in as: 'Admin'
[2023-12-21 12:32:53,151] DEBUG {cps.ub:82} Login and store session : 8b1f3e026c99c26da2710a803e79fd8c3731814f5a5be1b048a9411edb469756fa361953799483455066116aa7c0ded030ea40dafe80a1633b5985cd2bc19327
[2023-12-21 12:32:53,152] DEBUG {cps.ub:84} Found stored session: 8b1f3e026c99c26da2710a803e79fd8c3731814f5a5be1b048a9411edb469756fa361953799483455066116aa7c0ded030ea40dafe80a1633b5985cd2bc19327
[2023-12-21 12:37:16,430]  INFO {tornado.general:292} Malformed HTTP message from 127.0.0.1: Content-Length too long
root@box:~#

root@box:~# cat /var/log/xklb.log
cat: /var/log/xklb.log: No such file or directory

iiab-diagnostics: http://sprunge.us/mEmz50?en

[holta]

@deldesir after merging...

• PR #75

...and then upgrading Calibre-Web as follows...

https://github.com/iiab/calibre-web/wiki#upgrading

...the result is a bit different. While this ugly pop-up still appears right after uploading 867MB over ~2 minutes...

This time /var/log/calibre-web.log appears somewhat cleaner?

root@box:~# tail -25 /var/log/calibre-web.log
[2023-12-21 13:46:20,964] DEBUG {cps.updater:329} Stable version: {'version': '0.6.22 Beta'}
[2023-12-21 14:35:27,804]  INFO {cps.server:319} webserver stop (restart=False)
[2023-12-21 14:35:27,804]  INFO {apscheduler.scheduler:212} Scheduler has been shut down
[2023-12-21 14:35:27,805]  INFO {cps.server:297} Performing shutdown of Calibre-Web
[2023-12-21 14:35:29,936]  INFO {cps:168} *** "greenlet" version does not meet the requirements. Should: <2.1.0, Found: 3.0.2, please consider installing required version ***
[2023-12-21 14:35:29,936]  INFO {cps:177} Starting Calibre Web...
[2023-12-21 14:35:29,942] DEBUG {cps.services:26} Cannot import goodreads, showing authors-metadata will not work: No module named 'betterreads'
[2023-12-21 14:35:29,942] DEBUG {cps.services:34} Cannot import simpleldap, logging in with ldap will not work: No module named 'flask_simpleldap'
[2023-12-21 14:35:29,976] DEBUG {cps.services:49} Cannot import gmail, sending books via Gmail Oauth2 Verification will not work: No module named 'google_auth_oauthlib'
[2023-12-21 14:35:29,992]  WARN {py.warnings:109} /usr/local/calibre-web-py3/lib/python3.11/site-packages/flask_limiter/extension.py:336: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
  warnings.warn(

[2023-12-21 14:35:30,082] DEBUG {cps.gdriveutils:81} Cannot import pydrive, httplib2, using gdrive will not work: No module named 'pydrive'
[2023-12-21 14:35:30,239]  INFO {apscheduler.scheduler:181} Scheduler started
[2023-12-21 14:35:30,240]  INFO {apscheduler.scheduler:895} Added job "end scheduled task" to job store "default"
[2023-12-21 14:35:30,311] DEBUG {cps.comic:46} Cannot import comicapi, extracting comic metadata will not work: No module named 'comicapi'
[2023-12-21 14:35:30,313] DEBUG {cps.comic:53} Cannot import rarfile, extracting cover files from rar files will not work: No module named 'rarfile'
[2023-12-21 14:35:30,313] DEBUG {cps.comic:59} Cannot import py7zr, extracting cover files from CB7 files will not work: No module named 'py7zr'
[2023-12-21 14:35:30,390] DEBUG {cps.admin:81} Cannot import Flask-Dance, login with Oauth will not work: No module named 'flask_dance'
[2023-12-21 14:35:30,393] DEBUG {cps.gdrive:44} Cannot import googleapiclient, using GDrive will not work: No module named 'googleapiclient'
[2023-12-21 14:35:30,416] DEBUG {cps.search_metadata:62} Import error for metadata source: amazon - No module named 'bs4'
[2023-12-21 14:35:30,416] DEBUG {cps.search_metadata:62} Import error for metadata source: scholar - No module named 'scholarly'
[2023-12-21 14:35:30,416] DEBUG {cps.search_metadata:62} Import error for metadata source: douban - No module named 'html2text'
[2023-12-21 14:35:30,421] DEBUG {cps.search_metadata:62} Import error for metadata source: lubimyczytac - No module named 'html2text'
[2023-12-21 14:35:30,498]  INFO {cps.server:267} Starting Tornado server on :8083

Bad News is that Calibre-Web now crashes...

root@box:~# systemctl status calibre-web.service
× calibre-web.service - Calibre-Web
     Loaded: loaded (/etc/systemd/system/calibre-web.service; enabled; preset: enabled)
     Active: failed (Result: oom-kill) since Thu 2023-12-21 14:39:46 EST; 6min ago
   Duration: 4min 17.653s
    Process: 7669 ExecStart=/usr/local/calibre-web-py3/bin/python3 /usr/local/calibre-web-py3/cps.py -p /library/calibre-web/config/app.db (code=killed, signal=KILL)
   Main PID: 7669 (code=killed, signal=KILL)
        CPU: 3.099s

Dec 21 14:35:29 box systemd[1]: Started calibre-web.service - Calibre-Web.
Dec 21 14:39:46 box systemd[1]: calibre-web.service: A process of this unit has been killed by the OOM killer.
Dec 21 14:39:46 box systemd[1]: calibre-web.service: Main process exited, code=killed, status=9/KILL
Dec 21 14:39:46 box systemd[1]: calibre-web.service: Failed with result 'oom-kill'.
Dec 21 14:39:46 box systemd[1]: calibre-web.service: Consumed 3.099s CPU time.

root@box:~# journalctl -u calibre-web
Dec 21 12:31:50 box systemd[1]: Started calibre-web.service - Calibre-Web.
Dec 21 14:35:27 box systemd[1]: Stopping calibre-web.service - Calibre-Web...
Dec 21 14:35:28 box systemd[1]: calibre-web.service: Deactivated successfully.
Dec 21 14:35:28 box systemd[1]: Stopped calibre-web.service - Calibre-Web.
Dec 21 14:35:28 box systemd[1]: calibre-web.service: Consumed 8.425s CPU time.
Dec 21 14:35:29 box systemd[1]: Started calibre-web.service - Calibre-Web.
Dec 21 14:39:46 box systemd[1]: calibre-web.service: A process of this unit has been killed by the OOM killer.
Dec 21 14:39:46 box systemd[1]: calibre-web.service: Main process exited, code=killed, status=9/KILL
Dec 21 14:39:46 box systemd[1]: calibre-web.service: Failed with result 'oom-kill'.
Dec 21 14:39:46 box systemd[1]: calibre-web.service: Consumed 3.099s CPU time.

Is there possibly something to learn from this other OOM (out of memory) issue?

• 37

[holta]

Bad News is that Calibre-Web now crashes...

Example browser screenshot below shows "502 Bad Gateway" — i.e. this is what you get right after clicking "Close" in the bottom-right of the above-mentioned bright red "Bad Gateway: File size may be too big" pop-up:

Just FYI / FWIW: Running systemctl start calibre-web (as root) does in fact later work, as a partial workaround to at least restart the calibre-web.service, allowing browsing to continue at http://box/books

[holta]

@EMG70 it's quite possible this might take a few days to solve:

But if you get the chance to upload big MP4's and big WebM's — on a completely fresh/new VM in coming days, just to confirm the above patterns — that'd be really great!