Unrecoverable 500 Error (with readonly database)

[MinchinWeb]

Describe the bug/problem

Calibre-web's epub reader stopped working for a user, and just spits out a stack trace and returns an HTTP 500 error. Reloading the page (or any page) won't let the user get past it.

To Reproduce Steps to reproduce the behavior:

1. Using linuxserver.io ' s Docker image, in docker compose.
2. Database and library files are mounted as read-only
3. ?? (not sure what the user did)
4. see Error
5. Restarting the calibre-web docker container makes this go away.

Stack Trace

This is the "error" webpage the user got.

Internal Server Error
500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2073, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1518, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1516, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1502, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/app/calibre-web/cps/usermanagement.py", line 38, in decorated_view
return login_required(func)(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/flask_login/utils.py", line 272, in decorated_view
return func(*args, **kwargs)
File "/app/calibre-web/cps/web.py", line 763, in index
return render_books_list("newest", sort_param, 1, page)
File "/app/calibre-web/cps/web.py", line 411, in render_books_list
entries, random, pagination = calibre_db.fill_indexpage(page, 0, db.Books, True, order[0],
File "/app/calibre-web/cps/db.py", line 679, in fill_indexpage
return self.fill_indexpage_with_archived_books(page, pagesize, database, db_filter, order, False, *join)
File "/app/calibre-web/cps/db.py", line 685, in fill_indexpage_with_archived_books
randm = self.session.query(Books) \
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 2759, in all
return self._iter().all()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 2894, in _iter
result = self.session.execute(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1691, in execute
conn = self._connection_for_bind(bind)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1532, in _connection_for_bind
return self._transaction._connection_for_bind(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind
self._assert_active()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 601, in _assert_active
raise sa_exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely)
(sqlite3.OperationalError) attempt to write a readonly database
[SQL: DELETE FROM books_authors_link WHERE books_authors_link.book = ? AND books_authors_link.author = ?]
[parameters: (3374, 1568)]
(Background on this error at: https://sqlalche.me/e/14/e3q8) (Background on this error at: https://sqlalche.me/e/14/7s2a)
Please report this issue with all related information: [Create Issue](https://github.com/janeczku/calibre-web/issues/new?assignees=&labels=&template=bug_report.md&title=)
[Return to Home](http://books.trocadero.lan/)

Logfile

Sorry, I don't have one.

Expected behavior

Re-loading the page should allow calibre-web to recover.

Screenshots

see stack trace

Environment (please complete the following information):

• OS: Ubuntu host
• Python version: built-in in linuxserver.io's docker image (Python 3?)
• Calibre-Web version: [e.g. 0.6.8 or 087c4c59 (git rev-parse --short HEAD)]: Upgraded it today trying to fix it, so I'm not sure. Current version is 0.6.16-ls150 Build-date:- 2022-03-01T10:44:08+01:00, so sometime before that.
• Docker container: [None/LinuxServer]: LinuxServer
• Special Hardware: [e.g. Rasperry Pi Zero] "regular" x64 hardware
• Browser: [e.g. Chrome 83.0.4103.97, Safari 13.3.7, Firefox 68.0.1 ESR] : Chrome on Andriod

Additional context Add any other context about the problem here. [e.g. access via reverse proxy, database background sync, special database location]

Trimmed docker-compose:

version: '2.4'

# environmental variables *for Docker Compose* will be loaded from a `.env` file
# in the same directory as this file

services:
  calibre-web:
    # this is *just* the web interface from Calibre
    # with the Docker Mod, it pulls and installs a bunch of stuff from
    # `apt`, so it takes a while to start up
    image: lscr.io/linuxserver/calibre-web
    container_name: calibre-web
    environment:
      - PUID=${PUID}
      - PGID=${PGID}
      - TZ=${TZ}
      - DOCKER_MODS=linuxserver/calibre-web:calibre
    volumes:
      - ${DOCKER_USERDIR}/volumes/calibre-web/config:/config
      - ${NAS_DIR}/ebook/calibre:/books:ro
    ports:
      - 9936:8083
    restart: unless-stopped
    labels:
      - traefik.enable=true
      - traefik.http.routers.calibreweb.rule=Host("books.${LOCAL_DOMAIN_NAME}")

networks:
  default:
    external:
      name: meta_external

[erkannt]

Had the same issue. Fixed after restart :+1:

version: '3'

services:
  calibre:
    image: linuxserver/calibre-web@sha256:0670e46eebe6d178f1415434960572f1554dd6e97d8215b23df4a1c8fd53f1b3
    environment:
      - TZ=Europe/London
    volumes:
      - /var/volumes/calibre-web:/config
      - /var/volumes/syncthing/Calibre-Library:/books:ro
    networks:
      - traefik
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.calibre-web.rule=Host(`books.rknt.de`)"
      - "traefik.http.routers.calibre-web.entrypoints=websecure"
      - "traefik.http.routers.calibre-web.tls.certresolver=myresolver"
    restart: unless-stopped

networks:
  traefik:
    external: true

[erkannt]

User has been able to provoke the issue again. I can repro: Clicking on 'top rated' gives this message for the url: https://books.rknt.de/rated/stored/

Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1788, in _execute_context
self.dialect.do_executemany(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 729, in do_executemany
cursor.executemany(statement, parameters)
sqlite3.OperationalError: attempt to write a readonly database
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2073, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1518, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1516, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1502, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/app/calibre-web/cps/usermanagement.py", line 38, in decorated_view
return login_required(func)(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/flask_login/utils.py", line 272, in decorated_view
return func(*args, **kwargs)
File "/app/calibre-web/cps/web.py", line 753, in books_list
return render_books_list(data, sort_param, book_id, page)
File "/app/calibre-web/cps/web.py", line 341, in render_books_list
return render_rated_books(page, book_id, order=order)
File "/app/calibre-web/cps/web.py", line 389, in render_rated_books
entries, random, pagination = calibre_db.fill_indexpage(page, 0,
File "/app/calibre-web/cps/db.py", line 713, in fill_indexpage
return self.fill_indexpage_with_archived_books(page, database, pagesize, db_filter, order, False,
File "/app/calibre-web/cps/db.py", line 774, in fill_indexpage_with_archived_books
entries = self.order_authors(entries, True, join_archive_read)
File "/app/calibre-web/cps/db.py", line 786, in order_authors
ids = [a.id for a in entry.authors]
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/attributes.py", line 481, in __get__
return self.impl.get(state, dict_)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/attributes.py", line 941, in get
value = self._fire_loader_callables(state, key, passive)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/attributes.py", line 977, in _fire_loader_callables
return self.callable_(state, passive)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/strategies.py", line 911, in _load_for_state
return self._emit_lazyload(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/strategies.py", line 1047, in _emit_lazyload
result = session.execute(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1640, in execute
) = compile_state_cls.orm_pre_session_exec(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/context.py", line 319, in orm_pre_session_exec
session._autoflush()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 2237, in _autoflush
util.raise_(e, with_traceback=sys.exc_info()[2])
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 2226, in _autoflush
self.flush()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3363, in flush
self._flush(objects)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3503, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3463, in _flush
flush_context.execute()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
rec.execute(self)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 579, in execute
self.dependency_processor.process_saves(uow, states)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/dependency.py", line 1182, in process_saves
self._run_crud(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/dependency.py", line 1202, in _run_crud
result = connection.execute(statement, secondary_delete)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1295, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
return connection._execute_clauseelement(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1487, in _execute_clauseelement
ret = self._execute_context(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1851, in _execute_context
self._handle_dbapi_exception(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2032, in _handle_dbapi_exception
util.raise_(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1788, in _execute_context
self.dialect.do_executemany(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 729, in do_executemany
cursor.executemany(statement, parameters)
sqlalchemy.exc.OperationalError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely)
(sqlite3.OperationalError) attempt to write a readonly database
[SQL: DELETE FROM books_authors_link WHERE books_authors_link.book = ? AND books_authors_link.author = ?]
[parameters: ((449, 200), (449, 201))]
(Background on this error at: https://sqlalche.me/e/14/e3q8)

[OzzieIsaacs]

Thanks. this is something I can look after

[OzzieIsaacs]

The problem occurs if the user tries to rename a author in the books table. I'll fix this.

Side question: If you are having a read only database, why do you allow users to edit books?

[OzzieIsaacs]

Please update to the newest nightly version (there should be a docker image for this) and the application should not crash any more

[erkannt]

I can still provoke the issue.

from docker inspect of the container:

"Image": "linuxserver/calibre-web:nightly@sha256:bfb2ece64a08e6f411fa81f65b7aa0b24105f952441e6d8fdfa00a78b840c0e6",
                "build_version": "Linuxserver.io version:- 49692b4a-ls222 Build-date:- 2022-03-12T09:20:58+01:00",

/ loads fine. First click on /rated/stored/ produces:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1788, in _execute_context
self.dialect.do_executemany(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 729, in do_executemany
cursor.executemany(statement, parameters)
sqlite3.OperationalError: attempt to write a readonly database
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2073, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1518, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1516, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1502, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/app/calibre-web/cps/usermanagement.py", line 38, in decorated_view
return login_required(func)(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/flask_login/utils.py", line 272, in decorated_view
return func(*args, **kwargs)
File "/app/calibre-web/cps/web.py", line 753, in books_list
return render_books_list(data, sort_param, book_id, page)
File "/app/calibre-web/cps/web.py", line 341, in render_books_list
return render_rated_books(page, book_id, order=order)
File "/app/calibre-web/cps/web.py", line 389, in render_rated_books
entries, random, pagination = calibre_db.fill_indexpage(page, 0,
File "/app/calibre-web/cps/db.py", line 713, in fill_indexpage
return self.fill_indexpage_with_archived_books(page, database, pagesize, db_filter, order, False,
File "/app/calibre-web/cps/db.py", line 774, in fill_indexpage_with_archived_books
entries = self.order_authors(entries, True, join_archive_read)
File "/app/calibre-web/cps/db.py", line 786, in order_authors
ids = [a.id for a in entry.authors]
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/attributes.py", line 481, in __get__
return self.impl.get(state, dict_)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/attributes.py", line 941, in get
value = self._fire_loader_callables(state, key, passive)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/attributes.py", line 977, in _fire_loader_callables
return self.callable_(state, passive)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/strategies.py", line 911, in _load_for_state
return self._emit_lazyload(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/strategies.py", line 1047, in _emit_lazyload
result = session.execute(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1640, in execute
) = compile_state_cls.orm_pre_session_exec(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/context.py", line 319, in orm_pre_session_exec
session._autoflush()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 2237, in _autoflush
util.raise_(e, with_traceback=sys.exc_info()[2])
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 2226, in _autoflush
self.flush()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3363, in flush
self._flush(objects)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3503, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3463, in _flush
flush_context.execute()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
rec.execute(self)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 579, in execute
self.dependency_processor.process_saves(uow, states)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/dependency.py", line 1182, in process_saves
self._run_crud(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/dependency.py", line 1202, in _run_crud
result = connection.execute(statement, secondary_delete)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1295, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
return connection._execute_clauseelement(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1487, in _execute_clauseelement
ret = self._execute_context(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1851, in _execute_context
self._handle_dbapi_exception(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2032, in _handle_dbapi_exception
util.raise_(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1788, in _execute_context
self.dialect.do_executemany(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 729, in do_executemany
cursor.executemany(statement, parameters)
sqlalchemy.exc.OperationalError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely)
(sqlite3.OperationalError) attempt to write a readonly database
[SQL: DELETE FROM books_authors_link WHERE books_authors_link.book = ? AND books_authors_link.author = ?]
[parameters: ((449, 200), (449, 201))]
(Background on this error at: https://sqlalche.me/e/14/e3q8)

Apart from the admin account no user is allowed to edit books. I don't think I can disable it for admin.

I have rolled back to previous releases and found the following:

This version doesn't 500 when I visit the top rated page:

linuxserver/calibre-web@sha256:3e1c30daac3171f5a0117d876616dbc35d29ad99c8702604359e9d624a5ee84d
                "build_version": "Linuxserver.io version:- 0.6.16-ls150 Build-date:- 2022-03-01T10:44:08+01:00",

And this is the first one where I get the 500:

linuxserver/calibre-web@sha256:99af4e237a257b446f39ca7e91a6546596443775b023760605ee146d00998be8
                "build_version": "Linuxserver.io version:- 0.6.17-ls151 Build-date:- 2022-03-06T16:24:42+00:00"

Let me know I I can help with any other details.

PS: Thanks for looking into this. Such a useful piece of software.

[OzzieIsaacs]

I don't think I can disable it for admin.

Yes you can, also the admin can't edit the books. It's possible to remove the rights from admin. Admin doesn't me the account has all rights, it can only get all rights

[OzzieIsaacs]

I think I found it

[OzzieIsaacs]

I think you are having a database error, book 449 has two "author sort" strings, which aren't matching any entry in the authors table. The relevant authors having the number 201 and 200. Book 449 is displayed in the rated section on the first page, on the other pages it's displayed on one of the later pages. The newest nightly version from now can deal with this. I recommend to open the database with Calibre itself (there is a certain possibility that Calibre will complain at opening the database that it is inconsistent) and run a library maintenance 2nd option to fix this is to open book 449 and edit author (change one later), save, undo the change and save again (with r/w database).

Don't forget to backup before doing changes

If you are happy, please close the issue

[erkannt]

:partying_face: Nightly image no longer throws 500 when I go to top-rated. Thank you! Also thanks for the tip with the db. I'll investigate that.

[erkannt]

@MinchinWeb Does this also fix it for you? I can't close the issue. Could you please close it if it resolves the issue for you as well?

[MinchinWeb]

@erkannt Yes, the nightly version seems to fix this!

@OzzieIsaacs Thanks you for the amazingly fast fix!