Closed dalanmiller closed 2 months ago
dalanmiller
commented
3 months ago https://github.com/sissbruecker/linkding/issues/230#issuecomment-1085182826
I think this is likely the reason. I am mounting an NFS mount into the container and despite the sqlite file being accessible and writeable, there's something maybe returning incorrectly from the .ensure_connections method in Django.
There should be a way to disable the /health endpoint in the docker container.
dalanmiller
commented
3 months ago Okay I can get the healthcheck to pass by adding this to my docker-compose.yml file:
linkding:
...
healthcheck:
test: ['CMD', "curl", "-f", "http://localhost:9090"]However, (and maybe as expected) Traefik then picks up the routing given the container is now in a 'healthy' state, but there are 500 errors abound:
linkding | 2024-04-03 10:35:26,422 ERROR Internal Server Error: /login/
linkding | Traceback (most recent call last):
linkding | File "/opt/venv/lib/python3.11/site-packages/django/db/backends/base/base.py", line 275, in ensure_connection
linkding | self.connect()
linkding | File "/opt/venv/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
linkding | return func(*args, **kwargs)
linkding | ^^^^^^^^^^^^^^^^^^^^^
linkding | File "/opt/venv/lib/python3.11/site-packages/django/db/backends/base/base.py", line 256, in connect
linkding | self.connection = self.get_new_connection(conn_params)
linkding | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
linkding | File "/opt/venv/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
linkding | return func(*args, **kwargs)
linkding | ^^^^^^^^^^^^^^^^^^^^^
linkding | File "/opt/venv/lib/python3.11/site-packages/django/db/backends/sqlite3/base.py", line 181, in get_new_connection
linkding | conn = Database.connect(**conn_params)
linkding | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
linkding | sqlite3.OperationalError: unable to open database file
linkding |
linkding | The above exception was the direct cause of the following exception:
linkding |
linkding | Traceback (most recent call last):
linkding | File "/opt/venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
linkding | response = get_response(request)
linkding | ^^^^^^^^^^^^^^^^^^^^^
linkding | File "/opt/venv/lib/python3.11/site-packages/django/core/handlers/base.py", line 220, in _get_response
linkding | response = response.render()
linkding | ^^^^^^^^^^^^^^^^^
linkding | File "/opt/venv/lib/python3.11/site-packages/django/template/response.py", line 114, in render
linkding | self.content = self.rendered_content
linkding | ^^^^^^^^^^^^^^^^^^^^^
linkding | File "/opt/venv/lib/python3.11/site-packages/django/template/response.py", line 92, in rendered_content
linkding | return template.render(context, self._request)
linkding | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
linkding | File "/opt/venv/lib/python3.11/site-packages/django/template/backends/django.py", line 61, in render
linkding | return self.template.render(context)
linkding | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
linkding | File "/opt/venv/lib/python3.11/site-packages/django/template/base.py", line 169, in render
linkding | with context.bind_template(self):
linkding | File "/usr/local/lib/python3.11/contextlib.py", line 137, in __enter__
linkding | return next(self.gen)
linkding | ^^^^^^^^^^^^^^
linkding | File "/opt/venv/lib/python3.11/site-packages/django/template/context.py", line 254, in bind_template
linkding | context = processor(self.request)
linkding | ^^^^^^^^^^^^^^^^^^^^^^^
linkding | File "/etc/linkding/bookmarks/context_processors.py", line 27, in public_shares
linkding | has_public_shares = query_set.count() > 0
linkding | ^^^^^^^^^^^^^^^^^
linkding | File "/opt/venv/lib/python3.11/site-packages/django/db/models/query.py", line 620, in count
linkding | return self.query.get_count(using=self.db)
linkding | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
linkding | File "/opt/venv/lib/python3.11/site-packages/django/db/models/sql/query.py", line 629, in get_count
linkding | return obj.get_aggregation(using, {"__count": Count("*")})["__count"]
linkding | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
linkding | File "/opt/venv/lib/python3.11/site-packages/django/db/models/sql/query.py", line 615, in get_aggregation
linkding | result = compiler.execute_sql(SINGLE)
linkding | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
linkding | File "/opt/venv/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1560, in execute_sql
linkding | cursor = self.connection.cursor()
linkding | ^^^^^^^^^^^^^^^^^^^^^^^^
linkding | File "/opt/venv/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
linkding | return func(*args, **kwargs)
linkding | ^^^^^^^^^^^^^^^^^^^^^
linkding | File "/opt/venv/lib/python3.11/site-packages/django/db/backends/base/base.py", line 316, in cursor
linkding | return self._cursor()
linkding | ^^^^^^^^^^^^^^
linkding | File "/opt/venv/lib/python3.11/site-packages/django/db/backends/base/base.py", line 292, in _cursor
linkding | self.ensure_connection()
linkding | File "/opt/venv/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
linkding | return func(*args, **kwargs)
linkding | ^^^^^^^^^^^^^^^^^^^^^
linkding | File "/opt/venv/lib/python3.11/site-packages/django/db/backends/base/base.py", line 274, in ensure_connection
linkding | with self.wrap_database_errors:
linkding | File "/opt/venv/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
linkding | raise dj_exc_value.with_traceback(traceback) from exc_value
linkding | File "/opt/venv/lib/python3.11/site-packages/django/db/backends/base/base.py", line 275, in ensure_connection
linkding | self.connect()
linkding | File "/opt/venv/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
linkding | return func(*args, **kwargs)
linkding | ^^^^^^^^^^^^^^^^^^^^^
linkding | File "/opt/venv/lib/python3.11/site-packages/django/db/backends/base/base.py", line 256, in connect
linkding | self.connection = self.get_new_connection(conn_params)
linkding | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
linkding | File "/opt/venv/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
linkding | return func(*args, **kwargs)
linkding | ^^^^^^^^^^^^^^^^^^^^^
linkding | File "/opt/venv/lib/python3.11/site-packages/django/db/backends/sqlite3/base.py", line 181, in get_new_connection
linkding | conn = Database.connect(**conn_params)Ahhh this is probably why
Sonarr currently uses WAL mode for journals with SQLite. WAL mode has some advantages, but one major disadvantage is that it can not safely be used over non-local filesystems (https://sqlite.org/wal.html); docker for windows and other virtualization systems using CIFS mounted host paths often fail with sqlite locking or corruption errors when using WAL with sqlite file on host shared paths.
And given in the logs
linkding | 2024-04-03 09:48:20,491 INFO Switched to WAL journal mode
Need an option to change this journal mode if need be.
sissbruecker
commented
3 months ago It seems using SQLite over a network is discouraged in general.
https://www.sqlite.org/whentouse.html:
If there are many client programs sending SQL to the same database over a network, then use a client/server database engine instead of SQLite. SQLite will work over a network filesystem, but because of the latency associated with most network filesystems, performance will not be great. Also, file locking logic is buggy in many network filesystem implementations (on both Unix and Windows). If file locking does not work correctly, two or more clients might try to modify the same part of the same database at the same time, resulting in corruption. Because this problem results from bugs in the underlying filesystem implementation, there is nothing SQLite can do to prevent it.
https://sqlite.org/lockingv3.html:
SQLite uses POSIX advisory locks to implement locking on Unix. On Windows it uses the LockFile(), LockFileEx(), and UnlockFile() system calls. SQLite assumes that these system calls all work as advertised. If that is not the case, then database corruption can result. One should note that POSIX advisory locking is known to be buggy or even unimplemented on many NFS implementations (including recent versions of Mac OS X) and that there are reports of locking problems for network filesystems under Windows. Your best defense is to not use SQLite for files on a network filesystem.
I also remember people having issues with NFS (https://github.com/sissbruecker/linkding/issues/355) even before enabling WAL journal mode.
As an alternative you could take a look at https://litestream.io/, which supports streaming replicas to NFS. https://github.com/fspoettel/linkding-on-fly contains a setup for litestream that may help as guidance.
sissbruecker
commented
2 months ago Closing this for now as I think this is kind of expected with Sqlite. A workaround should be possible using Litestream at least.
Running linkding in a docker container and the health check is failing where I expected it to be working fine. The migrations seem to be running fine which indicates the connection to the sqlite database is fine.
Yet, when I looked at the logic for
/healththat seems to be the only check that is happening.https://github.com/sissbruecker/linkding/blob/bb6c5ca29e3b66a70c2ff1751ea6183c7011d4ae/bookmarks/views/health.py#L12
On the docker host I can see also that the sqlite file exists and was last modified just a moment ago when I spun up the container.
Confirming that this is accurate when I
exec linkding /bin/sh:Logs