search

hydrusnetwork / hydrus

A personal booru-style media tagger that can import files and tags from your hard drive and popular websites. Content can be shared with other users via user-run servers.
http://hydrusnetwork.github.io/hydrus/
Other
2.39k stars 158 forks source link

.db-shm and .db-wal files don't get cleaned up on shutdown #1369

Open roachcord3 opened 1 year ago

roachcord3 commented 1 year ago

Hydrus version

v527

Qt major version

Qt 6

Operating system

macOS other (specify in comments)

Install method

Running from source

Install and OS comments

macOS 13.3.1, source, python 3.11.3, I can dump my pip freeze if you really want too

Bug description and reproduction

When shutting my client down, the various .db-shm and .db-wal files do not get deleted, which I just learned today is supposed to happen.

Under normal circumstances, the .db-wal files will at least be empty by the time the main window shuts down, and so far I haven't had any reason to think that this has caused damage to my client DB, but after a discussion in discord about a deadlock in shutting down the main window and about when the client_running file gets deleted, it became clear that something is amiss.

Attached in the "log output" section will be what the logs looked like when I shut my client down and the splash screen got stuck on "waiting for services to exit", followed by some blank lines, then logs from shutting down when that didn't happen. Upon next client boot, I did not receive error messages in either of these circumstances, and everything seemed fine, but since this is apparently not the expected behavior, I'm filing an issue.

Also, here is a screenshot of Finder showing that these files are still lying around even after shutdown.

image

BTW, I'm reporting the symptoms here since they seem problematic, but for historical context, what I was actually asking about was whether I could rely on client_running being missing meaning it was safe to back up my database files. (And also what was up with the deadlock in the splash screen.)

Log output

v527, 2023/05/20 02:28:47: shutting down gui…
v527, 2023/05/20 02:28:47: waiting for managers to exit
v527, 2023/05/20 02:28:47: subscriptions, tag display sync
v527, 2023/05/20 02:28:48: waiting for workers to exit
v527, 2023/05/20 02:28:48: waiting for idle shutdown work
v527, 2023/05/20 02:28:48: db
v527, 2023/05/20 02:28:48: public tag repository processing
v527, 2023/05/20 02:28:48: public tag repository sync: processing updates
v527, 2023/05/20 02:32:26: public tag repository processed 30,911 definitions at 141 rows/s
v527, 2023/05/20 02:32:26: waiting for services to exit
v527, 2023/05/20 02:32:26: stopping services…
v527, 2023/05/20 02:32:26: shutting down db…
v527, 2023/05/20 02:32:26: saving and exiting objects
v527, 2023/05/20 02:32:26: file viewing stats flush
v527, 2023/05/20 02:32:26: services
v527, 2023/05/20 02:32:27: waiting for db to finish up…
v527, 2023/05/20 02:32:37: waiting for db to finish up…
v527, 2023/05/20 02:32:47: cleaning up…

v527, 2023/05/20 13:00:27: shutting down gui…
v527, 2023/05/20 13:00:27: waiting for managers to exit
v527, 2023/05/20 13:00:27: subscriptions, tag display sync
v527, 2023/05/20 13:00:27: tag display sync
v527, 2023/05/20 13:00:27: waiting for workers to exit
v527, 2023/05/20 13:00:27: waiting for idle shutdown work
v527, 2023/05/20 13:00:27: db
v527, 2023/05/20 13:00:40: public tag repository processing
v527, 2023/05/20 13:00:40: public tag repository sync: processing updates
v527, 2023/05/20 13:09:45: public tag repository processed 250,005 content rows at 458 rows/s
v527, 2023/05/20 13:09:45: waiting for services to exit
v527, 2023/05/20 13:09:45: stopping services…
v527, 2023/05/20 13:09:45: shutting down db…
v527, 2023/05/20 13:09:45: saving and exiting objects
v527, 2023/05/20 13:09:45: file viewing stats flush
v527, 2023/05/20 13:09:45: services
v527, 2023/05/20 13:09:46: waiting for db to finish up…
v527, 2023/05/20 13:09:47: waiting for db to finish up…
v527, 2023/05/20 13:09:48: cleaning up…
v527, 2023/05/20 13:09:48: shutting down controller…
v527, 2023/05/20 13:09:48: hydrus client shut down
hydrusnetwork commented 1 year ago

Thanks again for reporting this. I'm just adding to this issue for when I investigate it properly: another user is experiencing symptoms that sound like the exit 'last session' is not being saved, which sounds like a similar 'race to shut down', as we had talked about earlier.

hydrusnetwork commented 1 year ago

I looked around and cleaned a bit of the exit code for the dude with the session not saving, and I looked into your stuff too but couldn't find anything. That log output looks fine too--the db is closing ok. So I did a bit of extra searching and came across this post:

https://github.com/groue/GRDB.swift/issues/739#issuecomment-604363998

It seems like certain versions of macOS SQLite simply don't/didn't do this. They zero out the WAL journal, but don't remove the files. Can you tell me what version of SQLite you see in help->about? I assume this will be whatever is packaged in github's cloud's macOS python 3.9.

If this is the case, then please have a test sometime with my future build, as here: https://github.com/hydrusnetwork/hydrus/releases/tag/v528-future-1 , which has python 3.10. I expect to make one this week for v531. Let me know if it boots ok, what version of SQLite it has, and if the .db-wal files are cleaned up correctly on client exit.

As to the 'waiting for services to exit' issue, this is a different issue that I'm planning to fix in a longer term code cleanup.

roachcord3 commented 1 year ago

@hydrusnetwork I run from source and use python 3.11.3 (via pyenv-virtualenv.) Anyway, the sqlite version is 3.39.5. Also, I'm now running v530