search

hydrusnetwork / hydrus

A personal booru-style media tagger that can import files and tags from your hard drive and popular websites. Content can be shared with other users via user-run servers.
http://hydrusnetwork.github.io/hydrus/
Other
2.41k stars 160 forks source link

Downloaders not working (seemingly because of) Temp folder not working #1620

Open Oxian1 opened 1 month ago

Oxian1 commented 1 month ago

Hydrus version

v595

Qt major version

Qt 6

Operating system

Linux (specify distro and version in comments)

Install method

Extract

Install and OS comments

install comment: I specified extract because I had to extract from a zip file, but I also have to open hydrus through the console so i'm not sure if "running from source" might be more accurate or not

OS comment: Linux mint, specifically cinnamon 22

Bug description and reproduction

I was running a gallery Downloader when I got the following error message Hydrus Bug Sharable upon futher investigation I found that the directory highlighted in red, and its equivalents given the nature of tmp files, always exist. but they are equally always completely empty, even though the error message pops up unambiguously after the file finishes downloading.

also, the error message says "details should be shown" but I could not find any additional error info anywhere. no pop-up, and nothing on the logs or console that wasn't already in the above screenshot **

**edit: I have since retried the failed files with a few relevant seeming report modes turned on and pasted the results to the log section

I have tried: restarting my computer - no change restarting hydrus - no change re-trying the troublesome files - no change skipping the troublesome files - different tmp and destination directories, but ultimately the same error with every subsequent file downloading the troublesome files one at a time with a url downloader - no change removing the query entirely and starting from scratch - no change using modifiers in the console to put the temp folder on a different drive - the error message changed to reflect the new directory, but ultimately the same error using modifiers in the console to not use temp folders at all and store temporary files in memory instead - no change I stg it just ignored me and made non-functional temp folders anyway

the biggest difference I saw was when I completely clean installed hydrus with a fresh DB and you wanna know what changed? I got the error sooner than usual.

I'm completely at a loss, really hoping there's something obvious i've missed thanks a ton for any help yall can provide

Log output

v595, 2024-10-27 11:48:11: Running db job: read url_statuses
v595, 2024-10-27 11:48:11: File path request: (b"\xd6\xe2A\xb8\xc1\xba\xa1\x05\x86\xe1%\xe6\x9b\x14!\xd7w\x81\xf0\\'\xc7\xb8Zj$\x18\xd5b\x9e\x13-", 1)
v595, 2024-10-27 11:48:11: Running db job: read media_result
v595, 2024-10-27 11:48:12: Running db job: read tag_siblings_lookup
v595, 2024-10-27 11:48:12: Running db job: read url_statuses
v595, 2024-10-27 11:48:12: File path request: (b"\xd6\xe2A\xb8\xc1\xba\xa1\x05\x86\xe1%\xe6\x9b\x14!\xd7w\x81\xf0\\'\xc7\xb8Zj$\x18\xd5b\x9e\x13-", 1)
v595, 2024-10-27 11:48:12: Running db job: read media_result
v595, 2024-10-27 11:48:14: File import job created:
Source: https://e621.net/posts/4830081
Raw import path: /tmp/hydrus7vcrpd2n/hydrusd5img1sn.
v595, 2024-10-27 11:48:14: File import job starting work.
v595, 2024-10-27 11:48:14: File import job hash: d6e241b8c1baa10586e125e69b1421d77781f05c27c7b85a6a2418d5629e132d
v595, 2024-10-27 11:48:14: Running db job: read hash_status
v595, 2024-10-27 11:48:14: File path request: (b"\xd6\xe2A\xb8\xc1\xba\xa1\x05\x86\xe1%\xe6\x9b\x14!\xd7w\x81\xf0\\'\xc7\xb8Zj$\x18\xd5b\x9e\x13-", 1)
v595, 2024-10-27 11:48:14: File import job pre-import status: unknown, The client believed this file was already in the db, but it was truly missing! Import will go ahead, in an attempt to fix the situation.
v595, 2024-10-27 11:48:14: File import job mime: jpeg
v595, 2024-10-27 11:48:14: File import job file info: (393237, 1, 2500, 3450, None, None, False, None)
v595, 2024-10-27 11:48:14: File import job generating thumbnail
v595, 2024-10-27 11:48:14: File import job generating perceptual_hashes
v595, 2024-10-27 11:48:14: File import job generated 1 perceptual_hashes: ['c03f317ccac0372f']
v595, 2024-10-27 11:48:14: File import job generating other hashes
v595, 2024-10-27 11:48:14: File import job testing if good to import for file import options
v595, 2024-10-27 11:48:14: Adding file to client file structure: from /tmp/hydrus7vcrpd2n/hydrusd5img1sn to /media/oxian/Seagate/----/Hydrus Network/db/client_files/fd6/d6e241b8c1baa10586e125e69b1421d77781f05c27c7b85a6a2418d5629e132d.jpg
v595, 2024-10-27 11:48:14: Copying the file from "/tmp/hydrus7vcrpd2n/hydrusd5img1sn" to "/media/oxian/Seagate/----/Hydrus Network/db/client_files/fd6/d6e241b8c1baa10586e125e69b1421d77781f05c27c7b85a6a2418d5629e132d.jpg" failed! Details should be shown and other import queues should be paused. You should shut the client down now and fix this!
v595, 2024-10-27 11:48:14: A critical drive error has occurred. All importers--subscriptions, import folders, and paged file import queues--have been paused. Once the issue is clear, restart the client and resume your imports under the file and network menus!
hydrusnetwork commented 1 month ago

Hey, I am sorry for the trouble here. That /tmp/hydrustoehdunot stuff is supposed to be the normal temp folder that hydrus takes when it boots up and uses for all sorts of normal temp file stuff. Every imported file goes through there. It seems like your OS is not happy with hydrus using it, I presume for a permissions reason. I haven't seen anything quite like your problem before, but I've seen similar when guys have a temp dir that is mounted on a ramdisk that has some unusual properties. I vaguely remember a guy who has this problem when his anti-virus was unhappy about a dodgy pdf coming in, but I may be remembering it wrong.

Let's see if we can short-circuit this whole problem simply. You can tell hydrus to use a different folder than your OS default /tmp for its temp dir using the --temp_dir launch switch. Please check it out here:

https://hydrusnetwork.github.io/hydrus/launch_arguments.html#--temp_dir_temp_dir

So please configure your launch shortcut or script to do the appropriate variant of client --temp_dir="/some/other/path", and then double-check that that was set correctly under help->about, and let me know how it went.

hydrusnetwork commented 1 month ago

Looking again at your log, I am not sure if it is the temp dir that is the problem. The client would have processed the file ok at that point, and the actual error is when it is copying into your normal file structure. I think the 'details should be shown' is supposed to be appearing in the 'file log' of the respective importer, it would be in the notes section, but I have updated tomorrow's v596 release to just spam this exact error to the log, so if moving the temp dir doesn't help you here, please try it again in v596 when it is convenient and we'll see what it puts in the log.