New files aren't being detected after v505(?) update

[sbstratos79]

I think the Hydrus dev made some changes to the API after v505. This broke a lot of tools that work with the Hydrus API. Many have pushed out updates to fix the issue but Pydrus-Tagger still seems to be broken. I tried loading new files with IQDBTagger.py "Initialize" but it just says "No new file was found!".

[shephipster]

Noted. I'll create a patch that will also hopefully speed the whole thing up by just using a tag in the local tags (not PTR, don't want to get anyone in trouble for pushing weird tags) instead of a slow text file. I'll close the issue once the fix is confirmed working on Hydrus v.510

[shephipster]

Issue should be resolved along side some major enhancements to the speed of the initialization step. The first time you run the update it will take about 6x longer as it'll need to tag your processed images but after that things should run a lot faster. Closing for now but if any issues pop up either re-open of set a new issue.

[Drakonas]

This is still an issue for me with v512. Latest commit.

You might want to read all the v506 API changelog notes. It changed a lot and a number of structures were changed, including tags. I don't think anything in your code changed for tags, but I'm not an expert.

[shephipster]

It's working for me on version 510 but I'll update to v512 and see if I can't replicate the issue and get it patched up.

[shephipster]

I've removed the last vestiges of the old system for tracking unprocessed files. Hopefully that might resolve your issue, but unfortunately I was unable to reproduce your error locally using Hydrus version 512 so I can't say for sure. You might can also try removing the IQDB_PROCESSED tag from some of your images and see if the program recognizes them as new files. Otherwise I will continue to look into the issue, and I apologize for not being able to be sure it will work for you.

[Drakonas]

I've never used this tool. None of my files have that tag but some have just a hatate:not_found tag of my own choice, from a different tool. It takes all the scripts like 2 seconds to run. The hash script also does nothing (both the txt files are empty). I was under the impression that first run should take like 15 minutes. (I have 65000+ files)

[shephipster]

I've never used this tool. None of my files have that tag but some have just a hatate:not_found tag of my own choice, from a different tool. It takes all the scripts like 2 seconds to run. The hash script also does nothing (both the txt files are empty). I was under the impression that first run should take like 15 minutes. (I have 65000+ files)

The original version of the code would have taken a good while to run, but some recent changes have sped things up a decent bit. A side effect of this, however, is that it will only do up to 1000 files at a time (you can increase this limit by increasing the MAX_FILES value in IQDBTagger.py). As for the txt files, those are no longer used and if it's still generating them then that's a bug and I'll fix that shortly.

As for the program not working, I'm still trying to recreate the problem but in the mean time I've added some code that will hopefully help track down the cause. It will try to connect to the client, grab a file, give it a test tag, and then delete that tag. If any of these steps fail it will let you know which one and we can hopefully figure out a solution from there.

Again, I apologize for the program not working for you and hope I can find a resolution for you and anyone else having trouble as quickly as I can.

[Drakonas]

The original version of the code would have taken a good while to run, but some recent changes have sped things up a decent bit. A side effect of this, however, is that it will only do up to 1000 files at a time (you can increase this limit by increasing the MAX_FILES value in IQDBTagger.py). As for the txt files, those are no longer used and if it's still generating them then that's a bug and I'll fix that shortly. I appreciate you looking into this. Really. Hope I didn't sound angry.

Anyway, I wanted to clarify. It wasn't generating the files... initially. But the hashFileEstimater.py outputted an error saying "Cannot write to ./tempFiles/processedHashes.txt" (Don't remember exact wording). So I figured it had crashed since the ./tempFiles folder didn't exist. I created the folder and created the two empty .txt files (essentially touch from Linux), but only processedHashes.txt and needWork.txt. After all this testing, now there's 5 txt files:

Pydrus-Tagger\tempFiles\needWork.txt
Pydrus-Tagger\tempFiles\processedGifHashes.txt
Pydrus-Tagger\tempFiles\processedHashes.txt
Pydrus-Tagger\tempFiles\processedIQDBHashes.txt
Pydrus-Tagger\tempFiles\processedSauceNaoHashes.txt

These are all still 0 byte files.

I guess I'm not sure if the readme was outdated. I was following the instructions from them.

And by 2 seconds, I literally meant a second. And if you notice in my screenshot above, it says "Files processed: 0". That's after I clicked the Initialize button. It doesn't seem to be processing anything, as far as I can tell.

I did originally get errors when the Hydrus API key and such weren't in place. I also added SauceNAO keys and the like. So I'm pretty sure it should be good to go. No errors are sent to the console anymore.

I am using HYDRUS_URL = "http://127.0.0.1:45869/" in the script as my Hydrus client API address by the way. FYI it doesn't seem to detect when there's no trailing slash and fails if I don't add it. Not sure if this was just something you didn't care to account for, or something that was looked over. I don't mind adding it. Just thought I'd mention.

Forgive me if I'm not using environment variables.. but I'd rather be able to double click this and not have my API keys for these sites in public view. :S I did not change anything but the variable assignment lines from operating system environment variables. I even made sure LIMIT = float("100") is used for consistency in the sauceNaoApi.py.

I guess I should ask. What version of Python do you expect? I use Python 3.8.

[shephipster]

The readme was absolutely outdated and I'm sorry about that.

As long as you properly adjusted the os environment variables then that should all be fine.

For the Python version I'm currently using 3.9 but it was originally written using 3.8 I believe.

Have you had any luck with the most recent updates? As in, is the UI giving you any messages about failing to connect to Hydrus now or is it still getting to the end and saying it found no files?

[Drakonas]

Sorry I have been busy. I will try this out after the weekend. Thanks.

[Drakonas]

Seems that the updates may have done it. I also didn't realize .env was a thing. I switched to using that rather than editing the scripts. :) Really appreciate the readme update. Thank you!

Update: However, while I can change the MAX_FILES limit in the python script, the 100 stop after setting doesn't seem to change anything when changed. I think it's because in Windows, if I type anything in the box, it wouldn't actually change the number until the box "submits". Because there's nothing else in the interface to click on to "defocus" the box to submit the number, it still runs with the default of 100. If I instead set all the self.after(100) to self.after(70000) THAT works. And it processes more.

Unless I am misunderstanding that that is a time limit, and not a file limit.... I may be just stupid and increased the time to 70 seconds lol.

[justjakka]

sorry to revive this issue, but im getting this error now. hydrus client 518 running in podman

[shephipster]

sorry to revive this issue, but im getting this error now. hydrus client 518 running in podman

I'm unable to recreate the issue on my end after updating to v518. I understand that some people have trouble if their .env is missing, do you happen to have one in the folder that contains IQDBTagger.py?

[justjakka]

i didnt use the env file, i just exported the values for my current venv

[shephipster]

Just to confirm, the system is telling you that it found no new files for processing? Or are you getting one of the new error messages about connection issues?

[justjakka]

no new files

[shephipster]

Unfortunately I'm unable to recreate the issue, as mentioned earlier. The only thing I can think of is to make sure you have the most recent version of the IQDBTagger, as it was updated in January, and to see if the images in your Hydrus have the tag iqdb_processed . If they have that tag you can try deleting the tag and re-running the program, if not then I will continue to try and recreate the issue on my end.

[justjakka]

ive cloned the repo so i have the most up to date version and the tag doesnt even appear, even though the api entry for the key has all permissions

[shephipster]

The system will let you know if it can't connect, add tags, or find files. If you're seeing "No files found" and not "System could not fetch files from the Hydrus Client" then I'll have to continue testing to try and find what's going on. If you are seeing "System could not fetch files from the Hydrus Client" specifically though then that means the system does not have permission to access files, but it doesn't sound like that's the issue.

[justjakka]

nope, it just says no new files found

[justjakka]

maybe its because there was a python hydrus-api update in the end of february, so the code doesnt work now with it?