appleappleapplenanner
commented
1 year ago Thanks for the suggestions.
The defaults should remain as simple as possible. By adding more constraining defaults, I would need to add a parameter to remove defaults which makes people have to read more documentation, read more of the --help, etc. I want things to be as simple, and more importantly as straightforward as possible. There is no magic here; if you have 100k video files you will have to hash all of them if you want to sort them all for duplicates. How you choose to do that is and should be up to you.
I would like to improve the chunking while processing, more specifically the perceptual hashing. But those improvements would probably be silent to the user. There's no getting around hashing the videos you want to hash. But, if there are issues with things like running out of memory, crashes, etc that would be a legitimate concern and should be reported to me with as much information as possible.
I will add some tips about video comparison, but really it's just if one looks better than the other then archive that. If you have a shit ton of files and you aren't trying to archive them then it's as simple as that. People who care more about what videos they want to save probably have already invested the time into learning those things. The Hydrus duplicates page already has far more information than I would like to write. I'll add a link to it in Usage or the FAQ.
What fiddling around did you have to do to get the program working if you remember? I want the documentation to be as clear as possible since people already have to go through the hassle of installing WSL. I will probably end up adding a script to install WSL and the program for users who just want it to work.
GoAwayNow
micnorian14
After some fiddling around I did eventually get things to work as intended. At first, running the app as suggested with no query attached will attempt to read every "system:filetype is video" in your entire library. For users with +1000 webm/mp4 files this results in a queue several days long. I found that adding arguments like --query="system:duration < 2s" were especially helpful for initial prototyping and later narrowing searches down. I suggest including this by default or at least mentioning it on the usage page.
I'd also like to suggest a few tips since few (if any) users have used the duplicate processing function for video files before now. Most of this is common knowledge but including it for context and clarity seems appropriate.
1) Most duplicate videos scraped from various boorus will be re-encoded or compressed copies (usually in webm format) that have been scraped from other websites that likely compressed it for streaming purposes. This process is lossy at best - resulting in progressively lower quality copies of the same file being distributed across these websites. 2) A source-quality file will almost always be a much larger size and/or higher bitrate mp4 or webm 3) When sorting files in the duplicate processor; be wary of same-quality alts or a creators updated/corrected release (usually dated the following month) 4) Many 3rd party edits will simply add or replace audio 5) Most booru style websites compress videos to webm, so finding the mp4 version of a video likely means it has not been compressed - indicating it is a source-quality file. This has been true in every instance I've found, but there may be websites that still re-encode videos in mp4 6) Occasionally some chad will take a source quality file and interleave it to a higher framerate with better compression settings/codec. This results in a source-quality video but with a perceived higher framerate. These versions can be smaller in filesize by as much as 30%! 7) Most authors distribute higher quality versions of their content to paid supporters, usually omitting their watermark. Files without watermarks are almost always higher quality as a result.