Farmadupe
commented
2 years ago Hello, I am not the author of Czkawka but I am the author of two underlying libraries that creates video hashes with Ffmpeg (vid_dup_finder_lib, ffmpeg_cmdline_utils)
From your description I guess the most likely causes might be:
- Ffmpeg got stuck because that video contains some corruption (probably most likely)
- Ffmpeg got stuck because it has a bug (much less likely but still possible)
- My library ffmpeg_cmdline_utils has a bug. I tried to design it to never deadlock but I'm sure all edgecases have not been explored. (possible)
- Ffmpeg got stuck because it is waiting forever to read the first ~30 seconds of video data. Maybe windows has a bug (probably very unlikely)
I think there is no general solution to knowing if Ffmpeg got stuck or if the video is decoding slowly (the halting problem). Additionally I think it is in the nature of media encoding/decoding software to be unreliable because they contain many codecs written from many sources, probably with many bugs. So the best solution might be to add some sort of timeout to try and detect when Ffmpeg has gone wrong.
I think the following mitigations could be considered:
- Adding a timeout in ffmpeg_cmdline_utils for the same reason (I will add this to my todo list but I might never implement this -- I am mostly an FPGA engineer so I am a bit scared by software concurrency)
- Adding a timeout in czkawka on file read operations (or maybe only on media decoding as this is probably least reliable) -- in case dependent library authors cannot guarantee that their libraries will never get stuck :)
- Add an incremental saving option while loading large datasets. Perhaps every 100 videos or every 5 minutes.
- (maybe if possible, to round down the percentage in progress bars, so that czkawka does not tell a conflicting story about whether processing is complete or not)
To help your problem now:
- You will need to find the problem video and make sure Czkawka does not process it.
- If you are OK with scripting, you can use ffmpeg with some different settings to detect errors. Try googling "ffmpeg check corrupted" or something like that. But since you have 6k videos you will need to write a loop over all your videos and to process all of the results
- If not, then maybe consider loading the videos into Czkawka in small batches (say 500 at a time). If the same problem occurs then you can narrow down which video causes the error. Czkawka has a cache file that remembers which videos it has already seen before, so once you have loaded every chunk of 500 videos, you can finally search all 6k videos to find the duplicates.
If you can find the broken video and you are willing to share it, it woulld be nice if you can upload it somewhere. It can be used to understand if any bug exists in Czkawka, ffmpeg_cmdline_utils, or ffmpeg. The video would have to contain no personal information because I do not want to assume responsibility for protecting any person's privacy.
yaaaaa
So I tried to use the tool to check duplicates videos and it's stuck at 100%. It took like 6 hours to get there, notice that one of the ffmpeg windows didn't close up during the scan where as thousands opened then closed almost instantly during that time.