Randomly hangs and destroys save state file, but continues to print status without errors

[oakwhiz]

Tested this program out and I've had this issue twice now:

• Full disclosure - Compiled with -O3 and native optimizations etc. rather than defaults
• glibc and libssl etc. are compiled with native optimizations as well
• Launched in character generating mode, with -v option for periodic status
• Collecting stderr & stdout in log file
• After around 37 billion iterations, I noticed that it stopped iterating, but the -v status output still continues periodically, just with no change in the current status report.
• Nothing is reported in stderr and there is nothing in dmesg or system logs that coincides. The system is running Debian 12 (bookworm) and is headless with no additional software running on the system besides the absolute basics. There are no other services e.g. webserver etc.
• The save state file is replaced with a 0 byte file when this occurs. The durability of writing a new state is called into question here. It shouldn't be possible for the old state to be lost while writing the new state, as this kind of makes the state saving mechanism a lot less useful. Some suggestions would be either double-buffering between 2 files, or leaning on a battle-tested implementation like sqlite which can be made very durable - though it may be overkill here for a simple and lean program like this.

I highly recommend logging & monitoring, and having a process that backs up the save state file, and tails the logs periodically to see if the current password has stopped advancing. This would be a signal to kill the process, try to restore the save state file, etc.

This is making it very difficult to run the search unattended, I'm curious if anyone has had a similar experience.

[glv2]

I made some modifications (1c2bbc518d05c72a41c1a93200d810c20256eda3) that should solve a possible deadlock when saving the state file. Could you check if you can still reproduce the issue with the new code?

[oakwhiz]

I will report back on the progress with the patch applied. Average time to failure was about 4 hours roughly.

[oakwhiz]

I made some modifications (1c2bbc5) that should solve a possible deadlock when saving the state file. Could you check if you can still reproduce the issue with the new code?

So far there have been no issues. However there is a race condition in the -v output where the current password is completely blank. It seems to have a roughly 1% chance of occurring, while running with 8 worker threads.

[glv2]

Thanks for testing. The blank password in the status output should be fixed with 32ee7cd06010908b36a1dee04b51762da47020b9.