Limit number of threads to maintain system stability

[VL4DST3R]

First of all, i just wanted to say exceptional work on reviving/adding missing functionality to winthumbs, you are amazing.

However, i have encountered an issue, which could either be taken as a testament of just HOW multithreaded this process can get! but definitely can cause some issues if left unchecked. Namely, if you start too many instances of thumbnail generation, the amount of processes the program can spawn spirals in the hundreds or thousands, especially if you have SSDs in there. Nevermind the fact it pummels the cpu on all cores (12c 24t here) and chews through tens of gigs of ram like its nothing- that much is expected, but eventually it brings the entire operating system to a standstill, ultimately becoming unresponsive.

I have also encountered the issue where in such a state, any I/O operations within windows either fail, become corrupted (found files filled with NULLs instead of actual data if caught mid-generation) or not happen at all.

There is a chance if you manage to stop it in time and wait a while, it MAY unfreeze, but most of the time it requires a hard reboot (restarting via normal means doesn't work even if it doesn't render the explorer itself inoperable as well, which it often does.)

To add insult to injury, when this happens windows also wipes the entire thumbnail database, forcing you to start fresh 🤡.

I am also somewhat at fault here; given the limitations of the original version of the tool, i used to bootleg-multithread my thumbnail generation throughout my pc by starting separate instances for almost every subfolder where it felt appropriate. Needless to say, doing the same here is a guaranteed one-way ticket to a locked system. I still need to get used to this.

[Mfarooq360]

Hello, I'm glad you're enjoying the work I've done so far. I've been learning how to code for the first time due to this project's abandonment and wanted to be able to fix and maintain it along with other projects hopefully in the future.

I've noticed the same behavior on my system where it would cause my inputs to stop working and force me to restart, I thought that was due to debugging on Visual Studio, but actually while the program seems to run fine for a bit, suddenly the Remote Procedure Call service would start using the CPU a lot more than normal and the whole system stops responding.

It turns out that in the original code, the maximum amount of parallelism is set to 2048 where it should instead be set to use the amount of logical processors in a user's system.

I'm currently testing this fix out and it seems to be working great with the system suffering from less lag as well. I'm also trying to fix a couple of other things in the code such as the program crashing if a file is not found after scanning it.

Another fix I'm making is for the program to set itself to "Below Normal" process priority so the system doesn't lag as much while it's running. This helps a lot for generating .svg thumbnails since it uses PowerToys to do so and it causes a lot of lag.

Hopefully, I'll be able to release a new version soon once I'm able to implement these changes.

[VL4DST3R]

If it is of any use, i've noticed that the amount of threads seem to rack up faster for slower drives, i'm guessing since they take longer to actually process, but the tool keeps assigning threads for new ones faster than they get removed. On SSDs this seems to be less of a problem (or maybe i just didn't have enough files to process), here, an example:

first one is running on an SSD, second on a HDD

[Mfarooq360]

Yeah that's good to know as I've been testing on an NVME drive for full drive scanning, I also found out that the reason the system freezes is not always because of the excess amount of processes, but rather that windows' thumbnail generation has a chance of getting stuck on a file which holds up the entire system for some reason. This may be due to processing every file type rather than only processing files that can generate thumbnails.

So far I'm working to fix a few other issues that I'm noticing are causing problems and hopefully soon I'll be able to fix as many as possible before releasing the next version.

[Devicetron]

First of all, i just wanted to say exceptional work on reviving/adding missing functionality to winthumbs, you are amazing.

However, i have encountered an issue, which could either be taken as a testament of just HOW multithreaded this process can get! but definitely can cause some issues if left unchecked. Namely, if you start too many instances of thumbnail generation, the amount of processes the program can spawn spirals in the hundreds or thousands, especially if you have SSDs in there. Nevermind the fact it pummels the cpu on all cores (12c 24t here) and chews through tens of gigs of ram like its nothing- that much is expected, but eventually it brings the entire operating system to a standstill, ultimately becoming unresponsive.

I have also encountered the issue where in such a state, any I/O operations within windows either fail, become corrupted (found files filled with NULLs instead of actual data if caught mid-generation) or not happen at all.

There is a chance if you manage to stop it in time and wait a while, it MAY unfreeze, but most of the time it requires a hard reboot (restarting via normal means doesn't work even if it doesn't render the explorer itself inoperable as well, which it often does.)

To add insult to injury, when this happens windows also wipes the entire thumbnail database, forcing you to start fresh 🤡.

I am also somewhat at fault here; given the limitations of the original version of the tool, i used to bootleg-multithread my thumbnail generation throughout my pc by starting separate instances for almost every subfolder where it felt appropriate. Needless to say, doing the same here is a guaranteed one-way ticket to a locked system. I still need to get used to this.

https://www.partitionwizard.com/partitionmagic/prevent-windows-10-from-deleting-thumbnail-cache.html this will work keeping your thumbnails intact.

Thank you Mfarooq360 for your work in WinThumbsPreloader you made it my favorite!

[Mfarooq360]

After a while of searching for the cause of the system freezes, I've finally found and fixed the problem and it appears to be working great now. I'm currently adding a few other features and testing for stability, so a release should be out soon.

[VL4DST3R]

https://www.partitionwizard.com/partitionmagic/prevent-windows-10-from-deleting-thumbnail-cache.html this will work keeping your thumbnails intact.

I know about that settings but does nothing if a crash occurs. Thumb.db is one of the first "cached" things windows deletes if he craps the bed in any way (BSOD or otherwise), so it does nothing in this case.

[Mfarooq360]

I've made a new release which should fix this problem, let me know if it works well for you.

[VL4DST3R]

I planned to back up the dbs

I had a similar attempt and made myself a script that backs up all the .db files that windows generates, then another that terminates the explorer, plonks them black overwriting anything present in that folder, then restarts explorer.

I believe it worked like- maybe once and now i get access denied when trying to overwrite them, so be mindful this may not be as straightforward as we'd hope.

Not to mention windows may very well not recognise the previous dbs ahe seems to discard them at the smalles inconsistency, plus the db contents themselves seem to be proprietary (i.e. you cannot open them yourself)

[VL4DST3R]

I've made a new release which should fix this problem, let me know if it works well for you.

testing right now, so far so good, 4 processes started, 4 entries in the task manager, as expected:

disk usage pegged at 100% (expected) while ssds at ~10% (intended?)

Assuming all the above look as expected per your new changes, feel free to close this ticket, i am curious about the ssd usage though.

I did encounter the case where if intrerrupted (closing the generation popup) it remains stuck with an "inifinte loading" bar animation showing instead of the actual progress and has to be forece-closed manually, but this is not a new issue, it happened from time to time in the original as well. Maybe consider adding better handling for intrerupting operations?

[Mfarooq360]

disk usage pegged at 100% (expected) while ssds at ~10% (intended?)

I've changed the way the program was multithreaded back to a similar way it was done originally since the method used in the previous fork would cause RPC to crash.

Now it seems to use less CPU and RAM while using the drive more, although it won't use the full CPU even when increasing the max amount of threads.

I did encounter the case where if intrerrupted (closing the generation popup) it remains stuck with an "inifinte loading" bar animation showing instead of the actual progress and has to be forece-closed manually

I've noticed that as well and have attempted to fix it previously, though it appears now that it only closes sometimes. I'll be working on getting the cancellation to be more stable and will make a release if there's anything else that needs to be fixed.

This issue should be fixed so I'll close this ticket. I appreciate your help and am glad I was able to get this working better.

[VL4DST3R]

I've changed the way the program was multithreaded back to a similar way it was done originally since the method used in the previous fork would cause RPC to crash.

Now i'm curious though, was it indeed substantially faster? Since mine would cause a freeze/crash, i never got to actually check the end result if it did generate those correctly or not. Maybe there's still room to dabble into this, maybe as an optional extra flag to further boost multithreading (again, assuming its worth it), maybe providing a max number of allowed threads per thumbnail generation instance?

Regardless, glad to be of help! Thank you again for bringing life into this otherwise dead project!

[Mfarooq360]

Now i'm curious though, was it indeed substantially faster?

I've made a video comparison between the two versions along with modifications for each. 1 is the previous version while 3 is the current version. No limit signifies a fixed limit of 2048 threads as was first made 2 forks prior.

The previous versions and fork would introduce significant overhead and would cause problems with RPC.

https://github.com/Mfarooq360/WinThumbsPreloader/assets/97059075/2dc13cc8-1b30-41d8-8b0f-fbc603a75304

It appears that the modification made in 5 could save 2 minutes by using 2x the system's threads instead of 1x. This was done on an NVME drive, though I haven't tested it on a hard drive yet. If it's stable, I'll be able to implement it into the next version.

I've tried other methods that appear to use the CPU more, but even though they were more stable, the increased CPU usage was because of overhead, so it would be slower.

I'll be experimenting more with getting it faster, but it seems that I may be hitting windows' COM limit on thumbnail generation.

Once again, thanks for helping me fix this issue and I hope to continue improving on this project!

[Mfarooq360]

I think letting us choose the # of threads is ideal. Like Robocopy MT switch. If you wanted to add to the GUI could be a drop-down or input box. I have a pretty beefy machine and running a few folders concurrently with the first version brought it to a halt, but, I can surely get more than 2 out of it.

I could implement the ability to change the thread count, but based on the comparison video it seems that changing the thread count in the newer versions doesn't seem to affect it much since it's now fast enough to be bottlenecked by windows' thumbnail generation which is preventing it from using more threads than windows will allow.

I may still add it in the next release if it'll help certain use cases like as shown in the video where setting the thread count to double the system's threads may reduce wait time.

[Mfarooq360]

Ahh...That;s what I get for only half-reading. Thanks man! It is dreadfully slow, huh? I love Windows, in general, but, they make some curious decisions sometimes...

Indeed, I wish Windows would make their thumbnail generation multi-threaded. Perhaps there is a way to let this program run silently whenever a folder is opened so thumbnails are generated fast automatically. It would be nice to get that to work.

[Mfarooq360]

Also, I think I mentioned before, but, if you use Icarus it will give you thumbs Windows doesn't have yet

I'm trying Icaros out and I'm liking the increased speed and caching feature. It seems that preloading with it is faster for videos but slower for photos. I may do more tests with it to see if it speeds up the thumbnail preloader. I did notice that the preloader also doesn't use Icaros when running as administrator which also happened with PowerToys for some reason.

I run it as a scheduled task at idle with the -s switch

This is something I would like to add into the GUI but it may take a while to figure out how it should be implemented.

[VL4DST3R]

Following this thread and considering the amount of functionality you are considering for the tool, i reckon consolidating all of it inside an actual tool where you can configure all this (directories, thread counts, distinction between drive types, now with Icarus maybe as as an alternative to windows's stuff)- is the best course of action. With proper config file to store all the user preferences.

Can i just fanboy again and point out how glad i am that this topic of caching thumbnails is receiving such attention and care from you? Thank you again for all this.

[Mfarooq360]

Hello, just wanted to give an update on what I'm currently doing with the program.

I'm adding in a ton of new features and I'm constantly finding new things I can add. I've been implementing as many things that were mentioned here and from the main fork's issues page. Some of the changes include a new settings form which also includes changing extensions in the app, selecting a folder to preload from the app, preloading folder icons that contain files with thumbnails, choosing maximum thread count, adding support for multiple paths on the command line along with making it far more forgiving, windows task scheduler support to preload automatically at user set intervals, and many more features!

I'm thinking when I'm done, I may make a new repository page so it can be discoverable on Google. I'm very excited to release this new version soon, and I appreciate your thanks!

[VL4DST3R]

That's great to hear! Can't wait to see what you've made! Any ETA on the new version? I would love to test it out.

ALSO since you pointed out you are adding everything and the kitchen sink- I'm not sure if this would even be in the scope of such a tool, but windows has this bad habit of nuking the thumbnails .db whenever something bad happens. Not enough space? Boom. Something explorer-related crashes? Boom. You disconnect a drive? Bye bye thumbs for that drive.

From what i gather those get stared in %AppData%\Local\Microsoft\Windows\Explorer in various db for different resolutions. I have attempted to create a backup/snapshot of sorts of the files here, but windows is very anal about swapping them back when he inevitably craps the bed and wipes them. I have managed to do it, but seems to be very inconsistent.

Would it be too much of a hassle to include some way to back up and restore those files (after you generate thumbnails) to skip the need to generate them again in the future when it happens again?

[Mfarooq360]

I'm thinking I may be able to get the release out by the end of the month as I still have some features I have to add and a bunch of debugging to do with some of these features. This may change depending on if I find any other issues.

I'll have to do some research and experimenting in order to see if I can stop Windows from automatically deleting the thumbnail cache. So far I've seen one post where people have been having higher levels of success than previously mentioned methods. https://answers.microsoft.com/en-us/windows/forum/all/stop-windows-auto-deleting-thumbnail-cache-revised/5d09ef73-7777-4d59-8e47-965379fb91f0?page=1

If possible, could you help test the fixes in that post and see if it helps to fix it? If it does, I may be able to try and implement it. I'll also be testing it when I have time, though I'm still busy with working on the program and with college. If it doesn't work I'll see if I can implement some kind of alternative.

[VL4DST3R]

Oh I'm familiar with that thread and I have already applied a number of suggestions from there, i have a set of bats that sets those autorun flags to 0 and removed (some of) the scheduled tasks as well. Problem is: you don't necessarily always want to nuke ANY form of cleaning, i wouldn't have any issues with cleaning up logs or other redundant data, but AFAIK there is no way to more granularity choose (i.e. ignore only thumbnails).

Further, suggestions in the article about taking ownership of various windows files and renaming/removing them are not reliable, as if you ever need to run an integrity check or an update, they will just be regenerated back and you will have to do this flight-check all over the system again, which would be painful.

So i figured instead of trying to play wack-a-mole with it on all fronts, (considering as i said, you can trigger it even by managing to hang up your pc in certain way) instead keeping a backup of its DBs and just replacing them when its needed would be far simpler. When unmounting drives as well, there is nothing as far as i know to prevent it from immediatedly flushing the tumb cache for that device.

The last two mentions about increasing icon cache and a tool to fix them i have not tried. The first one, like they guy said, i doubt it makes a difference, atm mine take about 5 gb of space and windows seem to play nice with it. As for the second one to fix the cache, i was not aware its even a possibility, as the moment something goes bad with them they just get nuked and rebuilt, so i have no idea what use case he's covering there.

All in all, my advice to you would be to focus on saving and swapping back the databases themselves and leave it at that for now, as i reckon would be simpler for you as well to implement, assuming you manage to do it better than i have where it doesnt let me put them back.

My current scripts for doing this are as follows:

backup

REM Copy .db files from the source path to the current folder
set "sourcePath=C:\Users\Vlad\AppData\Local\Microsoft\Windows\Explorer"
set "destinationPath=%~dp0"

xcopy "%sourcePath%\*.db" "%destinationPath%" /Y /Q

echo File copy completed!
pause

and to restore

REM Close Windows Explorer
taskkill /f /im explorer.exe
taskkill /f /im taskmgr.exe

REM Wait for Windows Explorer to close
timeout /t 2 >nul

REM Set source and destination paths
set "sourcePath=%~dp0"
set "destinationPath=C:\Users\Vlad\AppData\Local\Microsoft\Windows\Explorer"

REM Delete existing .db files in the destination directory
for %%F in ("%destinationPath%\*.db") do (
    del /Q "%%F"
)
timeout /t 2 >nul
for %%F in ("%destinationPath%\*.db") do (
    del /Q "%%F"
)
timeout /t 2 >nul

REM Copy .db files to the destination path
xcopy "%sourcePath%\*.db" "%destinationPath%" /Y /Q
timeout /t 2 >nul

REM Restart Windows Explorer
start explorer.exe

REM Open an Explorer window to the folder where the batch script is located
explorer.exe /select,"%scriptPath%\Restore (RUN AS ADMINISTRATOR).bat"

echo File copy completed!
timeout /t 1 >nul
pause