[Bugs Masterlist] Backlog

[VL4DST3R]

Hey there! Just testing the new pre-release version and off the bat I just wanted to ask if you could provide a portable version (without an installer) for people like me who want to keep tools like this portable! Just save all the stuff in the same folder (and load from) as the exe, if you're not doing that already. Add the portable version on the Releases page as well please!

Secondly, I encountered this unhandled exception when opening the backup folder before it exists:

************** Exception Text **************
System.IO.FileNotFoundException: Could not find file 'E:\Programs\WinThumbsPreloader\New folder\Explorer Backup'.
File name: 'E:\Programs\WinThumbsPreloader\New folder\Explorer Backup'
   at System.IO.FileSystem.CreateDirectory(String fullPath, Byte[] securityDescriptor)
   at System.IO.Directory.CreateDirectory(String path)
   at WinThumbsPreloader.CacheForm.ClearButton_Click(Object sender, EventArgs e)
   at System.Windows.Forms.Control.OnClick(EventArgs e)
   at System.Windows.Forms.Button.OnClick(EventArgs e)
   at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
   at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
   at System.Windows.Forms.Control.WndProc(Message& m)
   at System.Windows.Forms.ButtonBase.WndProc(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, WM msg, IntPtr wparam, IntPtr lparam)

[Mfarooq360]

Hi, thanks for letting me know about this, It seemed to not crash when I tested it but I know how to fix it. I wanted to put the portable version out, but for some reason it wouldn't work due to updating to .net 6, which forced it to require dlls to be with it, so I'll try to see if I can force it to be self-contained again.

By the way did V2 help to fix the issue you had with recursive preloading not working?

[Mfarooq360]

By the way did V2 help to fix the issue you had with recursive preloading not working?

Actually I may have discovered the problem when trying to do it recursively in the main window, it's interesting how it doesn't happen anywhere else so I'll try to make the preloading happen from within the program instead of starting a new one and see if it helps.

[VL4DST3R]

Sorry I wasn't really able to play with this lately, but I'm glad you found something. So what causes it?

[Mfarooq360]

Sorry I wasn't really able to play with this lately, but I'm glad you found something. So what causes it?

It seems to not work when launching a path as a separate process, but only in the main form, so I'm currently integrating preloading within the same process which involves making the cancellation not just kill the entire program when one preloader is done, along with preventing each preloader from affecting one another.

I'm intrigued on why this was an issue in the old version since it just ran in the same program like what's fixing it now, and when I tested it, recursive preloading seemed to work fine in the previous version.

[Mfarooq360]

Portable version should be up and working now, file size may be slightly larger due to .NET 6.

[VL4DST3R]

Will test the first chance i get and report back! Thanks again for all the work!

[VL4DST3R]

Back to report my experiences with the new portable version! sadly I cannot seem to get it to work at all from the UI via the "Preload" button - I give it a folder but seems to do nothing? I've also added a few other mentions on #5

[Mfarooq360]

Back to report my experiences with the new portable version! sadly I cannot seem to get it to work at all from the UI via the "Preload" button - I give it a folder but seems to do nothing? I've also added a few other mentions on #5

It seems that the portable version may have caused some new bugs to show up, and since I didn't get to test it, I'm guessing some issues with permissions may be occurring.

On my current project I set the preload option to run in the same program instead of launching a new instance like how the current beta is doing, so it should help fix that, for the scheduler I have a lot of fixing to do, though I haven't seen that error message occur when using it so I'm wondering if its another bug with the portable version, but those same task settings saved when I tried them on the portable version.

I am not able to save or delete the scheduler task on the portable version as it says access is denied so it looks like I'm going to have to set the portable version to always run as administrator, see if doing that fixes some of the issues for now. In the meantime, I will be working on these issues along with other fixes and features, though I may make another beta once I get a few of them done to have a more stable beta out sooner.

[VL4DST3R]

I came back after further testing to discover that more exotic paths now also seem to not work (e.g. containing ~ or ( ))

I haven't seen that error message occur when using it so I'm wondering if its another bug with the portable version, but those same task settings saved when I tried them on the portable version.

I feel it may be due to formatting issues for different locales (e.g. 12/24 hour formats, dividers, etc) that may be causing this.

I'm going to have to set the portable version to always run as administrator

I also tried this the first thing I encountered issues, but it didn't seem to make much of a difference.

[VL4DST3R]

Finally, now that I got some more time to test it (and lost my thumb cache yet again so I have to rebuild them all XD) would you like me to open separate tickets for small things like the ones mentioned so far? Or rather keep this all here as a sort of back and forth as we test/discover new issues?

[Mfarooq360]

Finally, now that I got some more time to test it (and lost my thumb cache yet again so I have to rebuild them all XD) would you like me to open separate tickets for small things like the ones mentioned so far? Or rather keep this all here as a sort of back and forth as we test/discover new issues?

Yeah if you'd like we could keep the bugs at this issue and perhaps the features at the other since it's a beta so there would probably be a ton of issues to fix. I'll be looking into all of these problems, cause it seems like the portable version is being far buggier than the setup version as most of these problems weren't discovered when I tried them in the regular version.

[VL4DST3R]

Yeah if you'd like we could keep the bugs at this issue and perhaps the features at the other since it's a beta so there would probably be a ton of issues to fix.

Certainly, I will rename the two tickets accordingly.

it seems like the portable version is being far buggier than the setup version as most of these problems weren't discovered when I tried them in the regular version.

And I'm afraid I won't be using the installer version at all, so I can't really stress enough the importance of having the portable one work properly!

Now, I don't mean to impose, but maybe eventually you could merge the two version behaviors to be more in common, to avoid having to maintain two versions of it (and risk such bugs slipping between them), which I guess would mean basically maintaining just the portable version with the extra features/integrations available if installed via setup.

Actually I have to ask, why have an installable version at all? What can it do that the portable version cannot?

[Mfarooq360]

Actually I have to ask, why have an installable version at all? What can it do that the portable version cannot?

I believe the original purpose of the setup file was to add the context menus to folders, and then I added an environment path variable so it could be used in the command line without navigating to the directory along with adding the context menu to the drives themselves. It also allows the program to do things like use the Task Scheduler function without administrator permissions.

Now, I don't mean to impose, but maybe eventually you could merge the two version behaviors to be more in common, to avoid having to maintain two versions of it (and risk such bugs slipping between them), which I guess would mean basically maintaining just the portable version with the extra features/integrations available if installed via setup.

In theory the setup may not actually be needed, though I would have to integrate a toggle inside the GUI to enable context menus and the environment path variable, or I can also have it enabled by default, though the user would have to run the GUI first before those things occur, and if the exe is moved the environment path variable and running as administrator would probably also need to be reset, where the environment path would also affect the context menus and the task scheduler as well.

It would probably be simpler then to have the setup alongside the portable version so that the user doesn't have to go through all the steps of setup manually, though now that I know what problems have popped up from the new portable version compared to the old portable version, which was a lot simpler, I can try my best to compensate for some of these new issues.

Running as Administrator may also end up being a requirement as some features of the program such as the Task Scheduler and probably the previously mentioned things would need more permissions, ~though running as administrator wouldn't be able to be toggled in the program~ or kept set in the compatibility settings when the file is moved around, and would need to be reset manually, which could be helped by either a prompt on startup or giving error messages when access is denied.

Edit: I've found out that setting the app to publish as a self-contained executable has caused a new warning message to popup on the admin toggle button code, which helped to describe how to modify it to now set as admin within the application even with the portable version, though it still doesn't keep the setting if the exe is moved from the location where it was set.

I may also end up changing some of the error messages to indicate that the program needs to be run as administrator if it can't perform an action such as setting or deleting a task from the Task Scheduler. If I end up adding a toggle to enable context menus and the environment path, it would end up breaking functionality if the exe is moved around, which may be helped by checking for the application path at the startup of the GUI and adjusting it accordingly if it is able to discover the previous paths.

Overall, I have a lot more work that needs to be done to this application, though I may try putting out some betas when I get enough fixed in order to hopefully decrease wait times and prevent keeping people from being stuck on a buggy version.

[VL4DST3R]

Came back to report another issue I encountered, this time when restoring from cache/a backup. While backing up the files seemed to have worked fine, restoring it only works for a very small amount:

Attempting to restore again does nothing to increase it and if I insisted a few more times it silently crashed the entire application.

Which actually brings me to another point: is there no way to "minimize to tray" unless started with windows? Closing it seems to shut it down completely.

EDIT: Apparently, after the crash it managed to restore more from the total backup amount, and repeating the process a few more times ultimately got it within spitting distance from the full backup:

[VL4DST3R]

I came here to add on the previous issue, from what I understand, the issue preventing the restore from working properly is an error reported by windows as: The requested operation cannot be performed on a file with a user-mapped section open..

I found multiple mentions of this issue online when I made my own bat script to achieve more or less what you implemented with the cache function, but sadly I never found a reliable way to properly release those hooks. I tried terminating all explorer processes and a few others, even multiple times, but it seems unreliable.

Have you did anything in this regard with your implementation for the restore function?

[Mfarooq360]

Came back to report another issue I encountered, this time when restoring from cache/a backup. While backing up the files seemed to have worked fine, restoring it only works for a very small amount:

Attempting to restore again does nothing to increase it and if I insisted a few more times it silently crashed the entire application.

Interestingly, it did seem to restore the thumbnail cache fully when I tried it at a size of 2.5GB by deleting and restoring a few times to see if I could reproduce it, though I'm not sure if this is now happening because of the size of the thumbnail cache backup or if Windows 10 does things differently, but I'll be testing some other methods on other computers to see what I can do to fix it. If all else fails I may try adding an option to force restore by closing explorer for the transfer of each file to prevent it from reopening while the restore occurs. Which actually brings me to another point: is there no way to "minimize to tray" unless started with windows? Closing it seems to shut it down completely.

To minimize to tray, the minimize window button needs to be pressed and then it should be in the icon tray. EDIT: Apparently, after the crash it managed to restore more from the total backup amount, and repeating the process a few more times ultimately got it within spitting distance from the full backup:

The program should be trying to constantly restore the files automatically as long as the cache size is smaller than the backup, so that's interesting how doing it manually made it work more.

I came here to add on the previous issue, from what I understand, the issue preventing the restore from working properly is an error reported by windows as: The requested operation cannot be performed on a file with a user-mapped section open..

I found multiple mentions of this issue online when I made my own bat script to achieve more or less what you implemented with the cache function, but sadly I never found a reliable way to properly release those hooks. I tried terminating all explorer processes and a few others, even multiple times, but it seems unreliable.

Have you did anything in this regard with your implementation for the restore function?

I encountered this when trying the restore once, so I made it keep trying to restore until the backup and cache sizes were the same, which seemed to work when I tried it, though I'm going to try to look into this to see what I can do to make it work.

[VL4DST3R]

If all else fails I may try adding an option to force restore by closing explorer for the transfer of each file to prevent it from reopening while the restore occurs.

Unless you have a deeper understanding on how windows releases its hooks on those files, then I don't think there's any other method to achieve this more elegantly, hence why I went straight for terminating processes, and even then as I said, I didn't manage to properly account for all of them.

I'm guessing those files may be accessed by multiple varying processes at one time, so finding exactly which one are currently using any one of them requires a bit of iterating: checking each individual .db (because as i mentioned above, some DID replace without an issue) if there are processes are using it, closing them (repeatedly if need be throughout the restore process) and only when confirmed to be fully "unhooked" proceed to replace it with the backup.

To minimize to tray, the minimize window button needs to be pressed and then it should be in the icon tray.

I tried before coming here to ask - minimizing it this way just minimizes it to the taskbar for me, like any regular program (not the tray)

The program should be trying to constantly restore the files automatically as long as the cache size is smaller than the backup, so that's interesting how doing it manually made it work more.

I didn't have it set up to restore automatically (Auto Restore unticked), figured it would be best to do it myself when it loses its cache again so I can see if it works properly or not. Or is this not what you're referring to?

I encountered this when trying the restore once, so I made it keep trying to restore until the backup and cache sizes were the same, which seemed to work when I tried it, though I'm going to try to look into this to see what I can do to make it work.

Oh I see. You know, checking back now, I see both the cache and backup at the same size, but at this point I'm not sure if it just auto-backup'd my "almost" restored thumbnails and hence got them to be the same size this way or if your implementation did indeed eventually work. Whichever it is, I think you should make the "restore" process inform the user about this, maybe provide a loading bar as you repeatedly make sure each .db gets properly swapped with the backup.

[VL4DST3R]

Indeed, restore function seems to work only if explorer process is terminated, otherwise it hammers the drive (ssd in my case) where the backup resides, reading files constantly at max speed, but fails to transfer them to the cache folder (transfer stops at about 25% but can vary). Terminating the explorer process allows it to copy fully.

[Mfarooq360]

Indeed, restore function seems to work only if explorer process is terminated, otherwise it hammers the drive (ssd in my case) where the backup resides, reading files constantly at max speed, but fails to transfer them to the cache folder (transfer stops at about 25% but can vary). Terminating the explorer process allows it to copy fully.

Seeing this, I'll be adding a force restore option to the restore button in case the regular restore fails along with a retry cap, though I also want to set the portable version to only run as administrator since it'll prevent the permission bugs with the scheduler and perhaps will help with regular cache restores as it seems to work fine for my 2.5G thumbnail cache.

Here's an example of how it's working for me currently:

https://github.com/Mfarooq360/WinThumbsPreloader-V2/assets/97059075/2a105b7e-f435-4955-8872-8eb0d0b99f48

Currently, I'm searching for any large bugs to fix before I publish the next beta, so it should be out soon.

[VL4DST3R]

I also want to set the portable version to only run as administrator since it'll prevent the permission bugs with the scheduler and perhaps will help with regular cache restores as it seems to work fine for my 2.5G thumbnail cache.

Now that you mentioned, I haven't tried yet running it as admin, I'll do that the next time I lose my thumbnails and report back.

Currently, I'm searching for any large bugs to fix before I publish the next beta, so it should be out soon.

What have you got taken care of thus far? Anything from the stuff discussed here and on the suggestions list?

[Mfarooq360]

What have you got taken care of thus far? Anything from the stuff discussed here and on the suggestions list?

I've now released the next beta which should have the changelog of what's new compared to the previous beta. Let me know if the portable version is any more stable and if any bugs are found.

Unless you have a deeper understanding on how windows releases its hooks on those files, then I don't think there's any other method to achieve this more elegantly, hence why I went straight for terminating processes, and even then as I said, I didn't manage to properly account for all of them.

I've also looked into this and it turns out that ending file handles is considered a dangerous thing to do and is also extremely difficult to actually implement and the best way seems to be to just end the process using the file, so now when you hold shift you can do a force restore for the cache which will close explorer before trying to restore the cache. I'm not able to test this very well as the regular restore has worked fine so I'd like to hear if it works better for you.

The backup and restore buttons also have progress bars now which should help with seeing the progress of the backup and restore functions to ensure they are still working or trying to work.

I tried before coming here to ask - minimizing it this way just minimizes it to the taskbar for me, like any regular program (not the tray)

I believe currently you are only able to minimize to tray when you are in the cache form, I'm trying to find out how to make it work on any form and turn it into a setting.

[VL4DST3R]

I've also looked into this and it turns out that ending file handles is considered a dangerous thing to do and is also extremely difficult to actually implement and the best way seems to be to just end the process using the file, so now when you hold shift you can do a force restore for the cache which will close explorer before trying to restore the cache. I'm not able to test this very well as the regular restore has worked fine so I'd like to hear if it works better for you.

I believe this is one of the cases where different program setups (e.g. network locations, backup utilities) results in system file handles being treated differently from machine to machine.

I'm pretty sure terminating the explorer process(es) and possibly any COM Surrogate spawned or left hanging by file operations should cover 99% of handles that may be open. However this may need to be done a few times as sometimes I've seen stuff still kicking or starting back up.

If I were you, I'd go like this:

1. try to replace the cache "gracefully" once.
2. on failure show the user a popup about being unable to do so and offer a choice if they are ok with the tool forcefully releaseing the hooks to allow the transfer (with the potential risk of data loss)
3. if agreed then go ahead and force-close everything that may be interfering as I mentioned above.

I reckon this way you should cover pretty much everyone, lucky or not.

Regardless, I will report back the next time I get a chance to test the new version and report all my findings.

[VL4DST3R]

Back again after a new wipe:

While the regular mode proved to be equally unsuccessful, the new "shift" mode worked a lot better, but I had to run it twice to restore the databases fully. Still, a lot better than before and with no crashing this time!

However I have noticed another issue, but it's not exclusive to this release. I've noticed this ever since I started generating thumbnails via winthumbs: sometimes even though the databases are there, the thumbnails are being regenerated as if the db's were still gone.

I'm guessing this is due to the os not always reading from the database and instead whatever cached (e.g. .db-shm/db-wal file or other equivalent) untill the restored files kick in. Any idea how can this be triggered to happen right away? I haven't yet managed to reproduce exactly when the hand-off happens but you can test this by deleting and restoring the cache then quickly checking a known cached folder and find them generating one by one again.

[VL4DST3R]

Found a new bug: It seems path with spaces are not being handled properly, within quotes or not. Old versions of the program (pre-2.0) worked as intended (i still keep a few forks and older versions for reference when testing)

[Mfarooq360]

Found a new bug: It seems path with spaces are not being handled properly, within quotes or not. Old versions of the program (pre-2.0) worked as intended (i still keep a few forks and older versions for reference when testing)

Found the problem, gonna fix this along with currently researching the previously mentioned issue.

[Mfarooq360]

Found a new bug: It seems path with spaces are not being handled properly, within quotes or not. Old versions of the program (pre-2.0) worked as intended (i still keep a few forks and older versions for reference when testing)

Found the problem, gonna fix this along with currently researching the previously mentioned issue.

Took a while to make it work well, but that problem should now be fixed for the next beta. I also fixed some other issues with the preload button not working sometimes.

However I have noticed another issue, but it's not exclusive to this release. I've noticed this ever since I started generating thumbnails via winthumbs: sometimes even though the databases are there, the thumbnails are being regenerated as if the db's were still gone.

I'm guessing this is due to the os not always reading from the database and instead whatever cached (e.g. .db-shm/db-wal file or other equivalent) untill the restored files kick in. Any idea how can this be triggered to happen right away? I haven't yet managed to reproduce exactly when the hand-off happens but you can test this by deleting and restoring the cache then quickly checking a known cached folder and find them generating one by one again.

I also did some research into the previous issue and there are a few options that I've been looking at. The first thing I thought of was restarting explorer after deleting the thumbs cache to see if it would start reading the cache files right after deleting them.

The second thing I found out was that it is possible to use windows' disk cleanup in the program to clear the thumbnail cache instead of manually deleting the dbs, but the implementation requires a registry tweak which involves setting a sageset flag number to call the thumbs cache cleaner in particular to clean up. This could work and may help to solve the problem, though I'd have to check its specific behavior to ensure that functions like restoring the thumbs cache don't break as a result as it deletes the thumbscache before restoring.

[VL4DST3R]

problem should now be fixed for the next beta

Any ETA for the release? Can't find it under releases yet.

but the implementation requires a registry tweak which involves setting a sageset flag number to call the thumbs cache cleaner in particular to clean up.

I know windows has a few scheduled tasks that autonomously run disk cleanup with certain settings, is it not something that can be done this way? Or you don't have such granular control there?

All in all very interesting finds. Somewhat unrelated but may I ask, how are you this knowledgeable about the underlying stuff about how all this stuff works? Or is just educated guess after educated guess? I'm asking because this whole topic is something I've looked up on and off throughout the years and the data available out there, at least AFAIK, is pretty much nonexistent.

[Mfarooq360]

Any ETA for the release? Can't find it under releases yet.

I'm targeting this weekend to release the next beta if all goes well as this is a substantial bug that affects everything related to the preloader. So far it seems to work for paths with spaces and special characters as far as my testing goes.

I know windows has a few scheduled tasks that autonomously run disk cleanup with certain settings, is it not something that can be done this way? Or you don't have such granular control there?

I don't see any prescheduled tasks for disk cleanup in the task scheduler, though it seems the method I talked about before applies similarly when creating a new task except you have to manually specify what folders you want to clear in a dialog before the sageset number is set in the registry to the specific folders that are going to be cleaned up, then it can be called to clean up those selected folders by the sageset number, which is essentially just a preset.

I discovered that you can manually set the sageset number for clearing thumbnail cache in the registry within the program so the user doesn't have to manually specify that the thumbnail cache needs to be cleared in a separate dialog, which allows it to work in a hopefully seamless way, by running a command and waiting for it to complete before doing things like the restore.

All in all very interesting finds. Somewhat unrelated but may I ask, how are you this knowledgeable about the underlying stuff about how all this stuff works? Or is just educated guess after educated guess? I'm asking because this whole topic is something I've looked up on and off throughout the years and the data available out there, at least AFAIK, is pretty much nonexistent.

Honestly, I knew nothing about how Windows thumbnail cache or even coding in general worked less than a year ago. I did watch some coding tutorials to get started and help fix the old repository, but especially with this V2 repository, I've been relying mostly on GPT-4 to learn the specifics of how coding works along with it giving good results on how the Windows thumbnail cache works. It's helped a lot for making all these new features possible and allowed me to help build this version to be significantly more advanced than the previous version.

[Mfarooq360]

Beta 3 is out now, it should fix most of the issues, though I'm still looking into the implementation of the windows disk cleaner method of deleting the thumbnail cache.

[VL4DST3R]

Tested v3 for a bit and I noticed that the issue with resetting cache is now virtually instant. Whenever I restore it (via force restore since it never works for me otherwise) when I open a folder with images inside (folder previews oddly enough are present/presumably loaded from cache) it immediately nukes the cache and resets it (immediately visible within winthumbs). I also get a Force restore failed. Retry as admin. even though I am doing just that and the file sizes seem to suggest the cacahe files copied over successfully.

Any idea if there is any error logging for when this happens to help in debugging? I checked under Event Viewer and there's no mention of anything when it happens. Any sort of validation for the cache files or in general info about how they work?

I also noticed that when you kill the explorer, windows has the tendency to immediately reopen it (visible via the taskbar reappearing 1/3 into the restore process). I feel if you were to repeatedly kill it to prevent it from opening back and locking the files until the restore process is completed would maybe help this issue.

[Mfarooq360]

Tested v3 for a bit and I noticed that the issue with resetting cache is now virtually instant. Whenever I restore it (via force restore since it never works for me otherwise) when I open a folder with images inside (folder previews oddly enough are present/presumably loaded from cache) it immediately nukes the cache and resets it (immediately visible within winthumbs).

I've had a similar problem with restoring the thumbnail cache where if the thumbnail cache is corrupted, trying to restore it from a corrupted backup will cause Windows to delete it again anyways and refuses to use it, forcing me to have to preload the thumbnail cache again. I first encountered this when trying to transplant only the dbs that were larger than the current dbs which caused Windows to refuse to use the thumbnail cache that I transplanted.

I also get a Force restore failed. Retry as admin. even though I am doing just that and the file sizes seem to suggest the cacahe files copied over successfully.

This might be a bug with the program as it may give that message if the explorer cache size and backup cache size aren't the exact same size after the restore, so perhaps it may display if the explorer cache size increases right away. I'll see if making it so that if the explorer cache size is greater than or equal to the backup cache size will fix it.

Any idea if there is any error logging for when this happens to help in debugging? I checked under Event Viewer and there's no mention of anything when it happens. Any sort of validation for the cache files or in general info about how they work?

It doesn't seem to appear under Event Viewer when Windows decides to reject the thumbnail cache, so it seems that the only way to find out if the cache is rejected due to something like corruption or Windows' pickiness would be to watch the cache size change, whether it goes up as normal or resets to zero.

I also noticed that when you kill the explorer, windows has the tendency to immediately reopen it (visible via the taskbar reappearing 1/3 into the restore process). I feel if you were to repeatedly kill it to prevent it from opening back and locking the files until the restore process is completed would maybe help this issue.

Sounds like a good idea, I'll make it kill explorer before copying every thumbnail db file and hopefully it shouldn't cause any other issues with Windows.

[VL4DST3R]

[...] to find out if the cache is rejected due to something like corruption or Windows' pickiness [...]

I'm guessing there's no chance to know exactly how the db files are structured inside? No documentation or anything that would hopefully allow greater control over the content of the files?

Also then it is safe to assume my cached dbs are toast and I should just go ahead and rebuild them?

This might be a bug with the program as it may give that message if the explorer cache size and backup cache size aren't the exact same size after the restore, so perhaps it may display if the explorer cache size increases right away.

Indeed, the resulting restore in my case is ~1mb larger, so I'm guessing that's why.

Sounds like a good idea, I'll make it kill explorer before copying every thumbnail db file and hopefully it shouldn't cause any other issues with Windows.

Yep, if you recall that's what i said my bat file was doing as well, it waited a few seconds and re-killed everything 2-3 times before going ahead with the restore, and i believe it worked more reliably.

[Mfarooq360]

I'm guessing there's no chance to know exactly how the db files are structured inside? No documentation or anything that would hopefully allow greater control over the content of the files?

I do know of this one application to view the thumbnail cache though I'm not sure if or how it can be used within my program.

Also then it is safe to assume my cached dbs are toast and I should just go ahead and rebuild them?

It may be the case, though perhaps you could use the application I linked to check the contents of the corrupted thumbnail cache if there's anything noteworthy on why exactly they're considered corrupted by Windows compared to a non-corrupt cache.

Yep, if you recall that's what i said my bat file was doing as well, it waited a few seconds and re-killed everything 2-3 times before going ahead with the restore, and i believe it worked more reliably.

I have my program set to kill explorer before each file now just in case it decides to reopen at all during the process, though I would like to know if there was a reason as to why task manager gets killed in your bat file as compared to just closing explorer.

[VL4DST3R]

I do know of this one application to view the thumbnail cache though I'm not sure if or how it can be used within my program.

Very interesting find! Will play with it a bit when I get the chance and report back if I find anything. As to incorporating it- I wasn't suggesting that, just wondering if there is a way to detect such error states in any other way (besides when they get nuked by windows) so that they could ideally be brought back to a valid state by fixing whatever caused the corruption.

[...] though I would like to know if there was a reason as to why task manager gets killed in your bat file as compared to just closing explorer.

Wait, it does? When have I said that? I double checked my old bat to see if indeed I had that in and sure enough I couldn't find anything about killing the task manager because as you pointed out, it shouldn't affect the process at all.

[Mfarooq360]

Very interesting find! Will play with it a bit when I get the chance and report back if I find anything. As to incorporating it- I wasn't suggesting that, just wondering if there is a way to detect such error states in any other way (besides when they get nuked by windows) so that they could ideally be brought back to a valid state by fixing whatever caused the corruption.

I was wondering if it would be possible to directly modify the databases as I've heard SQLite was used, though after some research, it doesn't seem to be very possible or practical for a few reasons.

Wait, it does? When have I said that? I double checked my old bat to see if indeed I had that in and sure enough I couldn't find anything about killing the task manager because as you pointed out, it shouldn't affect the process at all.

I went off of this reference in the old repository where the bat file appears to state taskkill /f /im taskmgr.exe Just wanted to make sure I wasn't leaving something important out that may have been causing problems or failures in my program when closing explorer.

[VL4DST3R]

Back with finds! Sure enough, the cache viewer found errors in my corrupted backups:

Which leads me to hope that there is indeed a way to sanitize corrupted databases and restore functionality with only losing a few entries!

I was wondering if it would be possible to directly modify the databases as I've heard SQLite was used, though after some research, it doesn't seem to be very possible or practical for a few reasons.

Oh shucks, and here I hoped given the other tool being able to even calculate the correct checksums....

I went off of https://github.com/Mfarooq360/WinThumbsPreloader/issues/1#issuecomment-1643645958 in the old repository where the bat file appears to state taskkill /f /im taskmgr.exe Just wanted to make sure I wasn't leaving something important out that may have been causing problems or failures in my program when closing explorer.

Ah drat, sorry, I think I was testing at the time my script and was constantly opening task manager to double-check processes running so ultimately i just added it there for whatever reason. It was an oversight on my part and not intended as part of the script functionality.

[VL4DST3R]

Another idea that came to mind about the corruption stuff: Is it possible when generating thumbnails to somehow saturate the database with requests and cause this to happen? So would it be worth to introduce some sort of limit or validation before submitting more thumbnails to the cache dbs to make sure that they can indeed "take" them?

[Mfarooq360]

After doing some digging/research and some trial and error, I've found out that the thumbnail cache databases aren't SQLite (as far as I can tell) and are instead a proprietary binary file format, though the file information including the data checksum was able to be extracted and utilized in some other programs (Thumbcache Viewer technically does but its written in C++ which is a lot harder to understand), which led me to find a C# implementation under a similar name that is also able to read the information, though it's going to take some time to understand how exactly it works and how to utilize it myself if possible.

While the C# implementation doesn't have the logic for calculating the CRC64 checksum to compare to the stored data checksum and it isn't well documented on how the checksum is actually calculated, the C++ implementation can do it, though it's going to take some learning and understanding on how it works. The lookup table for the checksum was also used from a thumbscache dll and implemented into the C++ implementation, and the calculator was able to be implemented in C#, though it's not clear what part of the data actually gets calculated as trying to calculate it from the image data outputs a different checksum in my testing.

This may take a while to ensure it would actually work or how to get it to work properly as I also believe there is a master cache database index that keeps track of the files so I'll have to see if it's still fully possible in the end. I'm not sure if deleting or modifying the checksum/database entries will cause issues with the index database not accepting the thumbs cache.

I don't want to get your hopes too high, but if I can get this implemented eventually, I'd like to have an option to shift click the backup button to repair the thumbnail cache backup or something similar, though perhaps I could also turn this into its own program that can repair thumbnail cache which can interact with mine if I can figure out how to replace the checksum in the file or just remove the faulty entries to allow the cache to be usable by Windows if it doesn't disturb the index cache.

Like I said though, it's not guaranteed that I will be able to get this working soon if at all, but I am looking into it and actually making some progress using some resources to learn how the thumbscache works such as this journal and this documentation.

[VL4DST3R]

Those are some awesome finds and yeah, not holding my breath, but it would basically mean low-level access to how this entire system works, which would be quite awesome!

On a somewhat unrelated note, is folder thumb generation still functional? Because I've ran a batch of thumbnail generation and I noticed again missing folder content previews.

[VL4DST3R]

Found another small issue with paths, namely - within the folder name e.g. S:\webui\outputs\txt2img-images causes the script to not work.

[VL4DST3R]

Found this autostart reg key under Computer\HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Run when looking for something unrelated:

Is it ok to be without quotes? Wouldn't this fail if it were to be in a path with spaces? I see all other programs that are opened at startup have the path in quotes.

[Mfarooq360]

Thanks for finding all of these issues, I've managed to fix them for the next release and am currently adding other features beforehand.

For the folder thumbnail generation, I had initially created that feature and deemed it working by testing the amount of items that would be generated as there would be more with the option enabled. With thumbscache viewer, I found out it wasn't actually working which is most likely due to how it was initially supposed to only generate folder thumbs if the files within contained the specified extensions which wasn't implemented properly. It should now work and I'm adding an option to choose between only loading folder thumbs if they contain files with the set extensions or to load them all which could potentially have instabilities.

[VL4DST3R]

Great stuff, do tell when this new version will be published so I can play with it further!

On a similar note, I've noticed certain videos are not getting thumbnails generated for them after a gen pass on the folder where they reside, but so far I've been unable to pinpoint what's the cause. In my case it happened with random video files I have in a "memes" folder so the origin and format are wildly different but also on very few recordings done with shadowplay (nvidia's screen recorder). Any insight on this issue?

[Mfarooq360]

Great stuff, do tell when this new version will be published so I can play with it further!

I may try to target this weekend if possible, though with the feature changes being added, it's not guaranteed. On a similar note, I've noticed certain videos are not getting thumbnails generated for them after a gen pass on the folder where they reside, but so far I've been unable to pinpoint what's the cause. In my case it happened with random video files I have in a "memes" folder so the origin and format are wildly different but also on very few recordings done with shadowplay (nvidia's screen recorder). Any insight on this issue?

I'm wondering if this is due to the sizes of thumbnails that get generated being too small for the sizes they are viewed in the folder, especially for videos when scaled to a larger view size in explorer. It might not be the case if it's just a small number of videos that are next to others that do work, but perhaps a sample of an affected video could help me diagnose this if possible.

I'm experimenting with adding the ability to customize the sizes of the thumbnails that get generated since I've seen that the program doesn't work for Windows' photos app which uses 768 size thumbnails for the medium view size and sometimes the large view size instead of 256 size thumbnails which tend to get used for the small view size in the photos app most of the time.

The program has been using a size of 128 for a currently unknown reason, though it generates 256 size thumbs anyways.

[VL4DST3R]

I'm wondering if this is due to the sizes of thumbnails that get generated being too small for the sizes they are viewed in the folder,

Could very well be as indeed, I have those folders set to the maximum zoom level

[...] perhaps a sample of an affected video could help me diagnose this if possible.

Next time it poops the bed I'll try to see which ones get left behind, since once viewed they quickly get generated properly by windows.

doesn't work for Windows' photos app which uses 768 size thumbnails for the medium view size and sometimes the large view size instead of 256 size thumbnails which tend to get used for the small view size in the photos app most of the time.

I had no idea the photos app was using the same cache to show its thumbnails. Does that mean if you explore the images via the photos app the resulted thumbs generated from it are being used by explorer as well?

[Mfarooq360]

I had no idea the photos app was using the same cache to show its thumbnails. Does that mean if you explore the images via the photos app the resulted thumbs generated from it are being used by explorer as well?

I found out that based on my testing, the photos app being set to the small image mode will use the same 256 size cache that explorer uses which means they're both able to use the thumbnails that each ones generate. When using the photos app on the medium or large setting though, it uses 768 size thumbnails which I haven't seen explorer use from my testing.

I've released beta 4 now which includes the option to change the requested thumbnail cache size which allows thumbnails for the photos app to be generated for the medium and large sizes now.

[VL4DST3R]

I've released beta 4 now which includes the option to change the requested thumbnail cache size which allows thumbnails for the photos app to be generated for the medium and large sizes now.

Nice, however I'm not using that -thing- so I won't really be testing the sizes above what explorer uses. Will report back if I find anyting noteworthy.

[VL4DST3R]

I've been noticing that as of late almost always when I try to restore from backup it just instantly fails and wipes everything once it's done :/

[Mfarooq360]

I've been noticing that as of late almost always when I try to restore from backup it just instantly fails and wipes everything once it's done :/

I'm starting to wonder if Windows starts to corrupt the thumbnail cache at high sizes which causes it to always end up resetting it. Is ThumbCache Viewer detecting mismatched data checksums from the backup cache? If so, I'll be looking further into how to fix it or if its some other corruption like a different checksum or hash that needs its own way of being fixed.

So far I've gotten the calculation of the data checksums to match the actual checksum for correct checksums in most cases, though it seems like the calculations won't be accurate if the cache decides to have a bunch of blank entries at the start for whatever reason, yet the C++ implementation of ThumbCache Viewer seems to do it fine which will mean this would become a lot more challenging to deal with having to figure out how it works.

[VL4DST3R]

I'm starting to wonder if Windows starts to corrupt the thumbnail cache at high sizes which causes it to always end up resetting it.

I would lean on this if I didn't also have a laptop I use from time to time, which seems to lose its thumbnail cache almost as often (thankfully it has very few things I need cached so it's just a mild annoyance) and it reaches only about 400mb of thumbs tops.

Re: Restore from backup: indeed now it seems to never work, fresh backup or not, it always instantly fails to restore even with explorer closed beforehand. I think this is a regression over the previous two versions.

On a sidenote, the tool doesn't seem to be able to generate thumbnails for .webm files.

So far I've gotten the calculation of the data checksums to match the actual checksum for correct checksums in most cases, though it seems like the calculations won't be accurate if the cache decides to have a bunch of blank entries at the start for whatever reason

If this is the case, then could you maybe add a "validade checksums" button for the backed up files to see if at least you can catch a corrupted backup? Maybe we can pinpoint exactly what invalidates it.