WinThumbsPreloader seems to be really slow

[Snickbrack]

I am using this tool to preload dirs with image count above 100k and it seems that there is a really small bottleneck. Even the Windows Process for loading the Thumbs seems to be faster.

(Images have around 1-5 MB in size)

What could be the cause?

Windows 7 i7-4790 16 GB RAM 500 GB HDD

[lux-perpetua]

I'm pretty sure the reason is that the program is single-threaded and it's the main bottleneck for me. My solution was to split a workload into a number of folders and running a separate instance of the program over each of them(preferably number of folders equals the number of threads or cores). In your system, there will be also a bottleneck from a hard drive, and there is little you could do about it. SSD's are just much more suitable for the job.

[SMcCandlish]

Multi-threading would be cool, but I think speed mostly has to do with system capabilities (innate, or under current conditions, e.g. CPU load). I have a honkin' fast machine (3950x and loads of RAM), and I just ran this on a folder of over 2000 videos, on a hard drive not an SSD (though the OS is on an SSD). WTP churned through them in under two minutes. It would have taken much longer to manually page through the thumbnails-view list in that folder and let Windows do it the usual way.

I just ran it again on a whole drive worth of folders, with 33,000+ files (only some of which are media that will generate thumbnails), also on a hard drive, and the whole process took under 10 minutes, while I was doing stuff in other windows. Did this again on a folder of 26,000+ documents and another of 2,500+ photos simultaneously (two concurrent WTP jobs) on my boot SSD, and it was all done in under three minutes.

WTP does seem to access every file, as the 33,000-file process triggered Windows Defender several times with regard to RAR and EXE files it thought were suspicious. So WTP could potentially be more efficient by not looking at files that don't match particular extensions (and maybe have a way to control this, to account for additional media types down the road). Or that could at least be an option; some might actually prefer that it go through files and try to figure out of they're thumbnailable, since finding that a file that is misnamed "foo" or "bar.tmp" is actually an image or video might be helpful in some circumstances.

I don't know if it does any "dig around inside ZIP and other archives and try to find something to make a thumbnail for them" stuff. I would hope not, at least not as the default behavior, since it's annoying to have an archive icon replaced by a thumbnail as if it's not an archive. Presently I'm not seeing any evidence it is doing anything like this, so good.

[inthebrilliantblue]

@SMcCandlish Problem with excluding file types is that every file type does actually have a thumbnail that Windows can generate. Depending on the type it just shows the basic icon for that extension, or calls a generator when it has data it can make a thumbnail from. Pretty much any zip archive can generate an actual thumbnail if the archive contains an image, and you have a program installed that Windows can call to generate said thumbnail.

The actual slowdown of WTP is actually just how fast a HDD or SSD can service the IO required to read data. Lots of small files will always take a longer time to generate because opening / closing file handles takes longer than it does to read the data and do something with it. Thats why transferring 1 10GiB file will always be faster than transferring 10000 files that equal 10GiB. There is not much that can be done about that other than only use SSDs.

Even the multi-threaded patch I have on my repo only shaves a few minutes off of generation when working on an array of HDDs. It may be worth it to be able to exclude certain file types if you really mix data a lot, but unless its just tons and TONS of images mixed with other file types Im not sure that would be faster than just using Windows file explorer normally.