2024/cpp-query-file-attribs-faster/

[utterances-bot]

Boost File Scanning Speed: Query File Attributes on Windows 50x Faster - C++ Stories

Imagine you’re developing a tool that needs to scan for file changes across thousands of project files. Retrieving file attributes efficiently becomes critical for such scenarios. In this article, I’ll demonstrate a technique to get file attributes that can achieve a surprising speedup of over 50+ times compared to standard Windows methods.

https://www.cppstories.com/2024/cpp-query-file-attribs-faster/

[sehe]

Notably, std::filesystem performed on par with FindFirstFileEx

Seems like that should have read “on par with GetFileAttributesEx“

[fenbf]

thanks @sehe! should be fixed now.

[SimonSchroeder]

Just these days I have stumbled on the low level file io library (llfio on GitHub) which is on its way for C++ standardization. IIRC they also have handling of file attributes. It would be interesting how this library would stack up against the other methods.

[paulharris]

You should also benchmark the performance when hitting an SMB share / mapped drive.

[gmit3]

There is also a FIND_FIRST_EX_LARGE_FETCH flag for FindFirstFileEx that makes the difference...

[fenbf]

Thanks for suggestions, looks like I have to update the benchmark with a few other techniques:

• FIND_FIRST_EX_LARGE_FETCH
• NtQueryDirectoryFileEx
• GetFileInformationByHandleEx

[paulharris]

Just these days I have stumbled on the low level file io library (llfio on GitHub) which is on its way for C++ standardization. IIRC they also have handling of file attributes. It would be interesting how this library would stack up against the other methods.

This looks very interesting! Thanks