The speed-up is especially noticeable on repeated runs because walking over the directories now uses a more effective custom implementation using os.scandir instead of os.walk.
The speedup is achieved by:
truly skipping excluded directories
at least halving the number of os.stat requests
avoiding storing more than batch size file paths in memory at once (os.walk stores all files from the current subdir in memory)
Closes #75
The speed-up is especially noticeable on repeated runs because walking over the directories now uses a more effective custom implementation using
os.scandirinstead ofos.walk.The speedup is achieved by:
os.statrequestsos.walkstores all files from the current subdir in memory)