search

abantos / bolt

A task automation tool (similart to grunt) for Python
MIT License
15 stars 8 forks source link

Performance degraded on delete-files task as file tree grows #102

Closed CurroRodriguez closed 2 years ago

CurroRodriguez commented 5 years ago

The task was implemented using a FileFinder class that searches for all matching files. This class extends lists for each folder in the tree, which I believe it is what is causing the performance issues. You can see that once all the files are compiled, deleting them it is not too bad even with the added logging. There are several alternatives to improve how the task works; unfortunately, they don't all work on every version of Python. There has been improvements to the glob module, but they are only available in Python 3.5 and later. Another alternative might be using a combination of os.walk and the fnmatch module. The complication to come up with a version that works on all versions of Python is that the task also supports a recursive option which if set to false (the default), it only searches the specified folder.

CurroRodriguez commented 2 years ago

Updated in Monday

CurroRodriguez commented 2 years ago

This was reported by a project that had a mixed of Python and JavaScript files. The JS part used node_modules, which increased the number of files and folders exponentially. I haven't seen a problem with any Python projects, even large ones, so I will not spend time on this.