search

woubuc / sweep

Reduce the disk usage of your projects by removing dependencies & builds
https://sweep.woubuc.be
MIT License
179 stars 18 forks source link

Using a lot of memory #35

Open ghost opened 3 years ago

ghost commented 3 years ago

SPECS: OS: Arch Linux - Linux 5.10.4 (Linux 510) Sweep version: 1.0.3 Mem: 31.3GB

When running sweep against a big folder (my archive folder) it will eventually run out of memory. In my case it takes about 2 minutes to climb from 5% to 100% usage. The issue occurs during the 'find project' stage. I checked the configuration section of the documentation. But the configuration does not seem to be able to address my problem? Any ideas?

woubuc commented 3 years ago

I haven't done any specific memory profiling on this application, I probably should do this at some point. It does keep a bit of stuff in memory (a queue of all directories it hasn't scanned yet, and a list of all discovered projects) but unless your directory is really just extremely large with a ton of subdirectories it seems strange that it would run out of memory (and even then).

I'm not sure how sweep handles symlinks but it's probably worth checking if you have any recursive symlinks or other filesystem oddities in your directory tree, that would cause it to keep scanning?