Improve scanning phase - Githubissues

markfasheh / duperemove

Tools for deduping file systems

GNU General Public License v2.0

816 stars 81 forks source link

Closed JackSlateur closed 8 years ago

JackSlateur commented 8 years ago

Hello Mark,

The major stuff in these commits are:

remove a unused index. Really reduce the db size (30% reduction, according to some tests)
create index after all the data is inserted: this lower both cpu & memory consumption
use in-memory database, so a single code path can be used. Plus, according to tests, it reduces by 70% the memory consumptions (really, from 1.6GB to 500MB!)

markfasheh commented 8 years ago

Awesome, thanks for these!

lpirl commented 8 years ago

(Without having tested it) Excellent, thanks – the in-memory thingy was on my mind for long as well.