search

sepinf-inc / IPED

IPED Digital Forensic Tool. It is an open source software that can be used to process and analyze digital evidence, often seized at crime scenes by law enforcement or in a corporate investigation by private examiners.
Other
945 stars 218 forks source link

Optimize memory usage by FQLite library #1364

Open lfcnassif opened 1 year ago

lfcnassif commented 1 year ago

Analyzing another heap dump sent by another user, FQLite library is using ~21GB of memory to undelete data from 7 msgstore.db files, here is an excerpt: image

I'm asking for the triggering files to the user. @fmpfeifer, if I succeed to get the samples, since you know a lot about FQLite code, could you take a look at this?

lfcnassif commented 1 year ago

@fmpfeifer I got the triggering files, may you have time to take a look at this?

fmpfeifer commented 1 year ago

Sure, please send me the files

lfcnassif commented 1 year ago

Thanks for the help!

lfcnassif commented 1 year ago

Sure, please send me the files

Just sent to you privately by Teams. Thanks again for helping.

lfcnassif commented 1 year ago

Hi @fmpfeifer, did you have a chance to take a look at this?

fmpfeifer commented 1 year ago

Hi @lfcnassif, not yet. I'm on vacation now. Will try to look at this when I return next week.

lfcnassif commented 1 year ago

Thank you for replying, don't worry, have a good vacation!

lfcnassif commented 1 year ago

Hi @fmpfeifer, did you have a chance and time to look at this? I just got another heap dump from an user that hit this 2 times in sequence.

lfcnassif commented 1 year ago

Since the deleted records recovery is still not implemented for the new WA message database, I guess the feature is being called without bringing any results, right? Since those new DBs are the more common artifact we receive, and since this is aborting the processing for large DBs, maybe this feature could be disabled by default. What do you think?

PS: It could continue to run for the Contact DB, which is much smaller and shouldn't trigger this.

lfcnassif commented 1 year ago

Sorry, I just looked at the code, the undelete feature is being called just for old WA DBs.