Open ctb opened 1 year ago
it looks like the Rust ZipStorage
implementation might be the cause of the steady increase in memory. I generated the below graphs with mprof by doing a prefetch()
over ~2000 sketches in a ZipFileLinearIndex
with something that had no matches. With the current default ZipStorage
read-only implementation, I get the first graph. When I force the use of _RwZipStorage
in sbt_storage.py
, I get the second graph.
A quick perusal of the Rust code in src/core/src/storage.rs
doesn't turn up any obvious caching or anything. @luizirber any ideas?
I note the complicated discussion of memory mapping in the rust ZipStorage
PR here. Is this behavior maybe related to that?
more - I did use guppy3 (on the Python side) and the total heap usage in Python, at least, does not change.
I dug a bit more this morning, and saw this in the mprof docs -
memory_profiler supports different memory tracking backends including: ‘psutil’, ‘psutil_pss’, ‘psutil_uss’, ‘posix’, ‘tracemalloc’. If no specific backend is specified the default is to use “psutil” which measures RSS aka “Resident Set Size”. In some cases (particularly when tracking child processes) RSS may overestimate memory usage
which linked to this section in the psutil docs:
uss (Linux, macOS, Windows): aka “Unique Set Size”, this is the memory which is unique to a process and which would be freed if the process was terminated right now.
I ran my benchmarking script like so:
mprof run --backend psutil_uss python myprefetch.py
and got the same chart:
here's the code, which only takes 20 seconds or so to run:
# run in ~ctbrown/scratch/2022-pymagsearch
import sourmash
from sourmash import sourmash_args
class empty:
pass
obj = empty()
obj.picklist = 'mf.csv::manifest'
picklist = sourmash_args.load_picklist(obj)
sigfile = 'gtdb-rs207-k31/af7ac805.k=31.scaled=1000.DNA.dup=0.63.sig'
ss = list(sourmash.load_file_as_signatures(sigfile)).pop()
ss = ss.to_mutable()
ss.minhash = ss.minhash.flatten()
dbfile = '/group/ctbrowngrp/sourmash-db/gtdb-rs207/gtdb-rs207.genomic-reps.dna.k31.zip'
for i in range(2):
db = sourmash.load_file_as_index(dbfile)
db = sourmash_args.apply_picklist_and_pattern(db, picklist, None)
print("iteration:", i)
print("matches: ", list(db.prefetch(ss, 50000)))
this is a good discussion/approach: using a small container to verify small memory usage 😆 https://github.com/sourmash-bio/sourmash/pull/1909#issuecomment-1092307414
shows three phases: