Open taylorreiter opened 2 years ago
Put files here, https://github.com/ctb/2022-sourmash-slow-sig-subtract, and fixed them a bit so they'd run 😜.
I can't replicate with downsampled sigs (to scaled=1000, k=31 only) - I see
% /usr/bin/time snakemake -j 1 outputs/sourmash_sketch_subtract/C_2_1_k31.sig
...
133.06user 4.60system 2:24.23elapsed 95%CPU
for the Python API version, and
% /usr/bin/time snakemake -j 1 outputs/sourmash_sketch_subtract/SRR6032600-minus-SRR6032602-k31.sig
...
131.46user 5.15system 2:26.10elapsed 93%CPU
so, basically identical.
I will note that in the Python version, you don't need to reload mtx_sigobj
as abund_sig
. This was true even before we implemented FrozenMinHash
in https://github.com/sourmash-bio/sourmash/pull/1508 and FrozenSourmashSignature
in https://github.com/sourmash-bio/sourmash/pull/1610, but now any attempt to in-place modify a signature or minhash loaded from a file will barf on you! So this is a potential speed up.
I will now re-run with the scaled=200 sketches and see what happens.
tl;dr no substantial difference in time or maxresident, but a massive difference in inputs
(!?) - which is file system inputs. WTH?
from man time
on farm - "Number of file system inputs by the process."
and a lot more page faults.
huh.
% /usr/bin/time snakemake -j 1 outputs/sourmash_sketch_subtract/C_2_1_k31.sig
...
5973.95user 50.38system 1:40:46elapsed 99%CPU (0avgtext+0avgdata 10888868maxresident)k
8277424inputs+235944outputs (436major+23010636minor)pagefaults 0swaps
% /usr/bin/time snakemake -j 1 outputs/sourmash_sketch_subtract/SRR6032600-minus-SRR6032602-k31.sig
...
5871.86user 49.32system 1:38:48elapsed 99%CPU (0avgtext+0avgdata 10843796maxresident)k
1280inputs+235936outputs (4major+23577833minor)pagefaults 0swaps
I can replicate the inputs differences with a much smaller query - makes me suspect that time
is only counting inputs in the primary process, i.e. snakemake, so when Python runs within snakemake, we're seeing the input count - but not in the shell situation.
I'll think about it some more. But for now I cannot reproduce the performance delta itself.
I recently wrote a snakefile to subtract a one signature from another. I noticed that my implementation of sourmash sig subtract, which I wrote using the python API but basically copied from sourmash src, was super slow. Signature loading was slow, but subtracting was the slowest step (discovered with liberal print statements). I then switched it to running in sourmash sig subtract using the cli, but wrapped in
shell()
, so still executed within python. That was also slow in the same ways as above. Lastly, I ran it just on the command line, and then it was fast again!I looked at memory while these were running at it consistently stayed around 3GB (on a 64 gb machine). I also tried giving each rule 64GB of memory in case snakemake was silently limiting resources, and there was no change in perceived runtime.
I realize that the two approaches that use just python are written so that ALL signatures in the metatdata file would be run before the rule finished, while the one that uses the CLI operates on one sample (two sigs) at a time. However, I was monitoring this just running on the one sample provided here, and the first two implementations below were WAY slower than just using the cli. (this isn't really important for the reprex, as I only included on sample in the metadata table).
I have a working solution for what I want to do, but I wanted to record what I felt like was weird behavior that I couldn't figure out an explanation for.
I'm including a snakefile here that reproduces the behavior I observed. The metadata file (attached) I included only runs the code for one sample (two SRA run accessions) so it will hopefully be ~quick. You'll need sourmash, pandas, sra tools, and snakemake in your environment for the snakefiles to run.
metadata.txt
Conda environment file: