@ccbaumler and I have developed a benchmarking workflow to analyze various different metrics of sourmash commands. This will help us identify parts of the program which can be further optimized.
The input sig files vary in sample location, instrumentation, base quantity, and file size. They are listed below:
Sample
Sampling Location
Instrument
Base Quantity (gigabases)
Files size (gigabytes)
SRR1976948
USA: Alaska, North Slope, Schrader Bluff formation
Illumina MiSeq
8.65
4.96
SRR1977249
USA: Alaska, North Slope, Schrader Bluff formation
Flamegraphs constructed using python profiler (py-spy). An example of a flamegraph for the gather command:
Furthermore, the workflow measures Computational metrics by snakemake's benchmark directive. The benchmark outputs a tsv file with the following metrics ref :
colname
type (unit)
description
s
float (seconds)
Running time in seconds
h :m :s
string (-)
Running time in hour, minutes, seconds format
max_rss
float (MB)
Maximum "Resident Set Size”, this is the non-swapped physical memory a process has used.
max_vms
float (MB)
Maximum “Virtual Memory Size”, this is the total amount of virtual memory used by the process
max_uss
float (MB)
“Unique Set Size”, this is the memory which is unique to a process and which would be freed if the process was terminated right now.
max_pss
float (MB)
“Proportional Set Size”, is the amount of memory shared with other processes, accounted in a way that the amount is divided evenly between the processes that share it (Linux only)
io_in
float (-)
the number of read operations performed (cumulative).
io_out
float (-)
the number of write operations performed (cumulative).
mean_load
float (-)
CPU usage over time, divided by the total running time (first row)
cpu_time
float(-)
CPU time summed for user and system
We can compare the metrics among different samples with line graphs (example below) to check for inconsistencies.
Possible additions to the benchmarking work flow:
Running the command on each sample multiple times of an arbitrary standard (say 3 times) and aggregating these results. This will prevent one time events and outliers from influencing our analysis.
@ccbaumler and I have developed a benchmarking workflow to analyze various different metrics of sourmash commands. This will help us identify parts of the program which can be further optimized.
(Metadata found here and Project information here)
Metrics are measured two different ways:
Flamegraphs constructed using python profiler (py-spy). An example of a flamegraph for the gather command:
Furthermore, the workflow measures Computational metrics by snakemake's benchmark directive. The benchmark outputs a tsv file with the following metrics ref :
We can compare the metrics among different samples with line graphs (example below) to check for inconsistencies.
Possible additions to the benchmarking work flow: