Benchmark figures

[mbhall88]

Create benchmark figures for time and memory usage.

How do we want to do time? CPU time, wall time? I guess this gets complicated by threads, although I am using 2 for each tool at the moment.

[mbhall88]

Okay, so this is what I have for the nanopore data. Anything we want changed?

Runtime

Max. RAM usage

[lachlancoin]

Nice visualisation

On Wed, 12 Oct 2022 at 16:30, Michael Hall @.***> wrote:

Okay, so this is what I have for the nanopore data. Anything we want changed? Runtime

[image: image] https://user-images.githubusercontent.com/20403931/195257678-cde00e5c-8d00-4b2b-b807-cd266b75e77a.png Max. RAM usage

[image: image] https://user-images.githubusercontent.com/20403931/195257695-4863394e-9a86-4c5c-9c9c-d1c1365a2854.png

— Reply to this email directly, view it on GitHub https://github.com/mbhall88/drprg/issues/15#issuecomment-1275609531, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA6TKZBCDNI43ONLU5FJM6TWCZEFXANCNFSM6AAAAAARB5M3YM . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[iqbal-lab]

Looks great. Nothing to change, though I'm wondering what the p values are testing

[lachlancoin]

Just testing whether the means of the distributions are equal. Pretty clear they are not from the plot but I guess it helps if you want to say "significantly different" . I guess you used an unpaired t-test, but actually they are paired data (same sample using different approaches) If anything that should make the pvalues even more significant.

On Wed, 12 Oct 2022 at 17:40, Zamin Iqbal @.***> wrote:

Looks great. Nothing to change, though I'm wondering what the p values are testing

— Reply to this email directly, view it on GitHub https://github.com/mbhall88/drprg/issues/15#issuecomment-1275667700, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA6TKZATS354REDKE4BQDTLWCZMOZANCNFSM6AAAAAARB5M3YM . You are receiving this because you commented.Message ID: @.***>

[mbhall88]

I used a wilcoxon signed-rank test. Basically the nonparametric version of a paired t-test (because I'm lazy and couldn't be bothered testing for normality 😅 )

Pretty clear they are not from the plot but I guess it helps if you want to say "significantly different"

It wasn't actually clear to me that the runtimes would be significantly different for drprg and mykrobe

[mbhall88]

Here are the Illumina figures. These aren't final as this is only a subset of 400 samples, but I can't imagine it will change dramatically.

Memory

Runtime

[mbhall88]

The latest figures after adding in the racon denovo from https://github.com/rmcolq/pandora/pull/299 can be seen in https://github.com/rmcolq/pandora/issues/303#issuecomment-1305098855