source-separation / tutorial

Tutorial covering Open Source tools for Source Separation.
https://source-separation.github.io/tutorial
Other
355 stars 39 forks source link

Typo in REPET page, leads to incorrect evaluations #13

Closed Atralb closed 3 years ago

Atralb commented 3 years ago

What Section?

https://source-separation.github.io/tutorial/first_steps/repetition.html#sdr-friends

What does the website say?

The background is given a second time instead of the foreground in the estimated sources list:

bss_eval = nussl.evaluation.BSSEvalScale(
    true_sources_list=[gt_bg, gt_fg],
    estimated_sources_list=[repet_bg, repet_bg]
)
repet_eval = bss_eval.evaluate()

# Inspect the evaluation
pprint(repet_eval)

What SHOULD it say?

...
estimated_sources_list=[repet_bg, repet_fg]
...

We can also see afterwards that this was actually the executed code because you get very bad evaluations for the foreground, and for the foreground only:

foreground SI-SDR: -16.82 dB
background SI-SDR: +7.29 dB

foreground SI-SAR: -8.34 dB
background SI-SAR: +7.79 dB

foreground SI-SIR: -16.11 dB
background SI-SIR: +16.88 dB

On my machine, with the estimated sources list correctly set, I get:

foreground SI-SDR: +1.95 dB
background SI-SDR: +7.29 dB

foreground SI-SAR: +3.85 dB
background SI-SAR: +7.79 dB

foreground SI-SIR: +6.66 dB
background SI-SIR: +16.88 dB
ethman commented 3 years ago

Wow! This is a very subtle typo, but dramatically changes the results. Thanks for catching it. I'll try to fix it soon!