search

xmos / fwk_voice

Voice Framework
Other
11 stars 19 forks source link

New vs. Old IC benchmarks for full pipeline 3610 (eg Amazon noise tests) #274

Closed andrewdewhurst closed 2 years ago

andrewdewhurst commented 2 years ago

Deliverables

xcore implementation tests

  1. Compare Noise test performance between Avona pipelines using new and original IC (Run full pipeline (keyword) tests in sw_avona Jenkins)
  2. Compare Noise test results from (1) above with 3610 (same tests are in lib_audio_pipelines)
  3. Ensure Py IC = latest IC (the Avona version) - Jenkins

From acoustic team

  1. Compare Py VAD-IC vs VNR-IC vs Ground-truth
shuchitak commented 2 years ago

Full pipeline test results. Comparison plots for Avona head of develop, Avona new IC and lib_audio_pipelines.

Alt-arch new_ic_alt_arch.csv new_ic_alt_arch

Prev-arch new_ic_prev_arch.csv new_ic_prev_arch

shuchitak commented 2 years ago

The best results are on Avona head of develop. The reason for that is increasing IC delay samples from 180 to 600. New IC has lower kwd scores than Avona head of develop but is better than lib_audio_pipelines. The IC samples delay in lib_audio_pipelines is set to 180 samples though. We don't have lib_audio_pipelines results for IC delay set to 600 but we expect it would show an increased kwd performance as well.

For alt-arch, new IC is in general, better than lib_audio_pipelines but worse than Avona head of develop.

For prev-arch, new IC shows lower performance than lib_audio_pipelines for most of the barge-in streams. Copying results where new IC is worse than lib_audio_pipelines for prev-arch. <html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40">

Input | develop_results_Avona_prev_arch_xcore_Amazon_WR_250k.en-US | new_ic_results_Avona_prev_arch_xcore_Amazon_WR_250k.en-US | results_pipelines_prev_arch_xcore_Amazon_WR_250k.en-US |   | diff new IC vs lib_audio_pipelines -- | -- | -- | -- | -- | -- InHouse_XVF3510v080_v1.2_20190423_Loc1_Clean_XMOS_DUT1_70dB_Take1.wav | 99 | 98 | 100 |   | -2 InHouse_XVF3510v080_v1.2_20190423_Loc1_Clean_XMOS_DUT1_80dB_Take1.wav | 98 | 92 | 95 |   | -3 InHouse_XVF3510v080_v1.2_20190423_Loc1_Clean_XMOS_DUT1_90dB_Take1.wav | 5 | 3 | 10 |   | -7 InHouse_XVF3510v080_v1.2_20190423_Loc1_Noise1_65dB_XMOS_DUT1_80dB_Take1.wav | 92 | 83 | 92 |   | -9 InHouse_XVF3510v080_v1.2_20190423_Loc2_Clean_XMOS_DUT1_80dB_Take1.wav | 92 | 87 | 92 |   | -5 InHouse_XVF3510v080_v1.2_20190423_Loc2_Clean_XMOS_DUT1_90dB_Take1.wav | 5 | 7 | 11 |   | -4 InHouse_XVF3510v080_v1.2_20190423_Loc2_Noise1_65dB__Take1.wav | 98 | 95 | 97 |   | -2 InHouse_XVF3510v080_v1.2_20190423_Loc2_Noise1_70dB__Take1.wav | 96 | 94 | 95 |   | -1 InHouse_XVF3510v080_v1.2_20190423_Loc3_Clean_XMOS_DUT1_70dB_Take1.wav | 100 | 99 | 100 |   | -1 InHouse_XVF3510v080_v1.2_20190423_Loc3_Clean_XMOS_DUT1_80dB_Take1.wav | 96 | 89 | 95 |   | -6 InHouse_XVF3510v080_v1.2_20190423_Loc3_Clean_XMOS_DUT1_90dB_Take1.wav | 6 | 9 | 14 |   | -5 InHouse_XVF3510v080_v1.2_20190423_Loc3_Noise1_70dB__Take1.wav | 99 | 98 | 99 |   | -1

shuchitak commented 2 years ago

The lower performance for barge-in needs investigation. We need to check the VNR performance with AEC output as it's input.

shuchitak commented 2 years ago

avona_develop_after_ic_delay_change.zip

avona_develop_before_ic_delay_change.zip

lib_audio_pipelines_results.zip