search

ARM-software / CMSIS_5

CMSIS Version 5 Development Repository
http://arm-software.github.io/CMSIS_5/index.html
Apache License 2.0
1.33k stars 1.08k forks source link

SNR below threshold for TransformQ15 test #789

Closed stephanosio closed 4 years ago

stephanosio commented 4 years ago

Summary

The SNR value reported by the TransformQ15 IFFT tests are below the specified threshold (i.e. 30).

https://github.com/ARM-software/CMSIS_5/blob/73ed0410119f500ec5025c79c04f46379c46620e/CMSIS/DSP/Testing/Source/Tests/TransformQ15.cpp#L8

Notes

Log

Ignore test PASS status, as the SNR assertion was deliberately disabled

QEMU Cortex-M33 Target

[0/1] To exit from QEMU enter: 'CTRL+a, x'[QEMU] CPU: cortex-m33
qemu-system-arm: warning: nic lan9118.0 has no peer
*** Booting Zephyr OS build zephyr-v2.1.0-594-g17bab9c7b999  ***
Running test suite transform_q15
===================================================================
starting test - test_arm_cfft_q15_noisy_16
SNR = 68.607925
PASS - test_arm_cfft_q15_noisy_16
===================================================================
starting test - test_arm_cifft_q15_noisy_16
SNR = 56.335636
PASS - test_arm_cifft_q15_noisy_16
===================================================================
starting test - test_arm_cfft_q15_noisy_32
SNR = 59.494591
PASS - test_arm_cfft_q15_noisy_32
===================================================================
starting test - test_arm_cifft_q15_noisy_32
SNR = 44.092136
PASS - test_arm_cifft_q15_noisy_32
===================================================================
starting test - test_arm_cfft_q15_noisy_64
SNR = 61.709412
PASS - test_arm_cfft_q15_noisy_64
===================================================================
starting test - test_arm_cifft_q15_noisy_64
SNR = 45.335495
PASS - test_arm_cifft_q15_noisy_64
===================================================================
starting test - test_arm_cfft_q15_noisy_128
SNR = 52.768990
PASS - test_arm_cfft_q15_noisy_128
===================================================================
starting test - test_arm_cifft_q15_noisy_128
SNR = 32.719532
PASS - test_arm_cifft_q15_noisy_128
===================================================================
starting test - test_arm_cfft_q15_noisy_256
SNR = 55.635139
PASS - test_arm_cfft_q15_noisy_256
===================================================================
starting test - test_arm_cifft_q15_noisy_256
SNR = 32.935146
PASS - test_arm_cifft_q15_noisy_256
===================================================================
starting test - test_arm_cfft_q15_noisy_512
SNR = 46.698872
PASS - test_arm_cfft_q15_noisy_512
===================================================================
starting test - test_arm_cifft_q15_noisy_512
SNR = 20.629599
PASS - test_arm_cifft_q15_noisy_512
===================================================================
starting test - test_arm_cfft_q15_noisy_1024
SNR = 49.472736
PASS - test_arm_cfft_q15_noisy_1024
===================================================================
starting test - test_arm_cifft_q15_noisy_1024
SNR = 20.861605
PASS - test_arm_cifft_q15_noisy_1024
===================================================================
starting test - test_arm_cfft_q15_noisy_2048
SNR = 40.734348
PASS - test_arm_cfft_q15_noisy_2048
===================================================================
starting test - test_arm_cifft_q15_noisy_2048
SNR = 9.383556
PASS - test_arm_cifft_q15_noisy_2048
===================================================================
starting test - test_arm_cfft_q15_noisy_4096
SNR = 43.529823
PASS - test_arm_cfft_q15_noisy_4096
===================================================================
starting test - test_arm_cifft_q15_noisy_4096
SNR = 9.492506
PASS - test_arm_cifft_q15_noisy_4096
===================================================================
starting test - test_arm_cfft_q15_step_16
SNR = 72.957825
PASS - test_arm_cfft_q15_step_16
===================================================================
starting test - test_arm_cifft_q15_step_16
SNR = 62.298439
PASS - test_arm_cifft_q15_step_16
===================================================================
starting test - test_arm_cfft_q15_step_32
SNR = 66.048439
PASS - test_arm_cfft_q15_step_32
===================================================================
starting test - test_arm_cifft_q15_step_32
SNR = 49.075706
PASS - test_arm_cifft_q15_step_32
===================================================================
starting test - test_arm_cfft_q15_step_64
SNR = 68.320625
PASS - test_arm_cfft_q15_step_64
===================================================================
starting test - test_arm_cifft_q15_step_64
SNR = 49.508591
PASS - test_arm_cifft_q15_step_64
===================================================================
starting test - test_arm_cfft_q15_step_128
SNR = 58.543125
PASS - test_arm_cfft_q15_step_128
===================================================================
starting test - test_arm_cifft_q15_step_128
SNR = 36.639244
PASS - test_arm_cifft_q15_step_128
===================================================================
starting test - test_arm_cfft_q15_step_256
SNR = 61.092979
PASS - test_arm_cfft_q15_step_256
===================================================================
starting test - test_arm_cifft_q15_step_256
SNR = 36.763176
PASS - test_arm_cifft_q15_step_256
===================================================================
starting test - test_arm_cfft_q15_step_512
SNR = 52.817379
PASS - test_arm_cfft_q15_step_512
===================================================================
starting test - test_arm_cifft_q15_step_512
SNR = 25.217543
PASS - test_arm_cifft_q15_step_512
===================================================================
starting test - test_arm_cfft_q15_step_1024
SNR = 55.827888
PASS - test_arm_cfft_q15_step_1024
===================================================================
starting test - test_arm_cifft_q15_step_1024
SNR = 24.938820
PASS - test_arm_cifft_q15_step_1024
===================================================================
starting test - test_arm_cfft_q15_step_2048
SNR = 45.655018
PASS - test_arm_cfft_q15_step_2048
===================================================================
starting test - test_arm_cifft_q15_step_2048
SNR = 12.841646
PASS - test_arm_cifft_q15_step_2048
===================================================================
starting test - test_arm_cfft_q15_step_4096
SNR = 48.684822
PASS - test_arm_cfft_q15_step_4096
===================================================================
starting test - test_arm_cifft_q15_step_4096
SNR = 12.839550
PASS - test_arm_cifft_q15_step_4096
===================================================================
Test suite transform_q15 succeeded
===================================================================
PROJECT EXECUTION SUCCESSFUL
QEMU: Terminated

Physical Cortex-M7 Target (ATSAME70)

*** Booting Zephyr OS build zephyr-v2.1.0-594-g17bab9c7b999  ***
Running test suite transform_q15
===================================================================
starting test - test_arm_cfft_q15_noisy_16
SNR = 69.376678
PASS - test_arm_cfft_q15_noisy_16
===================================================================
starting test - test_arm_cifft_q15_noisy_16
SNR = 58.267242
PASS - test_arm_cifft_q15_noisy_16
===================================================================
starting test - test_arm_cfft_q15_noisy_32
SNR = 60.484566
PASS - test_arm_cfft_q15_noisy_32
===================================================================
starting test - test_arm_cifft_q15_noisy_32
SNR = 46.831203
PASS - test_arm_cifft_q15_noisy_32
===================================================================
starting test - test_arm_cfft_q15_noisy_64
SNR = 63.651337
PASS - test_arm_cfft_q15_noisy_64
===================================================================
starting test - test_arm_cifft_q15_noisy_64
SNR = 46.823196
PASS - test_arm_cifft_q15_noisy_64
===================================================================
starting test - test_arm_cfft_q15_noisy_128
SNR = 54.581600
PASS - test_arm_cfft_q15_noisy_128
===================================================================
starting test - test_arm_cifft_q15_noisy_128
SNR = 34.376881
PASS - test_arm_cifft_q15_noisy_128
===================================================================
starting test - test_arm_cfft_q15_noisy_256
SNR = 56.937832
PASS - test_arm_cfft_q15_noisy_256
===================================================================
starting test - test_arm_cifft_q15_noisy_256
SNR = 34.858944
PASS - test_arm_cifft_q15_noisy_256
===================================================================
starting test - test_arm_cfft_q15_noisy_512
SNR = 48.257835
PASS - test_arm_cfft_q15_noisy_512
===================================================================
starting test - test_arm_cifft_q15_noisy_512
SNR = 22.536110
PASS - test_arm_cifft_q15_noisy_512
===================================================================
starting test - test_arm_cfft_q15_noisy_1024
SNR = 51.076500
PASS - test_arm_cfft_q15_noisy_1024
===================================================================
starting test - test_arm_cifft_q15_noisy_1024
SNR = 22.839441
PASS - test_arm_cifft_q15_noisy_1024
===================================================================
starting test - test_arm_cfft_q15_noisy_2048
SNR = 42.368797
PASS - test_arm_cfft_q15_noisy_2048
===================================================================
starting test - test_arm_cifft_q15_noisy_2048
SNR = 11.142725
PASS - test_arm_cifft_q15_noisy_2048
===================================================================
starting test - test_arm_cfft_q15_noisy_4096
SNR = 44.986156
PASS - test_arm_cfft_q15_noisy_4096
===================================================================
starting test - test_arm_cifft_q15_noisy_4096
SNR = 11.373922
PASS - test_arm_cifft_q15_noisy_4096
===================================================================
starting test - test_arm_cfft_q15_step_16
SNR = 72.957809
PASS - test_arm_cfft_q15_step_16
===================================================================
starting test - test_arm_cifft_q15_step_16
SNR = 62.298439
PASS - test_arm_cifft_q15_step_16
===================================================================
starting test - test_arm_cfft_q15_step_32
SNR = 65.309509
PASS - test_arm_cfft_q15_step_32
===================================================================
starting test - test_arm_cifft_q15_step_32
SNR = 48.974701
PASS - test_arm_cifft_q15_step_32
===================================================================
starting test - test_arm_cfft_q15_step_64
SNR = 69.947800
PASS - test_arm_cfft_q15_step_64
===================================================================
starting test - test_arm_cifft_q15_step_64
SNR = 49.624405
PASS - test_arm_cifft_q15_step_64
===================================================================
starting test - test_arm_cfft_q15_step_128
SNR = 59.154884
PASS - test_arm_cfft_q15_step_128
===================================================================
starting test - test_arm_cifft_q15_step_128
SNR = 38.385559
PASS - test_arm_cifft_q15_step_128
===================================================================
starting test - test_arm_cfft_q15_step_256
SNR = 62.265705
PASS - test_arm_cfft_q15_step_256
===================================================================
starting test - test_arm_cifft_q15_step_256
SNR = 38.245472
PASS - test_arm_cifft_q15_step_256
===================================================================
starting test - test_arm_cfft_q15_step_512
SNR = 54.528000
PASS - test_arm_cfft_q15_step_512
===================================================================
starting test - test_arm_cifft_q15_step_512
SNR = 26.423458
PASS - test_arm_cifft_q15_step_512
===================================================================
starting test - test_arm_cfft_q15_step_1024
SNR = 57.416042
PASS - test_arm_cfft_q15_step_1024
===================================================================
starting test - test_arm_cifft_q15_step_1024
SNR = 25.809500
PASS - test_arm_cifft_q15_step_1024
===================================================================
starting test - test_arm_cfft_q15_step_2048
SNR = 47.052162
PASS - test_arm_cfft_q15_step_2048
===================================================================
starting test - test_arm_cifft_q15_step_2048
SNR = 13.337952
PASS - test_arm_cifft_q15_step_2048
===================================================================
starting test - test_arm_cfft_q15_step_4096
SNR = 50.012596
PASS - test_arm_cfft_q15_step_4096
===================================================================
starting test - test_arm_cifft_q15_step_4096
SNR = 13.383279
PASS - test_arm_cifft_q15_step_4096
===================================================================
Test suite transform_q15 succeeded
===================================================================
PROJECT EXECUTION SUCCESSFUL
christophe0606 commented 4 years ago

In the file Testing/desc.txt describing the tests, some tests are disabled:

disabled {cifft_noisy_512_q15:test_cifft_q15}
disabled {cifft_noisy_1024_q15:test_cifft_q15}
disabled {cifft_noisy_2048_q15:test_cifft_q15}
disabled {cifft_noisy_4096_q15:test_cifft_q15}
disabled {cifft_step_512_q15:test_cifft_q15}
disabled {cifft_step_1024_q15:test_cifft_q15}
disabled {cifft_step_2048_q15:test_cifft_q15}
disabled {cifft_step_4096_q15:test_cifft_q15}

I think this is corresponding to the failing tests you report above. So, it is normal.

The Q15 ifft is not accurate enough and will fail on big input patterns. Until the implementation is reworked, those tests will fail and thus are disabled.

stephanosio commented 4 years ago

In the file Testing/desc.txt describing the tests, some tests are disabled:

Ok, that makes a lot of sense. Thanks for clarifying.