0.3: test suite is failing in 3 units

[kloczek]

cmake config:

-- Cache values
AVX:BOOL=ON
AVX2:BOOL=OFF
BUILD_TESTING:BOOL=ON
CMAKE_BUILD_TYPE:STRING=RelWithDebInfo
CMAKE_INSTALL_PREFIX:PATH=/usr
CMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.9
DISABLE_CPU_OPTIMIZATION:BOOL=TRUE
FETCHCONTENT_BASE_DIR:PATH=/home/tkloczko/rpmbuild/BUILD/LPCNet-0.3/x86_64-redhat-linux-gnu/_deps
FETCHCONTENT_FULLY_DISCONNECTED:BOOL=OFF
FETCHCONTENT_QUIET:BOOL=ON
FETCHCONTENT_SOURCE_DIR_LPCNET:PATH=
FETCHCONTENT_UPDATES_DISCONNECTED:BOOL=OFF
FETCHCONTENT_UPDATES_DISCONNECTED_LPCNET:BOOL=OFF
NEON:BOOL=OFF
SSE:BOOL=OFF
codec2_DIR:PATH=/usr/lib64/cmake/codec2

and test suite is failing with:

+ cd LPCNet-0.3
+ /usr/bin/ctest --test-dir x86_64-redhat-linux-gnu --output-on-failure --force-new-ctest-process -j48
Internal ctest changing into directory: /home/tkloczko/rpmbuild/BUILD/LPCNet-0.3/x86_64-redhat-linux-gnu
Test project /home/tkloczko/rpmbuild/BUILD/LPCNet-0.3/x86_64-redhat-linux-gnu
    Start 1: core_synthesis_default
    Start 2: core_synthesis_load_20h
    Start 3: core_synthesis_mag
    Start 4: nnet2f32
    Start 5: SIMD_functions
1/5 Test #1: core_synthesis_default ...........***Failed    0.01 sec
+ '[' '!' -z ']'
+ '[' '!' -z 1 ']'
+ ../build_linux/src/dump_data --test --c2pitch ../wav/birch.wav birch.f32
./test_core_nn.sh: line 28: ../build_linux/src/dump_data: No such file or directory
+ diff birch_targ.f32 birch.f32
diff: birch.f32: No such file or directory
+ echo 'ERROR in synth .f32 output! Exiting...'
ERROR in synth .f32 output! Exiting...
+ exit 1

2/5 Test #2: core_synthesis_load_20h ..........***Failed    0.01 sec
+ '[' '!' -z ']'
+ '[' '!' -z ']'
+ '[' '!' -z 1 ']'
+ ../build_linux/src/dump_data --test --c2pitch ../wav/birch.wav birch.f32
./test_core_nn.sh: line 44: ../build_linux/src/dump_data: No such file or directory
+ diff birch_targ.f32 birch.f32
diff: birch.f32: No such file or directory
+ echo 'ERROR in synth .f32 output! Exiting...'
ERROR in synth .f32 output! Exiting...
+ exit 1

3/5 Test #5: SIMD_functions ...................   Passed    0.00 sec
4/5 Test #3: core_synthesis_mag ...............***Failed    0.01 sec
+ '[' '!' -z ']'
+ '[' '!' -z ']'
+ '[' '!' -z ']'
+ '[' '!' -z 1 ']'
+ ../build_linux/src/dump_data --mag --test --c2pitch ../wav/c01_01.wav c01_01.f32
./test_core_nn.sh: line 60: ../build_linux/src/dump_data: No such file or directory
+ diff c01_01_mag.f32 c01_01.f32
diff: c01_01.f32: No such file or directory
+ echo 'ERROR in synth .f32 output! Exiting...'
ERROR in synth .f32 output! Exiting...
+ exit 1

5/5 Test #4: nnet2f32 .........................   Passed    0.06 sec

40% tests passed, 3 tests failed out of 5

Total Test time (real) =   0.06 sec

The following tests FAILED:
          1 - core_synthesis_default (Failed)
          2 - core_synthesis_load_20h (Failed)
          3 - core_synthesis_mag (Failed)
Errors while running CTest

[kloczek]

Hmm .. I don't see anywhere those files ajt I don't see as well what could generate that

[tkloczko@pers-jacek LPCNet-0.3]$ find . -name birch.f32
[tkloczko@pers-jacek LPCNet-0.3]$ grep -r birch.f32
src/plot_features.m:feat_lpcnet=load_f32("../birch.f32", nb_lpcnet_features);
unittest/test_core_nn.sh:    ../build_linux/src/dump_data --test --c2pitch ../wav/birch.wav birch.f32
unittest/test_core_nn.sh:    diff birch_targ.f32 birch.f32 || { echo "ERROR in synth .f32 output! Exiting..."; exit 1; }
unittest/test_core_nn.sh:    ../build_linux/src/test_lpcnet -l birch_states.f32 birch.f32 birch_out.raw
unittest/test_core_nn.sh:    ../build_linux/src/dump_data --test --c2pitch ../wav/birch.wav birch.f32
unittest/test_core_nn.sh:    diff birch_targ.f32 birch.f32 || { echo "ERROR in synth .f32 output! Exiting..."; exit 1; }
unittest/test_core_nn.sh:    ../build_linux/src/test_lpcnet -n lpcnet_20h.f32 -l birch_states.f32 birch.f32 birch_out.raw
x86_64-redhat-linux-gnu/Testing/Temporary/LastTest.log:+ ../build_linux/src/dump_data --test --c2pitch ../wav/birch.wav birch.f32
x86_64-redhat-linux-gnu/Testing/Temporary/LastTest.log:+ diff birch_targ.f32 birch.f32
x86_64-redhat-linux-gnu/Testing/Temporary/LastTest.log:diff: birch.f32: No such file or directory
x86_64-redhat-linux-gnu/Testing/Temporary/LastTest.log:+ ../build_linux/src/dump_data --test --c2pitch ../wav/birch.wav birch.f32
x86_64-redhat-linux-gnu/Testing/Temporary/LastTest.log:+ diff birch_targ.f32 birch.f32
x86_64-redhat-linux-gnu/Testing/Temporary/LastTest.log:diff: birch.f32: No such file or directory

[tmiw]

Can you confirm what branch and commit you're building off of? The latest master that I have (101d925fcf8929575ee53755db16dcaac8f08daf) only seems to have the following tests:

• feature_extraction
• nnet2f32
• SIMD_functions
• lpcnet_enc_dec

[kloczek]

As it is in the subject I'm building what exactly is on 0.3 git tag. OK will check 0.3 + master .. will back shortly 😋

[kloczek]

Checked that PR and none of those 5 commits are adding missing files

```console warning: Downloading https://github.com/drowe67/LPCNet//archive/v0.3/lpcnetfreedv-0.3.tar.gz to /home/tkloczko/rpmbuild/SOURCES/lpcnetfreedv-0.3.tar.gz warning: Downloading https://github.com/drowe67/LPCNet//commit/70eff791.patch#/lpcnetfreedv-Experiment-just-use-the-latest-vec-files-from-upstre.patch to /home/tkloczko/rpmbuild/SOURCES/lpcnetfreedv-Experiment-just-use-the-latest-vec-files-from-upstre.patch warning: Downloading https://github.com/drowe67/LPCNet//commit/e41a8bdf.patch#/lpcnetfreedv-Bring-in-tansig_table.h-too-to-resolve-ARM-compiler-.patch to /home/tkloczko/rpmbuild/SOURCES/lpcnetfreedv-Bring-in-tansig_table.h-too-to-resolve-ARM-compiler-.patch warning: Downloading https://github.com/drowe67/LPCNet//commit/13e102a6.patch#/lpcnetfreedv-Avoid-accidental-multiple-includes.patch to /home/tkloczko/rpmbuild/SOURCES/lpcnetfreedv-Avoid-accidental-multiple-includes.patch warning: Downloading https://github.com/drowe67/LPCNet//commit/a9c31c74.patch#/lpcnetfreedv-Minor-tweaks-to-get-test_vec.c-to-build-again.patch to /home/tkloczko/rpmbuild/SOURCES/lpcnetfreedv-Minor-tweaks-to-get-test_vec.c-to-build-again.patch warning: Downloading https://github.com/drowe67/LPCNet//commit/855b0a69.patch#/lpcnetfreedv-Suppress-aarch64-specific-code-for-non-aarch64-build.patch to /home/tkloczko/rpmbuild/SOURCES/lpcnetfreedv-Suppress-aarch64-specific-code-for-non-aarch64-build.patch Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.sEb6Dh + umask 022 + cd /home/tkloczko/rpmbuild/BUILD + cd /home/tkloczko/rpmbuild/BUILD + rm -rf LPCNet-0.3 + /usr/bin/gzip -dc /home/tkloczko/rpmbuild/SOURCES/lpcnetfreedv-0.3.tar.gz + /usr/bin/tar -xof - + STATUS=0 + '[' 0 -ne 0 ']' + cd LPCNet-0.3 + /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w . + /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/lpcnetfreedv-Experiment-just-use-the-latest-vec-files-from-upstre.patch + /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f + /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/lpcnetfreedv-Bring-in-tansig_table.h-too-to-resolve-ARM-compiler-.patch + /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f + /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/lpcnetfreedv-Avoid-accidental-multiple-includes.patch + /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f + /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/lpcnetfreedv-Minor-tweaks-to-get-test_vec.c-to-build-again.patch + /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f + /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/lpcnetfreedv-Suppress-aarch64-specific-code-for-non-aarch64-build.patch + /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f + RPM_EC=0 [..] + cd LPCNet-0.3 + /usr/bin/ctest --test-dir x86_64-redhat-linux-gnu --output-on-failure --force-new-ctest-process -j48 ' ' Internal ctest changing into directory: /home/tkloczko/rpmbuild/BUILD/LPCNet-0.3/x86_64-redhat-linux-gnu Test project /home/tkloczko/rpmbuild/BUILD/LPCNet-0.3/x86_64-redhat-linux-gnu Start 1: core_synthesis_default Start 2: core_synthesis_load_20h Start 3: core_synthesis_mag Start 4: nnet2f32 Start 5: SIMD_functions 1/5 Test #1: core_synthesis_default ...........***Failed 0.01 sec + '[' '!' -z ']' + '[' '!' -z 1 ']' + ../build_linux/src/dump_data --test --c2pitch ../wav/birch.wav birch.f32 ./test_core_nn.sh: line 28: ../build_linux/src/dump_data: No such file or directory + diff birch_targ.f32 birch.f32 diff: birch.f32: No such file or directory + echo 'ERROR in synth .f32 output! Exiting...' ERROR in synth .f32 output! Exiting... + exit 1 2/5 Test #2: core_synthesis_load_20h ..........***Failed 0.01 sec + '[' '!' -z ']' + '[' '!' -z ']' + '[' '!' -z 1 ']' + ../build_linux/src/dump_data --test --c2pitch ../wav/birch.wav birch.f32 ./test_core_nn.sh: line 44: ../build_linux/src/dump_data: No such file or directory + diff birch_targ.f32 birch.f32 diff: birch.f32: No such file or directory + echo 'ERROR in synth .f32 output! Exiting...' ERROR in synth .f32 output! Exiting... + exit 1 3/5 Test #5: SIMD_functions ................... Passed 0.00 sec 4/5 Test #3: core_synthesis_mag ...............***Failed 0.01 sec + '[' '!' -z ']' + '[' '!' -z ']' + '[' '!' -z ']' + '[' '!' -z 1 ']' + ../build_linux/src/dump_data --mag --test --c2pitch ../wav/c01_01.wav c01_01.f32 ./test_core_nn.sh: line 60: ../build_linux/src/dump_data: No such file or directory + diff c01_01_mag.f32 c01_01.f32 diff: c01_01.f32: No such file or directory + echo 'ERROR in synth .f32 output! Exiting...' ERROR in synth .f32 output! Exiting... + exit 1 5/5 Test #4: nnet2f32 ......................... Passed 0.06 sec 40% tests passed, 3 tests failed out of 5 Total Test time (real) = 0.06 sec The following tests FAILED: 1 - core_synthesis_default (Failed) 2 - core_synthesis_load_20h (Failed) 3 - core_synthesis_mag (Failed) Errors while running CTest ```

[kloczek]

FYI: to have base line working temporary I've added those 3 failing units to ctest exclude list so it no-so-critical. As there are some missing files look slike it is test suite and not actual tested code issue.

[tmiw]

Oh right, I missed the title.

Anyway, git diff 391ea7c CMakeLists.txt, among other changes, shows that the failing tests were removed:

@@ -192,16 +203,26 @@ add_subdirectory(src)
 include(CTest)
 enable_testing()

-add_test(NAME core_synthesis_default
-         COMMAND sh -c "cd ${CMAKE_CURRENT_SOURCE_DIR}/unittest; SYNTH=1 ./test_core_nn.sh")
-add_test(NAME core_synthesis_load_20h
-         COMMAND sh -c "cd ${CMAKE_CURRENT_SOURCE_DIR}/unittest; SYNTH_20h=1 ./test_core_nn.sh")
-add_test(NAME core_synthesis_mag
-         COMMAND sh -c "cd ${CMAKE_CURRENT_SOURCE_DIR}/unittest; SYNTH_mag=1 ./test_core_nn.sh")
+# some basic tests - unfortunately the core NN synthsis is hard to test reliably as we get
+# different results on different runs due to numerical/precision issues
+add_test(NAME feature_extraction
+         COMMAND sh -c "PATH=$PATH:${CMAKE_CURRENT_BINARY_DIR}/src:${CMAKE_CURRENT_BINARY_DIR}/unittest;
+                        cd ${CMAKE_CURRENT_SOURCE_DIR}/unittest; 
+                        pwd;
+                        dump_data --test --c2pitch ${CMAKE_CURRENT_SOURCE_DIR}/wav/birch.wav birch.f32;
+                        md5sum birch.f32;
+                        md5sum birch_targ.f32;
+                        diff32 --cont birch_targ.f32 birch.f32")
 add_test(NAME nnet2f32
          COMMAND sh -c "cd ${CMAKE_CURRENT_BINARY_DIR}; ./src/nnet2f32 t.f32")
 add_test(NAME SIMD_functions
          COMMAND sh -c "cd ${CMAKE_CURRENT_BINARY_DIR}; ./src/test_vec")
+add_test(NAME lpcnet_enc_dec
+         COMMAND sh -c "PATH=$PATH:${CMAKE_CURRENT_BINARY_DIR}/src;
+                        cd ${CMAKE_CURRENT_SOURCE_DIR};
+                        sox wav/wia.wav -t raw -r 16000 - | 
+                        lpcnet_enc -s | 
+                        lpcnet_dec -s > /dev/null")

 # Packaging
 set(CPACK_PACKAGE_DESCRIPTION_SUMMARY "Next-Generation Digital Voice for Two-Way Radio")

(391ea7c is the change that marks the release of 0.3. Diffing just up to the release of 0.4 -- 1431ee3 -- shows this same change.)

Anyway, I'd recommend trying version 0.4 instead, though neither 0.3 nor 0.4 have officially been released on https://github.com/drowe67/LPCNet/releases. @drowe67, should I go ahead and tag/create GitHub releases accordingly?

[drowe67]

@tmiw - sorry I might have missed your latest comment :point_up:

Do you feel master ready to be tagged and released? It builds and 4/4 ctests pass for me.

[Tyrbiter]

FWIW I have been using a local build of the master branch, I think this is effectively v0.4, which is what I have named it in my own packages.

It's been working quite happily, and decodes all the 2020 modes which the current released lpcnetfreedv-0.2 in Fedora 38 doesn't.

[tmiw]

@tmiw - sorry I might have missed your latest comment ☝️

Do you feel master ready to be tagged and released? It builds and 4/4 ctests pass for me.

That works for me. Should this be tagged as 0.5? It seems there were quite a few commits after CMakeLists.txt was updated to 0.4.

[Tyrbiter]

0.5 seems sensible to me.

[drowe67]

Thanks everyone I've had a go at releasing v0.5

[Tyrbiter]

Thanks, I now have a new build of up-to-date codec2, lpcnetfreedv and freedv packages installed and running on Fedora 38.