1.1.0 - Test failures on `x86` when using external mspack

[Kangie]

Describe the bug

Failing unit tests on x86; does not seem to be detecting expected files:

AssertionError: Expected item `clam_ISmsi_int.exe: ClamAV-Test-File.UNOFFICIAL FOUND` not found in output:

/tmp/clamav-1.1.0/build # lsb_release -a
LSB Version:    n/a
Distributor ID: Gentoo
Description:    Gentoo Linux
Release:        2.13
Codename:       n/a

How to reproduce the problem

clamav 1.1.0 built from clamav.net sources, configured with:

cmake .. -G Ninja -DCMAKE_INSTALL_PREFIX=/usr -DAPP_CONFIG_DIRECTORY=/etc/clamav -DBYTECODE_RUNTIME=interpreter -DCLAMAV_GROUP=clamav -DCLAMAV_USER=clamav -DDATABASE_DIRECTORY=/var/lib/clamav -DENABLE_APP=ON -DENABLE_CLAMONACC=OFF -DENABLE_DOXYGEN=no -DENABLE_EXPERIMENTAL=OFF -DENABLE_EXTERNAL_MSPACK=ON -DENABLE_JSON_SHARED=ON -DENABLE_MAN_PAGES=ON -DENABLE_MILTER=OFF -DENABLE_SHARED_LIB=ON -DENABLE_STATIC_LIB=OFF -DENABLE_SYSTEMD=ON -DENABLE_TESTS=ON -DENABLE_UNRAR=OFF -DOPTIMIZE=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo

monolith /tmp/clamav-1.1.0/build # ctest
Test project /tmp/clamav-1.1.0/build
    Start 1: libclamav
1/6 Test #1: libclamav ........................***Failed    4.28 sec
    Start 2: libclamav_rust
2/6 Test #2: libclamav_rust ...................   Passed    0.08 sec
    Start 3: clamscan
3/6 Test #3: clamscan .........................***Failed    3.68 sec
    Start 4: clamd
4/6 Test #4: clamd ............................***Failed   11.38 sec
    Start 5: freshclam
5/6 Test #5: freshclam ........................   Passed   41.15 sec
    Start 6: sigtool
6/6 Test #6: sigtool ..........................   Passed    0.74 sec

50% tests passed, 3 tests failed out of 6

Total Test time (real) =  61.31 sec

The following tests FAILED:
          1 - libclamav (Failed)
          3 - clamscan (Failed)
          4 - clamd (Failed)

Attachments

clamav_x86_failures.zip

[Kangie]

I can confirm that tests pass in a lightweight amd64 build container, it seems limited to x86.

I did encounter a test failure in libclamav_rust on amd64 but that was with musl libc - I'm not sure if that's supposed to work?

[micahsnyder]

I haven't seen these failures on x86 in testing, or with libclamav_rust on amd64 with musl (tested via Alpine). So... yeah I would expect both scenarios to pass.

[micahsnyder]

Oh I see in your build options you're using -DENABLE_EXTERNAL_MSPACK=ON. It's likely this is where the bug is on x86, and quite likely with the libclamav_rust test as well, which may have some sort of linking bug with an external libmspack.so.

[Kangie]

Is there any additional info that I can provide / testing that I can do to assist with this one?

I might try and repro both failures -DENABLE_EXTERNAL_MSPACK=OFF and get back to you regardless.

We've also marked clamav 1.1.0 as stable in Gentoo; does this test failure indicate likely false negatives for our users on x86 and musl amd64?

Thanks for the response!

Edit:

x86 with -DENABLE_EXTERNAL_MSPACK=OFF :

(chroot) monolith /tmp/clamav-1.1.0/build # ctest
Test project /tmp/clamav-1.1.0/build
    Start 1: libclamav
1/6 Test #1: libclamav ........................   Passed    5.01 sec
    Start 2: libclamav_rust
2/6 Test #2: libclamav_rust ...................   Passed    3.35 sec
    Start 3: clamscan
3/6 Test #3: clamscan .........................   Passed    3.85 sec
    Start 4: clamd
4/6 Test #4: clamd ............................   Passed   11.83 sec
    Start 5: freshclam
5/6 Test #5: freshclam ........................   Passed   41.22 sec
    Start 6: sigtool
6/6 Test #6: sigtool ..........................   Passed    0.76 sec

100% tests passed, 0 tests failed out of 6

Total Test time (real) =  66.02 sec

amd64 musl resulted in a build failure instead (which I guess is an improvement?)

clamav-musl-build.log.txt

Edit the second: Gentoo currently has libmspack 0.10.1alpha packaged; I'll see about bumping that to 0.11alpha in Gentoo to match clamav's bundled version to see if that improves the situation. It doesn't, same failures as before.

[micahsnyder]

We've also marked clamav 1.1.0 as stable in Gentoo; does this test failure indicate likely false negatives for our users on x86 and musl amd64?

With the failures relating to external mspack - yes that likely means false negatives when scanning CAB and CHM files.

Regarding the clamav-musl-build.log.txt, I see it is failing with:

/usr/lib/gcc/x86_64-gentoo-linux-musl/12/../../../../x86_64-gentoo-linux-musl/bin/ld: cannot find -lunwind: No such file or directory
collect2: error: ld returned 1 exit status

I think it's a complication related to rust on musl. I found a similar error here https://github.com/rust-lang/wg-cargo-std-aware/issues/76. I'm not sure what the solution is. Perhaps if you set this env variable before building it will remove the unwind dependency: CARGO_PROFILE_RELEASE_PANIC="abort" I found this idea here: https://users.rust-lang.org/t/remove-unwind-routines-from-a-musl-linked-build/91634

[Kangie]

We've been able to repro issues on 32-bit arm and ppc - It seems like this might impact all 32-bit platforms when built using system mspack.

https://bugs.gentoo.org/911402