search

simsong / bulk_extractor

This is the development tree. Production downloads are at:
https://github.com/simsong/bulk_extractor/releases
Other
1.08k stars 185 forks source link

test_be failed with latest version 2.0.0 beta 3 only on 32 bit architectures #318

Closed sbrun closed 2 years ago

sbrun commented 2 years ago

Hello

Trying to package the latest version 2.0.0 beta 3 for Kali, we got an error during the build, when it runs the tests. We do not have the error on amd64 or arm64, but only on i386, armel and armhf, all of our 32 bit architectures.

====================================================
   BULK_EXTRACTOR 2.0.0-beta3: src/test-suite.log
====================================================

# TOTAL: 1
# PASS:  0
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: test_be
=============

## output in "/tmp/be_tmp9382916d7d45453f" for email
## output in "/tmp/be_tmp589b0e11399e262b" for email
## output in "/tmp/be_tmp6e24ce73378a8268" for pdf
## output in "/tmp/be_tmp1897aa2b8a04cb5a" for email
## output in "/tmp/be_tmp2decc29d1371c843" for json
## output in "/tmp/be_tmpfa29b3ae7ef1f818" for vcard
## output in "/tmp/be_tmpfc9c19d9bfac5950" for wordlist
## output in "/tmp/be_tmp1d9866519fbef15a" for zip
## image_fname: ram_2pages.bin outdir: "/tmp/be_tmp779358886e9a4352"
test_be: test_be.cpp:619: std::filesystem::__cxx11::path validate(std::string, std::vector<Check>&, bool, size_t): Assertion `ss.bytes_in_queue == 0' failed.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
test_be is a Catch v2.13.6 host application.
Run with -? for options

-----------------------------------------------------------------------------------------------------------------------
test_aes
-----------------------------------------------------------------------------------------------------------------------
test_be.cpp:689
.......................................................................................................................

test_be.cpp:689: FAILED:
  {Unknown expression after the reported line}
due to a fatal error condition:
  SIGABRT - Abort (abnormal termination) signal

=======================================================================================================================
test cases:  18 |  17 passed | 1 failed
assertions: 102 | 101 passed | 1 failed

FAIL test_be (exit status: 134)
simsong commented 2 years ago

This is a multi threading error. It’s surprising it doesn’t appear on the other platforms. It may be how atomics are implemented on your platform. Can you please post the compiler you are using and the hardware?

I am also getting errors with thread sanitizer but only with gcc on Linux, not with llvm on macOS. Can you try the same platform with a different compiler?

Can you give me a compelling use case to support 32 bit architectures moving forward? How fast is the cpu? This may not warrant a lot of effort to fix. However, it does represent an underlying bug.

What happens if you compile in 32 bit mode on a 64 bit system?

Sent from my phone.

On Dec 21, 2021, at 5:05 AM, Sophie Brun @.***> wrote:

 Hello

Trying to package the latest version 2.0.0 beta 3 for Kali, we got an error during the build, when it runs the tests. We do not have the error on amd64 or arm64, but only on i386, armel and armhf, all of our 32 bit architectures.

==================================================== BULK_EXTRACTOR 2.0.0-beta3: src/test-suite.log

TOTAL: 1

PASS: 0

SKIP: 0

XFAIL: 0

FAIL: 1

XPASS: 0

ERROR: 0

.. contents:: :depth: 2

FAIL: test_be

output in "/tmp/be_tmp9382916d7d45453f" for email

output in "/tmp/be_tmp589b0e11399e262b" for email

output in "/tmp/be_tmp6e24ce73378a8268" for pdf

output in "/tmp/be_tmp1897aa2b8a04cb5a" for email

output in "/tmp/be_tmp2decc29d1371c843" for json

output in "/tmp/be_tmpfa29b3ae7ef1f818" for vcard

output in "/tmp/be_tmpfc9c19d9bfac5950" for wordlist

output in "/tmp/be_tmp1d9866519fbef15a" for zip

image_fname: ram_2pages.bin outdir: "/tmp/be_tmp779358886e9a4352"

test_be: test_be.cpp:619: std::filesystem::__cxx11::path validate(std::string, std::vector&, bool, size_t): Assertion `ss.bytes_in_queue == 0' failed.


test_be is a Catch v2.13.6 host application.
Run with -? for options

-----------------------------------------------------------------------------------------------------------------------
test_aes
-----------------------------------------------------------------------------------------------------------------------
test_be.cpp:689
.......................................................................................................................

test_be.cpp:689: FAILED:
  {Unknown expression after the reported line}
due to a fatal error condition:
  SIGABRT - Abort (abnormal termination) signal

=======================================================================================================================
test cases:  18 |  17 passed | 1 failed
assertions: 102 | 101 passed | 1 failed

FAIL test_be (exit status: 134)
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android. 
You are receiving this because you are subscribed to this thread.
simsong commented 2 years ago

As a follow-up, please note that ss.bytes_in_queue is a 64-bit atomic:

https://github.com/simsong/be13_api/blob/bcf7e5c2ec0f2f0a802d15c4942b45b98ae5d2e1/scanner_set.h#L200

Is it possible that your 32-bit system does not properly implement 64-bit atomics?

simsong commented 2 years ago

See also:

This is the problem. Your compiler is allowing you to compile 32-bit code that specifies a 64-bit atomic. The compiler should throw an error or provide sufficient code such that the 64-bit atomic operations are actually atomic. The exception that is being thrown is checking to make sure that the 64-bit counter has been properly decremented to 0. It gets incremented in the main thread and decremented in worker threads.

I am using 64-bit atomics because a 32-bit byte counter would rapidly be wrapped on a digital forensics tool that is designed to process disk images of any size and is tested up to 2TB disk images. This is not an application where any user would likely want to use a 32-bit processor.

Thanks for bringing this to my attention, but this is a compiler bug that I will not code around.

sbrun commented 2 years ago

Thanks for your quick and detailed answer! Maybe you can document in the README that 32 bit arch are not supported.

simsong commented 2 years ago

Thanks for the email. 32-bit architectures should be supported if they implement std::atomic for 64-bit values. There is nothing in the spec that should prevent this. We are seeing a compiler bug.

Perhaps @jonstewart can provide some background here, as he is my go-to C++ expert. My understanding is that C++20 provides better support for std::atomic, automatically adding a mutex if one is required.

There may also be 32-bit architectures that support 64-bit atomic operations.

However, it appears that the self-test is working, and that your have operationally determined that this particular 32-bit architectures with the particular compiler does not produce reliable code. Can you please post the compiler that you are using?

sbrun commented 2 years ago

I'm using the default gcc from Debian Testing / Kali: gcc (Debian 11.2.0-12) 11.2.0

I will investigate more to try to find a solution.

simsong commented 2 years ago

A simple solution is to just disable multithreading. There is full support for running bulk_extractor with a single thread. If you can give me a compile-time #define or some other way to detect that bulk_extractor is running on a 32-bit processor, I will make this the default on 32-bit systems.

On Dec 22, 2021, at 4:33 AM, Sophie Brun @.***> wrote:

I'm using the default gcc from Debian Testing / Kali: gcc (Debian 11.2.0-12) 11.2.0

I will investigate more to try to find a solution.

— Reply to this email directly, view it on GitHub https://github.com/simsong/bulk_extractor/issues/318#issuecomment-999426153, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMFHLBLSKTUPZXN3CHT4FLUSGLNZANCNFSM5KPYPVVA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you modified the open/close state.