Closed sbrun closed 2 years ago
This is a multi threading error. It’s surprising it doesn’t appear on the other platforms. It may be how atomics are implemented on your platform. Can you please post the compiler you are using and the hardware?
I am also getting errors with thread sanitizer but only with gcc on Linux, not with llvm on macOS. Can you try the same platform with a different compiler?
Can you give me a compelling use case to support 32 bit architectures moving forward? How fast is the cpu? This may not warrant a lot of effort to fix. However, it does represent an underlying bug.
What happens if you compile in 32 bit mode on a 64 bit system?
Sent from my phone.
On Dec 21, 2021, at 5:05 AM, Sophie Brun @.***> wrote:
Hello
Trying to package the latest version 2.0.0 beta 3 for Kali, we got an error during the build, when it runs the tests. We do not have the error on amd64 or arm64, but only on i386, armel and armhf, all of our 32 bit architectures.
.. contents:: :depth: 2
test_be: test_be.cpp:619: std::filesystem::__cxx11::path validate(std::string, std::vector
test_be is a Catch v2.13.6 host application.
Run with -? for options
-----------------------------------------------------------------------------------------------------------------------
test_aes
-----------------------------------------------------------------------------------------------------------------------
test_be.cpp:689
.......................................................................................................................
test_be.cpp:689: FAILED:
{Unknown expression after the reported line}
due to a fatal error condition:
SIGABRT - Abort (abnormal termination) signal
=======================================================================================================================
test cases: 18 | 17 passed | 1 failed
assertions: 102 | 101 passed | 1 failed
FAIL test_be (exit status: 134)
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you are subscribed to this thread.
As a follow-up, please note that ss.bytes_in_queue
is a 64-bit atomic:
https://github.com/simsong/be13_api/blob/bcf7e5c2ec0f2f0a802d15c4942b45b98ae5d2e1/scanner_set.h#L200
Is it possible that your 32-bit system does not properly implement 64-bit atomics?
See also:
This is the problem. Your compiler is allowing you to compile 32-bit code that specifies a 64-bit atomic. The compiler should throw an error or provide sufficient code such that the 64-bit atomic operations are actually atomic. The exception that is being thrown is checking to make sure that the 64-bit counter has been properly decremented to 0. It gets incremented in the main thread and decremented in worker threads.
I am using 64-bit atomics because a 32-bit byte counter would rapidly be wrapped on a digital forensics tool that is designed to process disk images of any size and is tested up to 2TB disk images. This is not an application where any user would likely want to use a 32-bit processor.
Thanks for bringing this to my attention, but this is a compiler bug that I will not code around.
Thanks for your quick and detailed answer! Maybe you can document in the README that 32 bit arch are not supported.
Thanks for the email. 32-bit architectures should be supported if they implement std::atomic
for 64-bit values. There is nothing in the spec that should prevent this. We are seeing a compiler bug.
Perhaps @jonstewart can provide some background here, as he is my go-to C++ expert. My understanding is that C++20 provides better support for std::atomic
, automatically adding a mutex if one is required.
There may also be 32-bit architectures that support 64-bit atomic operations.
However, it appears that the self-test is working, and that your have operationally determined that this particular 32-bit architectures with the particular compiler does not produce reliable code. Can you please post the compiler that you are using?
I'm using the default gcc from Debian Testing / Kali: gcc (Debian 11.2.0-12) 11.2.0
I will investigate more to try to find a solution.
A simple solution is to just disable multithreading. There is full support for running bulk_extractor with a single thread. If you can give me a compile-time #define or some other way to detect that bulk_extractor is running on a 32-bit processor, I will make this the default on 32-bit systems.
On Dec 22, 2021, at 4:33 AM, Sophie Brun @.***> wrote:
I'm using the default gcc from Debian Testing / Kali: gcc (Debian 11.2.0-12) 11.2.0
I will investigate more to try to find a solution.
— Reply to this email directly, view it on GitHub https://github.com/simsong/bulk_extractor/issues/318#issuecomment-999426153, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMFHLBLSKTUPZXN3CHT4FLUSGLNZANCNFSM5KPYPVVA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you modified the open/close state.
Hello
Trying to package the latest version 2.0.0 beta 3 for Kali, we got an error during the build, when it runs the tests. We do not have the error on amd64 or arm64, but only on i386, armel and armhf, all of our 32 bit architectures.