search

open-quantum-safe / liboqs

C library for prototyping and experimenting with quantum-resistant cryptography
https://openquantumsafe.org/
Other
1.82k stars 447 forks source link

The library always links against libpthreads #1811

Open maroueneboubakri opened 3 months ago

maroueneboubakri commented 3 months ago

Hello,

Some PRs introduced OQS_USE_PTHREADS flag to decide if to use pthreads or not, however seems that this does not have an effect. The library always links against pthread.

The flag is hardcoded to 1 in https://github.com/open-quantum-safe/liboqs/blob/main/src/oqsconfig.h.cmake#L30

If you look at https://github.com/open-quantum-safe/liboqs/blob/main/src/common/sha3/xkcp_sha3.c#L16 means that the code will reference always pthread.

This issue for example does not allow the oqsprovider tests to build (static): undefined reference to pthread_once

Could you please check.

Best Maro

SWilson4 commented 3 months ago

Hi @maroueneboubakri, thanks for filing an issue. Could you please provide details of your environment (OS, architecture, etc) and the build commands you're using? This will allow me to try to reproduce the issue.

maroueneboubakri commented 3 months ago

Hi @SWilson4,

Machine: x64 with Ubuntu 20.04

Build instructions:

cd ~/openssl
OPENSSL_BRANCH="openssl-3.2.0"
OPENSSL_PREFIX=`pwd`/build
git clone --depth 1 --branch $OPENSSL_BRANCH https://github.com/openssl/openssl.git && cd openssl && LDFLAGS="-Wl,-rpath -Wl,${OPENSSL_PREFIX}/lib64" ./config --prefix=$OPENSSL_PREFIX && make && make install_sw install_ssldirs

cd ~/liboqs
LIBOQS_BRANCH="0.10.0"
git clone --depth 1 --branch $LIBOQS_BRANCH https://github.com/open-quantum-safe/liboqs.git
cd liboqs
cmake -GNinja -DOQS_ALGS_ENABLED=STD -DOQS_BUILD_ONLY_LIB=ON -DOQS_MINIMAL_BUILD="SIG_dilithium_3" -DOQS_DIST_BUILD=ON -DOQS_USE_OPENSSL=OFF -DOQS_USE_PTHREADS=OFF -DCMAKE_INSTALL_PREFIX=./build -S . -B build && cd build && ninja && ninja install

cd ~/oqsprovider
OQSPROVIDER_BRANCH="0.6.0"
git clone --depth 1 --branch $OQSPROVIDER_BRANCH https://github.com/open-quantum-safe/oqs-provider.git
cd oqs-provider
liboqs_DIR=$(pwd)/../liboqs/build/ cmake -DOPENSSL_ROOT_DIR=$(pwd)/../openssl -DOQS_PROVIDER_BUILD_STATIC=ON -S . -B build && cmake --build build

> Fails here due to unresolved symbol pthread_once.

Hope this helps.

Br Maro

SWilson4 commented 3 months ago

I'm unable to duplicate the issue in an Ubuntu 20.04 Docker image. What version of CMake are you using?

maroueneboubakri commented 3 months ago

I'm unable to duplicate the issue in an Ubuntu 20.04 Docker image. What version of CMake are you using?

3.27.7

SWilson4 commented 3 months ago

Hi @maroueneboubakri, sorry it's taken me so long to get back to this.

I was able to duplicate the build failure. At second glance, I don't believe it's a bug. The OQS_USE_PTHREADS variable is not intended to be user-configurable: note that it does not appear in CONFIGURE.md. Instead, it's automatically set by CMake depending on whether or not pthreads is available. We do expose the OQS_EMBEDDED_BUILD configuration option, which will ensure that pthreads is not used. This option also affects random number generation. Would this option be applicable to your use case?

maroueneboubakri commented 3 months ago

Hi @maroueneboubakri, sorry it's taken me so long to get back to this.

I was able to duplicate the build failure. At second glance, I don't believe it's a bug. The OQS_USE_PTHREADS variable is not intended to be user-configurable: note that it does not appear in CONFIGURE.md. Instead, it's automatically set by CMake depending on whether or not pthreads is available. We do expose the OQS_EMBEDDED_BUILD configuration option, which will ensure that pthreads is not used. This option also affects random number generation. Would this option be applicable to your use case?

Hi @SWilson4,

Thank you for your time ! Not sure if this flag is applicable in my case, since I'm building for a desktop machine, not an embedded environment. I mean as a workaround to get rid of pthread does not sound a good approach. Could you consider filing a bug for this one ?

Br Maro

SWilson4 commented 3 months ago

Hi @SWilson4,

Thank you for your time !

No worries!

Not sure if this flag is applicable in my case, since I'm building for a desktop machine, not an embedded environment. I mean as a workaround to get rid of pthread does not sound a good approach. Could you consider filing a bug for this one ?

Br Maro

Would you be willing to create a PR implementing the functionality you're looking for? Then we can review it and try to get it merged in.

MMDHBZ commented 2 months ago

Hello, I turned on this option “OQS_USE_PTHREADS”when cmake. OQS_USE_PTHREADS cmake -GNinja.. -DBUILD_SHARED_LIBS=ON -DOQS_USE_PTHREADS=ON, it is still found that there will be competition between threads. How to eliminate contention between threads when only multithreading?

SWilson4 commented 2 months ago

Hello, I turned on this option “OQS_USE_PTHREADS”when cmake. OQS_USE_PTHREADS cmake -GNinja.. -DBUILD_SHARED_LIBS=ON -DOQS_USE_PTHREADS=ON, it is still found that there will be competition between threads. How to eliminate contention between threads when only multithreading?

Hi @MMDHBZ, thanks for the report! As I wrote above, the OQS_USE_PTHREADS variable is not intended to be set manually, so I can't really comment on the library behaviour when it's built in this fashion.

The OQS_USE_PTHREADS variable is not intended to be user-configurable: note that it does not appear in CONFIGURE.md. Instead, it's automatically set by CMake depending on whether or not pthreads is available.

Could you please let me know how you observed/measured the thread contention?

MMDHBZ commented 2 months ago

Hello, I turned on this option “OQS_USE_PTHREADS”when cmake. OQS_USE_PTHREADS cmake -GNinja.. -DBUILD_SHARED_LIBS=ON -DOQS_USE_PTHREADS=ON, it is still found that there will be competition between threads. How to eliminate contention between threads when only multithreading?

Hi @MMDHBZ, thanks for the report! As I wrote above, the OQS_USE_PTHREADS variable is not intended to be set manually, so I can't really comment on the library behaviour when it's built in this fashion.

The OQS_USE_PTHREADS variable is not intended to be user-configurable: note that it does not appear in CONFIGURE.md. Instead, it's automatically set by CMake depending on whether or not pthreads is available.

Could you please let me know how you observed/measured the thread contention?

My machine tested has 12 cores, and the test unit is Tps, that is, how many times it is run per second. When executing a single thread, the measured performance of kyber512 encryption is 27000Tps, but when executing 12 threads, the performance does not increase, which seems to be abnormal. It is said that multi-threading has higher performance than single-threading. Why does the above situation occur? Or, how should I configure it so that the performance of multi-threaded testing will be higher than that of single-threaded testing?

baentsch commented 2 months ago

how should I configure it so that the performance of multi-threaded testing will be higher than that of single-threaded testing?

That pretty much entirely depends on whether you have a test program that makes use of threading. Is that the case? How? The OQS built-in tests don't.