Visa-Research / volepsi

Efficient Private Set Intersection base on VOLE
MIT License
104 stars 32 forks source link

warnings and errors #70

Closed Feir2 closed 4 months ago

Feir2 commented 4 months ago

Hi, I'm sorry to bother you. I'm trying to use OPRF of volepsi in https://github.com/asu-crypto/mPSI, but I encountered a difficulty that has been bothering me for a long time. Could you please give me some help?

I build volepsi successfully and install it as follows. python3 build.py --install=/home/user/MyFile/mPSI5/mPSI/libVolepsi Then, I wrote the following code in the file frontend/CMakeLists.txt in https://github.com/asu-crypto/mPSI.

find_package(volePSI REQUIRED HINTS "../libVolepsi")
target_link_libraries(frontend.exe visa::volePSI)

When I cmake it, I encountered these warnings.

user@user-virtual-machine:~/MyFile/mPSI5/mPSI/build$ cmake ..
CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found Boost: /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost (found version "1.64.0") found components: system thread missing components: chrono date_time atomic
Found Boost library at /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_system.a;/home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_thread.a
Found Boost library at /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib
Found Boost include at /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost
Found Boost system library at /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_system.a
Found Boost thread library at /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_thread.a
frontend is using boost at: /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_system.a;/home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_thread.a
-- VOLEPSI_THIRDPARTY_DIR=/home/user/MyFile/mPSI5/mPSI/libVolepsi
SPARSEHASH_INCLUDE_DIRS=/home/user/MyFile/mPSI5/mPSI/libVolepsi/include

libOTe_options=silentot;silent_vole;circuits;sse;boost;bitpolymul;no_asan;no_pic;sodium;sodium_montgomery
-- SODIUM_INCLUDE_DIRS:  /home/user/MyFile/mPSI5/mPSI/libVolepsi/include
-- SODIUM_LIBRARIES:  /home/user/MyFile/mPSI5/mPSI/libVolepsi/lib/libsodium.a
-- SODIUM_MONTGOMERY:  ON

macoroConfig.cmake : /home/user/MyFile/mPSI5/mPSI/libVolepsi/lib/cmake/macoro/macoroConfig.cmake
CMake Warning (dev) at libVolepsi/lib/cmake/coproto/coprotoFindDeps.cmake:130 (option):
  Policy CMP0077 is not set: option() honors normal variables.  Run "cmake
  --help-policy CMP0077" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

  For compatibility with older versions of CMake, option is clearing the
  normal variable 'Boost_USE_MULTITHREADED'.
Call Stack (most recent call first):
  libVolepsi/lib/cmake/coproto/coprotoFindDeps.cmake:160 (FIND_BOOST)
  libVolepsi/lib/cmake/coproto/coprotoConfig.cmake:15 (include)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsDepHelper.cmake:215 (find_package)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsDepHelper.cmake:225 (FIND_COPROTO)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsConfig.cmake:26 (include)
  libVolepsi/lib/cmake/libOTe/libOTeConfig.cmake:12 (find_package)
  libVolepsi/lib/cmake/volePSI/findDependancies.cmake:104 (find_package)
  libVolepsi/lib/cmake/volePSI/findDependancies.cmake:118 (FIND_LIBOTE)
  libVolepsi/lib/cmake/volePSI/volePSIConfig.cmake:26 (include)
  frontend/CMakeLists.txt:30 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at libVolepsi/lib/cmake/coproto/coprotoFindDeps.cmake:142 (option):
  Policy CMP0077 is not set: option() honors normal variables.  Run "cmake
  --help-policy CMP0077" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

  For compatibility with older versions of CMake, option is clearing the
  normal variable 'Boost_USE_DEBUG_RUNTIME'.
Call Stack (most recent call first):
  libVolepsi/lib/cmake/coproto/coprotoFindDeps.cmake:160 (FIND_BOOST)
  libVolepsi/lib/cmake/coproto/coprotoConfig.cmake:15 (include)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsDepHelper.cmake:215 (find_package)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsDepHelper.cmake:225 (FIND_COPROTO)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsConfig.cmake:26 (include)
  libVolepsi/lib/cmake/libOTe/libOTeConfig.cmake:12 (find_package)
  libVolepsi/lib/cmake/volePSI/findDependancies.cmake:104 (find_package)
  libVolepsi/lib/cmake/volePSI/findDependancies.cmake:118 (FIND_LIBOTE)
  libVolepsi/lib/cmake/volePSI/volePSIConfig.cmake:26 (include)
  frontend/CMakeLists.txt:30 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at libVolepsi/lib/cmake/coproto/coprotoFindDeps.cmake:146 (find_package):
  Policy CMP0144 is not set: find_package uses upper-case <PACKAGENAME>_ROOT
  variables.  Run "cmake --help-policy CMP0144" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  CMake variable BOOST_ROOT is set to:

    /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/

  For compatibility, find_package is ignoring the variable, but code in a
  .cmake module might still use it.
Call Stack (most recent call first):
  libVolepsi/lib/cmake/coproto/coprotoFindDeps.cmake:160 (FIND_BOOST)
  libVolepsi/lib/cmake/coproto/coprotoConfig.cmake:15 (include)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsDepHelper.cmake:215 (find_package)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsDepHelper.cmake:225 (FIND_COPROTO)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsConfig.cmake:26 (include)
  libVolepsi/lib/cmake/libOTe/libOTeConfig.cmake:12 (find_package)
  libVolepsi/lib/cmake/volePSI/findDependancies.cmake:104 (find_package)
  libVolepsi/lib/cmake/volePSI/findDependancies.cmake:118 (FIND_LIBOTE)
  libVolepsi/lib/cmake/volePSI/volePSIConfig.cmake:26 (include)
  frontend/CMakeLists.txt:30 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Found Boost: /home/user/MyFile/mPSI5/mPSI/libVolepsi/lib/cmake/Boost-1.77.0/BoostConfig.cmake (found suitable version "1.77.0", minimum required is "1.77.0") found components: system thread regex
-- 

Boost_LIB: Boost::system;Boost::thread;Boost::regex
-- Boost_INC: /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost

-- LIBDIVIDE_INCLUDE_DIRS:  /home/user/MyFile/mPSI5/mPSI/libVolepsi/include
-- bitpolymulConfig.cmake : /home/user/MyFile/mPSI5/mPSI/libVolepsi/lib/cmake/bitpolymul
volePSI_INCLUDE_DIRS=/home/user/MyFile/mPSI5/mPSI/libVolepsi/include;/home/user/MyFile/mPSI5/mPSI/libVolepsi/include
volePSI_LIBRARIES=/home/user/MyFile/mPSI5/mPSI/libVolepsi/lib/libvolePSI.a
-- Configuring done (1.3s)
CMake Warning (dev) at /snap/cmake/1384/share/cmake-3.29/Modules/FindBoost.cmake:2503 (set_target_properties):
  Policy CMP0028 is not set: Double colon in target name means ALIAS or
  IMPORTED target.  Run "cmake --help-policy CMP0028" for policy details.
  Use the cmake_policy command to set the policy and suppress this warning.

  The link interface of target "Boost::thread" contains:

    Boost::chrono

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.

Call Stack (most recent call first):
  frontend/CMakeLists.txt:25 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done (0.2s)
-- Build files have been written to: /home/user/MyFile/mPSI5/mPSI/build

After that, I make it and encounter the following error. Could you please tell me what I can do to solve this problem or how can I use OPRF of volepsi in https://github.com/asu-crypto/mPSI? Thank you very much.

user@user-virtual-machine:~/MyFile/mPSI5/mPSI/build$ make
...
[100%] Linking CXX executable /home/user/MyFile/mPSI5/mPSI/bin/frontend.exe
/usr/bin/ld: Can't find -lBoost::chrono: no such file or directory
/usr/bin/ld: Can't find -lBoost::date_time: no such file or directory
/usr/bin/ld: Can't find -lBoost::atomic: no such file or directory
collect2: error: ld returned 1 exit status
make[2]: *** [frontend/CMakeFiles/frontend.exe.dir/build.make:211:/home/user/MyFile/mPSI5/mPSI/bin/frontend.exe] error 1
make[1]: *** [CMakeFiles/Makefile2:305:frontend/CMakeFiles/frontend.exe.dir/all] error 2
make: *** [Makefile:91:all] error 2
ladnir commented 4 months ago

try increasing your cmake version, currently 2.8. Change it to something like 3.20.

Feir2 commented 4 months ago

try increasing your cmake version, currently 2.8. Change it to something like 3.20.

Thank you for your reply. But the cmake version I use is 3.29.3.

user@user-virtual-machine:~/MyFile/mPSI5/mPSI/build$ cmake --version
cmake version 3.29.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).

I tried to change the first line of CMakeLists.txt from cmake_minimum_required (VERSION 2.8) to

cmake_minimum_required (VERSION 3.15)
cmake_policy(SET CMP0144 NEW)

but the following error occurred.

user@user-virtual-machine:~/MyFile/mPSI5/mPSI/build$ cmake ..
-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found Boost: /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost (found version "1.64.0") found components: system thread missing components: chrono date_time atomic
Found Boost library at /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_system.a;/home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_thread.a
Found Boost library at /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib
Found Boost include at /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost
Found Boost system library at /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_system.a
Found Boost thread library at /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_thread.a
frontend is using boost at: /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_system.a;/home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_thread.a
-- VOLEPSI_THIRDPARTY_DIR=/home/user/MyFile/mPSI5/mPSI/libVolepsi
SPARSEHASH_INCLUDE_DIRS=/home/user/MyFile/mPSI5/mPSI/libVolepsi/include

libOTe_options=silentot;silent_vole;circuits;sse;boost;bitpolymul;no_asan;no_pic;sodium;sodium_montgomery
-- SODIUM_INCLUDE_DIRS:  /home/user/MyFile/mPSI5/mPSI/libVolepsi/include
-- SODIUM_LIBRARIES:  /home/user/MyFile/mPSI5/mPSI/libVolepsi/lib/libsodium.a
-- SODIUM_MONTGOMERY:  ON

macoroConfig.cmake : /home/user/MyFile/mPSI5/mPSI/libVolepsi/lib/cmake/macoro/macoroConfig.cmake
CMake Error at libVolepsi/lib/cmake/Boost-1.77.0/BoostConfig.cmake:141 (find_package):
  Found package configuration file:

    /home/user/MyFile/mPSI5/mPSI/libVolepsi/lib/cmake/boost_regex-1.77.0/boost_regex-config.cmake

  but it set boost_regex_FOUND to FALSE so package "boost_regex" is
  considered to be NOT FOUND.  Reason given by package:

  No suitable build variant has been found.

  The following variants have been tried and rejected:

  * libboost_regex.so.1.77.0 (shared, Boost_USE_STATIC_LIBS=ON)

  * libboost_regex.a (release runtime, Boost_USE_DEBUG_RUNTIME=TRUE)

Call Stack (most recent call first):
  libVolepsi/lib/cmake/Boost-1.77.0/BoostConfig.cmake:258 (boost_find_component)
  /snap/cmake/1384/share/cmake-3.29/Modules/FindBoost.cmake:594 (find_package)
  libVolepsi/lib/cmake/coproto/coprotoFindDeps.cmake:146 (find_package)
  libVolepsi/lib/cmake/coproto/coprotoFindDeps.cmake:160 (FIND_BOOST)
  libVolepsi/lib/cmake/coproto/coprotoConfig.cmake:15 (include)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsDepHelper.cmake:215 (find_package)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsDepHelper.cmake:225 (FIND_COPROTO)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsConfig.cmake:26 (include)
  libVolepsi/lib/cmake/libOTe/libOTeConfig.cmake:12 (find_package)
  libVolepsi/lib/cmake/volePSI/findDependancies.cmake:104 (find_package)
  libVolepsi/lib/cmake/volePSI/findDependancies.cmake:118 (FIND_LIBOTE)
  libVolepsi/lib/cmake/volePSI/volePSIConfig.cmake:26 (include)
  frontend/CMakeLists.txt:30 (find_package)
ladnir commented 4 months ago

Hmm, so it's hard to say what exactly is the issue. Seems boost is being built without the regex library or at least without the correct settings.

It also seems you are using an older version of the library?

What project are you trying to build? It appears your including volepsi as a sub project?

Feir2 commented 4 months ago

Hmm, so it's hard to say what exactly is the issue. Seems boost is being built without the regex library or at least without the correct settings.

It also seems you are using an older version of the library?

What project are you trying to build? It appears your including volepsi as a sub project?

Thank you again for your reply. I want to use OPRF of volepsi in https://github.com/asu-crypto/mPSI. I build volepsi successfully and install it to /home/user/MyFile/mPSI5/mPSI/libVolepsi by python3 build.py --install=/home/user/MyFile/mPSI5/mPSI/libVolepsi. Then I link it to frontend.exe in the file frontend/CMakeLists.txt using the following code. I want to use OPRF of volepsi in frontend/tpsi.h.

find_package(volePSI REQUIRED HINTS "../libVolepsi")
target_link_libraries(frontend.exe visa::volePSI)

After that, I build the https://github.com/asu-crypto/mPSI and the above error occurs. https://github.com/asu-crypto/mPSI use boost 1.64, but volepsi use boost 1.77. I'm not sure if the error is related to the boost version. I have no idea how to solve it.

ladnir commented 4 months ago

Are you building both versions? Like does mpsi download and build its own boost?

Feir2 commented 4 months ago

Are you building both versions? Like does mpsi download and build its own boost?

Yes, I'm afraid this is what caused the error. I am considering if I can use only one boost. What can I do?

ladnir commented 4 months ago

Correct, you can not mix version. You'd need to upgrade to a common version. For example, make mpsi depend on the same version of libOTe, or maybe simpler, depend on volepsi.

Feir2 commented 4 months ago

Correct, you can not mix version. You'd need to upgrade to a common version. For example, make mpsi depend on the same version of libOTe, or maybe simpler, depend on volepsi.

Thank you very much for your help. I try that's way and make it 100%.

Feir2 commented 4 months ago

I build the project by python3 build.py -DVOLE_PSI_ENABLE_BOOST=ON.

I find the following code in many places in volepsi, but I'm not sure about the effect of ‘useSilver’ and ‘useQC’, and how to use it. Can you please tell me more about it?

#ifdef ENABLE_INSECURE_SILVER
        type = cmd.isSet("useSilver") ? oc::MultType::slv5 : type;
    #endif
#ifdef ENABLE_BITPOLYMUL
        type = cmd.isSet("useQC") ? oc::MultType::QuasiCyclic : type;
#endif

I find silver can improve performance in https://github.com/Visa-Research/volepsi/issues/10#issuecomment-1344552585. Then, I try the following tests by -useSilver or -useQC, but the efficiency doesn't seem to change. Why is that? If I want to improve performance, what can I do about it. I set the number of trials 5, I'm wondering why the first one took more time than the remaining four?

user@user-virtual-machine:~/MyFile/volepsi$ ./out/build/linux/frontend/frontend -perf -psi -malicious -v -t 5 -nt 1 -nn 15
nt 1 fakeBase 0 n 32768
Label     Time (ms)  diff (ms)
__________________________________
end            68.2     68.182  **********
begin          68.4      0.232  
end            90.3     21.888  *******
begin          90.3      0.002  
end           108.7     18.398  *******
begin         108.7      0.002  
end           127.0     18.288  *******
begin         127.0      0.002  
end           145.2     18.160  *******

7734281 3241345
user@user-virtual-machine:~/MyFile/volepsi$ ./out/build/linux/frontend/frontend -perf -psi -malicious -v -t 5 -nt 1 -nn 15 -useSilver
nt 1 fakeBase 0 n 32768
Label     Time (ms)  diff (ms)
__________________________________
end            68.9     68.922  **********
begin          69.2      0.235  
end            90.2     21.008  *******
begin          90.2      0.002  
end           109.2     19.072  *******
begin         109.2      0.004  
end           129.7     20.456  *******
begin         129.7      0.003  
end           148.2     18.527  *******

7734281 3241345
user@user-virtual-machine:~/MyFile/volepsi$ ./out/build/linux/frontend/frontend -perf -psi -malicious -v -t 5 -nt 1 -nn 15 -useQC
nt 1 fakeBase 0 n 32768
Label     Time (ms)  diff (ms)
__________________________________
end           108.5    108.527  **********
begin         108.8      0.230  
end           151.3     42.541  ********
begin         151.3      0.002  
end           190.7     39.370  ********
begin         190.7      0.002  
end           229.5     38.862  ********
begin         229.5      0.002  
end           268.5     38.932  ********

5373961 2940225

Another thing I want to confirm is whether volePSI uses OKVS in CCS 2022-Blazing Fast PSI from Improved OKVS and Subfield VOLE by default.

Looking forward to your reply.

ladnir commented 4 months ago

Silver is an insecure construction. By default, you can't use it even if you set useSilver. You have to compile the library with special settings to allow it to be used.

Not sure about useQC, but that should be slower than the default which is Expand Convolute codes, https://eprint.iacr.org/2023/882

So for all these vole based protocols, you first generate a sparse vector e, and then compress it by a random looking matric G by computing Ge. The question of which G should be used is what silver, QC, Expand Convolute answers. Each specifies a different G.

Slightly more concretely, the choice value in the vole correlation is the Ge value.

I'm away from my computer so cant look at the code right now. But the default version should be the fastest version. I think you should expend maybe a 4x slowdown compared to what's in the blazing fast psi paper. Not actually sure what the slowdown is, need to look into it to be sure. Basically the main cost will be to generate a vole of size 1.3n. So you should look at the paper above. You should expend psi times slightly slower than the time reported to just generate the vole.

Feir2 commented 4 months ago

Silver is an insecure construction. By default, you can't use it even if you set useSilver. You have to compile the library with special settings to allow it to be used.

Not sure about useQC, but that should be slower than the default which is Expand Convolute codes, https://eprint.iacr.org/2023/882

So for all these vole based protocols, you first generate a sparse vector e, and then compress it by a random looking matric G by computing Ge. The question of which G should be used is what silver, QC, Expand Convolute answers. Each specifies a different G.

Slightly more concretely, the choice value in the vole correlation is the Ge value.

I'm away from my computer so cant look at the code right now. But the default version should be the fastest version. I think you should expend maybe a 4x slowdown compared to what's in the blazing fast psi paper. Not actually sure what the slowdown is, need to look into it to be sure. Basically the main cost will be to generate a vole of size 1.3n. So you should look at the paper above. You should expend psi times slightly slower than the time reported to just generate the vole.

That's very useful, thank you very much for your reply.