herumi / mcl

a portable and fast pairing-based cryptography library
BSD 3-Clause "New" or "Revised" License
450 stars 151 forks source link

Can't build with `-fsanitize=memory` #167

Closed aguycalled closed 1 year ago

aguycalled commented 1 year ago

Related to #163

We are seeing the following error when building with -fsanitize=memory:

==65875==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x311dad1 in bool mcl::fp::isEqualArray<unsigned long>(unsigned long const*, unsigned long const*, unsigned long) /tmp/cirrus-ci-build/ci/scratch/build/navcoin-x86_64-pc-linux-gnu/src/bls/mcl/include/mcl/util.hpp:122:7
    #1 0x311dad1 in mcl::FpT<mcl::bn::local::FpTag, 384ul>::operator==(mcl::FpT<mcl::bn::local::FpTag, 384ul> const&) const /tmp/cirrus-ci-build/ci/scratch/build/navcoin-x86_64-pc-linux-gnu/src/bls/mcl/include/mcl/fp.hpp:688:49
    #2 0x311dad1 in mcl::Fp2T<mcl::FpT<mcl::bn::local::FpTag, 384ul> >::operator==(mcl::Fp2T<mcl::FpT<mcl::bn::local::FpTag, 384ul> > const&) const /tmp/cirrus-ci-build/ci/scratch/build/navcoin-x86_64-pc-linux-gnu/src/bls/mcl/include/mcl/fp_tower.hpp:355:52
    #3 0x311dad1 in mcl::bn::local::Param::init(bool*, mcl::CurveParam const&, mcl::fp::Mode) /tmp/cirrus-ci-build/ci/scratch/build/navcoin-x86_64-pc-linux-gnu/src/bls/mcl/include/mcl/bn.hpp:916:15
    #4 0x30caa01 in mcl::bn::BN::init(bool*, mcl::CurveParam const&, mcl::fp::Mode) /tmp/cirrus-ci-build/ci/scratch/build/navcoin-x86_64-pc-linux-gnu/src/bls/mcl/include/mcl/bn.hpp:2268:20
    #5 0x30caa01 in mcl::bn::initPairing(bool*, mcl::CurveParam const&, mcl::fp::Mode) /tmp/cirrus-ci-build/ci/scratch/build/navcoin-x86_64-pc-linux-gnu/src/bls/mcl/include/mcl/bn.hpp:2297:2
    #6 0x30bbc8f in mclBn_init /tmp/cirrus-ci-build/ci/scratch/build/navcoin-x86_64-pc-linux-gnu/src/bls/mcl/include/mcl/impl/bn_c_impl.hpp:83:2
    #7 0x19c5393 in MclInitializer::Init() src/blsct/arith/mcl_initializer.cpp:13:9
    #8 0x19e7e58 in MclTestingSetup::MclTestingSetup() src/test/util/setup_common.cpp:451:5
    #9 0xa5f268 in elements_tests::test_elements_mulvec_elements::test_elements_mulvec_elements() src/test/blsct/arith/elements_tests.cpp:469:1
    #10 0xa54418 in elements_tests::test_elements_mulvec_elements_invoker() src/test/blsct/arith/elements_tests.cpp:469:1
    #11 0x84f133 in boost::detail::function::void_function_invoker0<void (*)(), void>::invoke(boost::detail::function::function_buffer&) /tmp/cirrus-ci-build/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:117:11
    #12 0x6377e5 in boost::function0<void>::operator()() const /tmp/cirrus-ci-build/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:763:14
    #13 0x726d37 in boost::detail::forward::operator()() /tmp/cirrus-ci-build/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1388:32
    #14 0x726836 in boost::detail::function::function_obj_invoker0<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) /tmp/cirrus-ci-build/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:137:18
    #15 0x71c5ad in boost::function0<int>::operator()() const /tmp/cirrus-ci-build/depends/x86_64-pc-linux-gnu/include/boost/function/function_template.hpp:763:14
    #16 0x5e02a3 in int boost::detail::do_invoke<boost::shared_ptr<boost::detail::translator_holder_base>, boost::function<int ()> >(boost::shared_ptr<boost::detail::translator_holder_base> const&, boost::function<int ()> const&) /tmp/cirrus-ci-build/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:301:30
    #17 0x567c2a in boost::execution_monitor::catch_signals(boost::function<int ()> const&) /tmp/cirrus-ci-build/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:903:16
    #18 0x567fa6 in boost::execution_monitor::execute(boost::function<int ()> const&) /tmp/cirrus-ci-build/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1301:16
    #19 0x55dd48 in boost::execution_monitor::vexecute(boost::function<void ()> const&) /tmp/cirrus-ci-build/depends/x86_64-pc-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1397:5
    #20 0x5627d3 in boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned long) /tmp/cirrus-ci-build/depends/x86_64-pc-linux-gnu/include/boost/test/impl/unit_test_monitor.ipp:49:9
    #21 0x5dc848 in boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /tmp/cirrus-ci-build/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:815:44
    #22 0x5daf25 in boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /tmp/cirrus-ci-build/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:784:58
    #23 0x5daf25 in boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /tmp/cirrus-ci-build/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:784:58
    #24 0x560db7 in boost::unit_test::framework::run(unsigned long, bool) /tmp/cirrus-ci-build/depends/x86_64-pc-linux-gnu/include/boost/test/impl/framework.ipp:1722:29
    #25 0x587c07 in boost::unit_test::unit_test_main(boost::unit_test::test_suite* (*)(int, char**), int, char**) /tmp/cirrus-ci-build/depends/x86_64-pc-linux-gnu/include/boost/test/impl/unit_test_main.ipp:250:9
    #26 0x58851e in main /tmp/cirrus-ci-build/depends/x86_64-pc-linux-gnu/include/boost/test/impl/unit_test_main.ipp:306:12
    #27 0x7ff2cefa3082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082)
    #28 0x4c8e2d in _start (/tmp/cirrus-ci-build/ci/scratch/build/navcoin-x86_64-pc-linux-gnu/src/test/test_navcoin+0x4c8e2d)
  Uninitialized value was stored to memory at
    #0 0x33ff37c in void mcl::fp::copyArray<unsigned long>(unsigned long*, unsigned long const*, unsigned long) /tmp/cirrus-ci-build/ci/scratch/build/navcoin-x86_64-pc-linux-gnu/src/bls/mcl/include/mcl/util.hpp:145:38
    #1 0x33ff37c in void mcl::fp::copyC<6ul>(unsigned long*, unsigned long const*) /tmp/cirrus-ci-build/ci/scratch/build/navcoin-x86_64-pc-linux-gnu/src/bls/mcl/src/low_func.hpp:58:2
  Uninitialized value was created by an allocation of 'ref.tmp125' in the stack frame of function '_ZN3mcl2bn5local5Param4initEPbRKNS_10CurveParamENS_2fp4ModeE'
    #0 0x3117850 in mcl::bn::local::Param::init(bool*, mcl::CurveParam const&, mcl::fp::Mode) /tmp/cirrus-ci-build/ci/scratch/build/navcoin-x86_64-pc-linux-gnu/src/bls/mcl/include/mcl/bn.hpp:860
SUMMARY: MemorySanitizer: use-of-uninitialized-value /tmp/cirrus-ci-build/ci/scratch/build/navcoin-x86_64-pc-linux-gnu/src/bls/mcl/include/mcl/util.hpp:122:7 in bool mcl::fp::isEqualArray<unsigned long>(unsigned long const*, unsigned long const*, unsigned long)

Compiling with the following flags:

/usr/bin/ccache clang++ -std=c++17 -pipe -std=c++17 -O2 -fsanitize=memory -fsanitize-memory-track-origins=2 -fno-omit-frame-pointer -g -O1 -fno-optimize-sibling-calls -nostdinc++ -stdlib=libc++ -L/tmp/cirrus-ci-build/ci/scratch/msan/build/lib -lc++abi -I/tmp/cirrus-ci-build/ci/scratch/msan/build/include -I/tmp/cirrus-ci-build/ci/scratch/msan/build/include/c++/v1 -lpthread -Wl,-rpath,/tmp/cirrus-ci-build/ci/scratch/msan/build/lib -Wno-unused-command-line-argument -g3 -Wall -Wextra -Wformat=2 -Wcast-qual -Wcast-align -Wwrite-strings -Wfloat-equal -Wpointer-arith -Wundef -m64 -I include -I test -fomit-frame-pointer -DNDEBUG -fno-stack-protector -O3  -DMCL_USE_VINT -DMCL_DONT_USE_OPENSSL -fPIC -std=c++11 -I/tmp/cirrus-ci-build/ci/scratch/build/navcoin-x86_64-pc-linux-gnu/src/bls/mcl/include -c src/bls_c384_256.cpp -o obj/bls_c384_256.o -MMD -MP -MF obj/bls_c384_256.d
make[3]: Entering directory '/tmp/cirrus-ci-build/ci/scratch/build/navcoin-x86_64-pc-linux-gnu/src/bls/mcl'
/usr/bin/ccache clang++ -std=c++17 -pipe -std=c++17 -O2 -fsanitize=memory -fsanitize-memory-track-origins=2 -fno-omit-frame-pointer -g -O1 -fno-optimize-sibling-calls -nostdinc++ -stdlib=libc++ -L/tmp/cirrus-ci-build/ci/scratch/msan/build/lib -lc++abi -I/tmp/cirrus-ci-build/ci/scratch/msan/build/include -I/tmp/cirrus-ci-build/ci/scratch/msan/build/include/c++/v1 -lpthread -Wl,-rpath,/tmp/cirrus-ci-build/ci/scratch/msan/build/lib -Wno-unused-command-line-argument -g3 -Wall -Wextra -Wformat=2 -Wcast-qual -Wcast-align -Wwrite-strings -Wfloat-equal -Wpointer-arith -Wundef -m64 -I include -I test -fomit-frame-pointer -DNDEBUG -fno-stack-protector -O3  -DMCL_USE_VINT -DMCL_DONT_USE_OPENSSL -fPIC -c src/fp.cpp -o obj/fp.o -MMD -MP -MF obj/fp.d

Adding -DMCL_MAX_BIT_SIZE=384 does not fix the issue.

mcl_initializer.cpp:

#include <blsct/arith/mcl_initializer.h>

void MclInitializer::Init()
{
    boost::lock_guard<boost::mutex> lock(MclInitializer::m_init_mutex);
    static bool is_initialized = false;
    if (is_initialized) return;

    if (mclBn_init(MCL_BLS12_381, MCLBN_COMPILED_TIME_VAR) != 0) {
        throw std::runtime_error("blsInit failed");
    }
    mclBn_setETHserialization(1);

    is_initialized = true;
}

mcl_initializer.h:

#include <mcl/bn_c384_256.h>
#include <boost/thread/lock_guard.hpp>
#include <boost/thread/mutex.hpp>

class MclInitializer
{
public:
    static void Init();

private:
    inline static boost::mutex m_init_mutex;
};
herumi commented 1 year ago

Thank you for the report. I'll investigate the error. I already found two false positive errors of STL during the problem isolation. https://github.com/herumi/misc/tree/main/sanitize

aguycalled commented 1 year ago

thanks! :)

herumi commented 1 year ago

I found another false positive of memory sanitizer. https://github.com/herumi/misc/blob/main/sanitize/strlen_err.cpp

#include <string>
#include <stdio.h>
#include <string.h>

int main()
{
  std::string s = "ABC";
  printf("len=%zd\n", strlen(s.c_str()));
}

These trivial three false positives are used in mcl. They interfere with the investigation. valgrind does not anything errors.

herumi commented 1 year ago

I found that the memory sanitizer reports the following code, which is the same reason reported by https://github.com/herumi/mcl/issues/167#issue-1465448040

// test/t.cpp
include <mcl/bint.hpp>

int main()
{
    using namespace mcl;
    const size_t N = 2;
    const Unit x[N] = { 1, 2 };
    Unit buf[N * 2];
    bint::mulT<N>(buf, x, x);
    for (size_t i = 0; i < N * 2; i++) {
        printf("buf[%zd]=%lx\n", i, buf[i]);
        if (buf[i]) printf("!!\n");  // MSAN error
    }
}
% make lib/libmcl.a -j DEBUG=2 bin/t.exe CXX=clang++-14 && bin/t.exe
buf[0]=1
==2718937==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x55abdbe26e91 in main mcl/test/t.cpp:12:7
    #1 0x7f3ea7a4bd8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #2 0x7f3ea7a4be3f in __libc_start_main csu/../csu/libc-start.c:392:3
    #3 0x55abdbd9e904 in _start (mcl/bin/t.exe+0x23904) (BuildId: 9408e7ff3de3ab88a19d433a513ea791c2b74903)

  Uninitialized value was created by an allocation of 'buf' in the stack frame of function 'main'
    #0 0x55abdbe26a00 in main mcl/test/t.cpp:4

mulT<N>(z, x, y) takes x[N] and y[N] and outputs z[N * 2] as the result of x[N] * y[N]. But the function is written by the assembler, so the memory sanitizer does not know it and reports a false positive.

If MCL_BINT_ASM=0 then mcl does not use the asm functions, and

% make lib/libmcl.a -j bin/t.exe CXX=clang++-14 DEBUG=2 MCL_BINT_ASM=0 && bin/t.exe

does not show the error. The correctness of the asm code is confirmed by test/bint_test.cpp.