search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.09k stars 4.32k forks source link

[ASAN] heap-buffer-overflow in FileBlob::read() #40407

Open makortel opened 1 year ago

makortel commented 1 year ago

CMSSW_13_0_ASAN_X_2022-12-26-2300 reports in workflow 138.5 step 3

==2966==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61e00026c274 at pc 0x2aca3fee68ce bp 0x2aca927197e0 sp 0x2aca92718f90
WRITE of size 3041 at 0x61e00026c274 thread T2
    #0 0x2aca3fee68cd in __interceptor_memcpy ../../../../libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:827
    #1 0x2aca432dab74  (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/external/el8_amd64_gcc11/lib/libz.so.1+0x6b74)
    #2 0x2aca432db54f in deflate (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/external/el8_amd64_gcc11/lib/libz.so.1+0x754f)
    #3 0x2aca432e4c34 in compress2 (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/external/el8_amd64_gcc11/lib/libz.so.1+0x10c34)
    #4 0x2aca6a92241c in FileBlob::read(std::istream&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/libCondFormatsCommon.so+0x6e41c)
    #5 0x2aca6a9234ee in FileBlob::read(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/libCondFormatsCommon.so+0x6f4ee)
    #6 0x2aca6a923893 in FileBlob::FileBlob(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/libCondFormatsCommon.so+0x6f893)
    #7 0x2acaa03f4a3e in MillePedeFileConverter::endLuminosityBlockProduce(edm::LuminosityBlock&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/pluginAlignmentMillePedeAlignmentAlgorithmAuto.so+0x184a3e)
    #8 0x2aca411daf16 in edm::one::EDProducerBase::doEndLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x95cf16)
    #9 0x2aca4115ae06 in edm::WorkerT<edm::one::EDProducerBase>::implDoEnd(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x8dce06)
    #10 0x2aca40c916d1 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::Context const*)::{lambda()#1}) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x4136d1)
    #11 0x2aca40c925b9 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::Context const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x4145b9)
    #12 0x2aca40c93003 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}::operator()() const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x415003)
    #13 0x2aca40cb09cc in void edm::SerialTaskQueueChain::actionToRun<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x4329cc)
    #14 0x2aca40cb0f14 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x432f14)
    #15 0x2aca41bf27c0 in tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/libFWCoreConcurrency.so+0x97c0)
    #16 0x2aca4327a303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-rc1-3cf8e8a4c68bb188a8b58034aba8291d/tbb-v2021.8.0-rc1/src/tbb/task_dispatcher.h:322
    #17 0x2aca4327a303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-rc1-3cf8e8a4c68bb188a8b58034aba8291d/tbb-v2021.8.0-rc1/src/tbb/task_dispatcher.h:458
    #18 0x2aca4327a303 in tbb::detail::r1::arena::process(tbb::detail::r1::thread_data&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-rc1-3cf8e8a4c68bb188a8b58034aba8291d/tbb-v2021.8.0-rc1/src/tbb/arena.cpp:137
    #19 0x2aca4327a303 in tbb::detail::r1::market::process(rml::job&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-rc1-3cf8e8a4c68bb188a8b58034aba8291d/tbb-v2021.8.0-rc1/src/tbb/market.cpp:599
    #20 0x2aca4327c4c5 in tbb::detail::r1::rml::private_worker::run() /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-rc1-3cf8e8a4c68bb188a8b58034aba8291d/tbb-v2021.8.0-rc1/src/tbb/private_server.cpp:271
    #21 0x2aca4327c4c5 in tbb::detail::r1::rml::private_worker::thread_routine(void*) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-rc1-3cf8e8a4c68bb188a8b58034aba8291d/tbb-v2021.8.0-rc1/src/tbb/private_server.cpp:221
    #22 0x2aca43f261ce in start_thread (/lib64/libpthread.so.0+0x81ce)
    #23 0x2aca44177e72 in clone (/lib64/libc.so.6+0x39e72)

0x61e00026c274 is located 0 bytes to the right of 2548-byte region [0x61e00026b880,0x61e00026c274)
allocated by thread T2 here:
    #0 0x2aca3ff5cf37 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x2aca4d27b48b in std::vector<unsigned char, std::allocator<unsigned char> >::reserve(unsigned long) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/libDQMServicesCore.so+0x5a48b)
    #2 0x2aca6a923888 in FileBlob::FileBlob(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/libCondFormatsCommon.so+0x6f888)
    #3 0x2acaa03f4a3e in MillePedeFileConverter::endLuminosityBlockProduce(edm::LuminosityBlock&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/pluginAlignmentMillePedeAlignmentAlgorithmAuto.so+0x184a3e)
    #4 0x2aca411daf16 in edm::one::EDProducerBase::doEndLuminosityBlock(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x95cf16)
    #5 0x2aca4115ae06 in edm::WorkerT<edm::one::EDProducerBase>::implDoEnd(edm::LumiTransitionInfo const&, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x8dce06)
    #6 0x2aca40c916d1 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >(edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::Context const*)::{lambda()#1}) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x4136d1)
    #7 0x2aca40c925b9 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3>::Context const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x4145b9)
    #8 0x2aca40c93003 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}::operator()() const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x415003)
    #9 0x2aca40cb09cc in void edm::SerialTaskQueueChain::actionToRun<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x4329cc)
    #10 0x2aca40cb0f14 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::LuminosityBlockPrincipal, (edm::BranchActionType)3> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so+0x432f14)
    #11 0x2aca41bf27c0 in tbb::detail::d1::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02765/el8_amd64_gcc11/cms/cmssw/CMSSW_13_0_ASAN_X_2022-12-26-2300/lib/el8_amd64_gcc11/libFWCoreConcurrency.so+0x97c0)
    #12 0x2aca4327a303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-rc1-3cf8e8a4c68bb188a8b58034aba8291d/tbb-v2021.8.0-rc1/src/tbb/task_dispatcher.h:322
    #13 0x2aca4327a303 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-rc1-3cf8e8a4c68bb188a8b58034aba8291d/tbb-v2021.8.0-rc1/src/tbb/task_dispatcher.h:458
    #14 0x2aca4327a303 in tbb::detail::r1::arena::process(tbb::detail::r1::thread_data&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-rc1-3cf8e8a4c68bb188a8b58034aba8291d/tbb-v2021.8.0-rc1/src/tbb/arena.cpp:137
    #15 0x2aca4327a303 in tbb::detail::r1::market::process(rml::job&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-rc1-3cf8e8a4c68bb188a8b58034aba8291d/tbb-v2021.8.0-rc1/src/tbb/market.cpp:599
    #16 0x2aca4327c4c5 in tbb::detail::r1::rml::private_worker::run() /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-rc1-3cf8e8a4c68bb188a8b58034aba8291d/tbb-v2021.8.0-rc1/src/tbb/private_server.cpp:271
    #17 0x2aca4327c4c5 in tbb::detail::r1::rml::private_worker::thread_routine(void*) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-rc1-3cf8e8a4c68bb188a8b58034aba8291d/tbb-v2021.8.0-rc1/src/tbb/private_server.cpp:221

Thread T2 created by T0 here:
    #0 0x2aca3ff03716 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cpp:216
    #1 0x2aca4327bb9a in tbb::detail::r1::rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-rc1-3cf8e8a4c68bb188a8b58034aba8291d/tbb-v2021.8.0-rc1/src/tbb/rml_thread_monitor.h:195
    #2 0x2aca4327bb9a in tbb::detail::r1::rml::private_worker::wake_or_launch() /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-rc1-3cf8e8a4c68bb188a8b58034aba8291d/tbb-v2021.8.0-rc1/src/tbb/private_server.cpp:305
    #3 0x2aca4327bb9a in tbb::detail::r1::rml::private_server::wake_some(int) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/external/tbb/v2021.8.0-rc1-3cf8e8a4c68bb188a8b58034aba8291d/tbb-v2021.8.0-rc1/src/tbb/private_server.cpp:412
    #4 0x60c0000ed0ff  (<unknown module>)

SUMMARY: AddressSanitizer: heap-buffer-overflow ../../../../libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:827 in __interceptor_memcpy
Shadow bytes around the buggy address:
  0x0c3c800457f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c3c80045800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c3c80045810: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c3c80045820: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c3c80045830: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c3c80045840: 00 00 00 00 00 00 00 00 00 00 00 00 00 00[04]fa
  0x0c3c80045850: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c3c80045860: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c3c80045870: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c3c80045880: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c3c80045890: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==2966==ABORTING

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc11/CMSSW_13_0_ASAN_X_2022-12-26-2300/pyRelValMatrixLogs/run/138.5_ExpressCollisions2021/step3_ExpressCollisions2021.log#/

cmsbuild commented 1 year ago

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel commented 1 year ago

assign alca,db

cmsbuild commented 1 year ago

New categories assigned: db,alca

@yuanchao,@ggovi,@francescobrivio,@francescobrivio,@malbouis,@malbouis,@saumyaphor4252,@saumyaphor4252,@tvami,@tvami,@ChrisMisan you have been requested to review this Pull request/Issue and eventually sign? Thanks

makortel commented 1 year ago

I think the problem is the following:

FileBlock constructor (MillePedeFileConverter::endLuminosityBlockProduce() calls with zip=true) reads the file once to figure out its size, and asks blob to reserve memory for as many elements https://github.com/cms-sw/cmssw/blob/038eb56268f2fb41e45e46828e556c174dc1f589/CondFormats/Common/src/FileBlob.cc#L11-L23

FileBlob::read() reads the file again (this time also storing the data) https://github.com/cms-sw/cmssw/blob/038eb56268f2fb41e45e46828e556c174dc1f589/CondFormats/Common/src/FileBlob.cc#L34-L38 then sets the size of the blob to the number of characters in the file, computes an upper bound for the compressed data size with computeBound(), and calls compress2() https://github.com/cms-sw/cmssw/blob/038eb56268f2fb41e45e46828e556c174dc1f589/CondFormats/Common/src/FileBlob.cc#L45-L50

I think what happens is that compressBound(isize) is larger than isize (which is allowed behavior mentioned in the documentation

This function may return a conservative value that may be larger than sourceLen.

), and that leads to a write beyond the allocated buffer.

makortel commented 1 year ago

https://github.com/cms-sw/cmssw/pull/40408 proposes a minimal fix.

I noticed the compression and uncompression failures are reported with LogError() (that, I suppose, end up being effectively ignored in production jobs?). I can't avoid wondering if this is sufficient, or if production use cases would benefit from exceptions instead.

makortel commented 1 year ago

@cms-sw/alca-l2 @cms-sw/db-l2 The fix https://github.com/cms-sw/cmssw/pull/40408 was merged. Do you want to keep this issue open for the other question I raised https://github.com/cms-sw/cmssw/issues/40407#issuecomment-1366008357 or should we close it?

tvami commented 1 year ago

hi @makortel thanks for the fix! Can you please point me to the LogError() msg you are referring to?

makortel commented 1 year ago

I'm wondering in particular https://github.com/cms-sw/cmssw/blob/a22313c07c6ccf65d78b792730fbf0f36db16d45/CondFormats/Common/src/FileBlob.cc#L47-L49 https://github.com/cms-sw/cmssw/blob/a22313c07c6ccf65d78b792730fbf0f36db16d45/CondFormats/Common/src/FileBlob.cc#L65-L68 https://github.com/cms-sw/cmssw/blob/a22313c07c6ccf65d78b792730fbf0f36db16d45/CondFormats/Common/src/FileBlob.cc#L81-L84 https://github.com/cms-sw/cmssw/blob/a22313c07c6ccf65d78b792730fbf0f36db16d45/CondFormats/Common/src/FileBlob.cc#L95-L98

(the other two in read() and computeFileSize() look less suspicious, although I could wonder also there what exactly ends up being happening in case of those errors)