search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.08k stars 4.3k forks source link

ASAN problem in CSCXonStrip_MatchGatti #32274

Closed makortel closed 3 years ago

makortel commented 3 years ago

Workflow 4.37 step 3 fails in CMSSW_11_2_ASAN_X_2020-11-23-2300 with

=================================================================
==22914==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62d0003e7470 at pc 0x2addaee30467 bp 0x2add78fd84c0 sp 0x2add78fd84b8
    #0 0x2addaee30466 in CSCXonStrip_MatchGatti::estimated2GattiCorrection(double, float, bool) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/pluginRecoLocalMuonCSCRecHitD.so+0xc4466)
    #1 0x2addaee30495 in CSCXonStrip_MatchGatti::estimated2Gatti(double, float, bool) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/pluginRecoLocalMuonCSCRecHitD.so+0xc4495)
    #2 0x2addaee35ad0 in CSCXonStrip_MatchGatti::findXOnStrip(CSCDetId const&, CSCLayer const*, CSCStripHit const&, int, float&, float&, float const&, float&, float&, int&) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/pluginRecoLocalMuonCSCRecHitD.so+0xc9ad0)
    #3 0x2addaedc7839 in CSCMake2DRecHit::hitFromStripAndWire(CSCDetId const&, CSCLayer const*, CSCWireHit const&, CSCStripHit const&) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/pluginRecoLocalMuonCSCRecHitD.so+0x5b839)
    #4 0x2addaedcd66d in CSCRecHitDBuilder::build(MuonDigiCollection<CSCDetId, CSCStripDigi> const*, MuonDigiCollection<CSCDetId, CSCWireDigi> const*, edm::RangeMap<CSCDetId, edm::OwnVector<CSCRecHit2D, edm::ClonePolicy<CSCRecHit2D> >, edm::ClonePolicy<CSCRecHit2D> >&) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/pluginRecoLocalMuonCSCRecHitD.so+0x6166d)
    #5 0x2addaedd8958 in CSCRecHitDProducer::produce(edm::Event&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/pluginRecoLocalMuonCSCRecHitD.so+0x6c958)
    #6 0x2add34838c4b in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x8a3c4b)
    #7 0x2add34779171 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x7e4171)
    #8 0x2add34492f19 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x4fdf19)
    #9 0x2add3449341b in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x4fe41b)
    #10 0x2add34493dff in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x4fedff)
    #11 0x2add3449b097 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x506097)
    #12 0x2add36868bfc in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop(tbb::internal::context_guard_helper<false>&, tbb::task*, long) ../../src/tbb/custom_scheduler.h:474
    #13 0x2add36868ef4 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all(tbb::task&, tbb::task*) ../../src/tbb/custom_scheduler.h:636
    #14 0x2add368629fe in tbb::internal::arena::process(tbb::internal::generic_scheduler&) ../../src/tbb/arena.cpp:196
    #15 0x2add368613d2 in tbb::internal::market::process(rml::job&) ../../src/tbb/market.cpp:667
    #16 0x2add3685d7db in tbb::internal::rml::private_worker::run() ../../src/tbb/private_server.cpp:266
    #17 0x2add3685d9e8 in tbb::internal::rml::private_worker::thread_routine(void*) ../../src/tbb/private_server.cpp:219
    #18 0x2add3761cea4 in start_thread (/lib64/libpthread.so.0+0x7ea4)
    #19 0x2add3792f96c in clone (/lib64/libc.so.6+0xfe96c)

0x62d0003e7470 is located 376 bytes to the right of 36600-byte region [0x62d0003de400,0x62d0003e72f8)
allocated by thread T0 here:
    #0 0x2add332b9db0 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cc:90
    #1 0x2addaedc2957 in CSCMake2DRecHit::CSCMake2DRecHit(edm::ParameterSet const&) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/pluginRecoLocalMuonCSCRecHitD.so+0x56957)
    #2 0x2addaedcace0 in CSCRecHitDBuilder::CSCRecHitDBuilder(edm::ParameterSet const&) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/pluginRecoLocalMuonCSCRecHitD.so+0x5ece0)
    #3 0x2addaedd6122 in CSCRecHitDProducer::CSCRecHitDProducer(edm::ParameterSet const&) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/pluginRecoLocalMuonCSCRecHitD.so+0x6a122)
    #4 0x2addaedf21b1 in edm::stream::ProducingModuleAdaptor<CSCRecHitDProducer, edm::stream::EDProducerBase, edm::stream::EDProducerAdaptorBase>::setupStreamModules() (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/pluginRecoLocalMuonCSCRecHitD.so+0x861b1)
    #5 0x2add3483ed1e in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doPreallocate(edm::PreallocationConfiguration const&) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x8a9d1e)
    #6 0x2add34749e8e in edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x7b4e8e)
    #7 0x2add343e1166 in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x44c166)
    #8 0x2add3443a354 in edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x4a5354)
    #9 0x2add34754b08 in edm::WorkerRegistry::getWorker(edm::WorkerParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x7bfb08)
    #10 0x2add3474be4c in edm::WorkerManager::getWorker(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x7b6e4c)
    #11 0x2add347501e4 in edm::WorkerManager::addToUnscheduledWorkers(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x7bb1e4)
    #12 0x2add346a4f93 in edm::StreamSchedule::StreamSchedule(std::shared_ptr<edm::TriggerResultInserter>, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> > > >&, std::shared_ptr<edm::ModuleRegistry>, edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProductRegistry&, edm::BranchIDListHelper&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration>, bool, edm::StreamID, edm::ProcessContext const*) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x70ff93)
    #13 0x2add345ff7f0 in edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::ProductRegistry&, edm::BranchIDListHelper&, edm::ThinnedAssociationsHelper&, edm::SubProcessParentageHelper const*, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration>, bool, edm::PreallocationConfiguration const&, edm::ProcessContext const*) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x66a7f0)
    #14 0x2add34658dcf in edm::ScheduleItems::initSchedule(edm::ParameterSet&, bool, edm::PreallocationConfiguration const&, edm::ProcessContext const*) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x6c3dcf)
    #15 0x2add34246a56 in edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x2b1a56)
    #16 0x2add3424f598 in edm::EventProcessor::EventProcessor(std::shared_ptr<edm::ProcessDesc>, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x2ba598)
    #17 0x411f93 in tbb::interface7::internal::delegated_function<main::{lambda()#1}::operator()() const::{lambda()#1} const, void>::operator()() const (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/bin/slc7_amd64_gcc820/cmsRun+0x411f93)
    #18 0x2add36863bc0 in tbb::interface7::internal::task_arena_base::internal_execute(tbb::interface7::internal::delegate_base&) const ../../src/tbb/arena.cpp:1105
    #19 0x2add3c5cfeff  (<unknown module>)

Thread T2 created by T0 here:
    #0 0x2add3321fdc0 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cc:202
    #1 0x2add3685d6c9 in rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) ../../src/tbb/../rml/server/thread_monitor.h:218
    #2 0x2add3685d6c9 in tbb::internal::rml::private_worker::wake_or_launch() ../../src/tbb/private_server.cpp:297
    #3 0x2add3685d6c9 in tbb::internal::rml::private_server::wake_some(int) ../../src/tbb/private_server.cpp:395
    #4 0x60c00042a47f  (<unknown module>)

SUMMARY: AddressSanitizer: heap-buffer-overflow (/cvmfs/cms-ib.cern.ch/nweek-02656/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-23-2300/lib/slc7_amd64_gcc820/pluginRecoLocalMuonCSCRecHitD.so+0xc4466) in CSCXonStrip_MatchGatti::estimated2GattiCorrection(double, float, bool)
Shadow bytes around the buggy address:
  0x0c5a80074e30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80074e40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80074e50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fa
  0x0c5a80074e60: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5a80074e70: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c5a80074e80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa[fa]fa
  0x0c5a80074e90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5a80074ea0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5a80074eb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5a80074ec0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5a80074ed0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==22914==ABORTING
makortel commented 3 years ago

assign reconstruction

cmsbuild commented 3 years ago

New categories assigned: reconstruction

@slava77,@perrotta,@jpata you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild commented 3 years ago

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @dpiparo, @silviodonato, @smuzaffar, @makortel, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel commented 3 years ago

From the set of PRs merged since the previous ASAN builds (that did not have this error) https://github.com/cms-sw/cmssw/pull/32052 could be the cause (the only one with "CSC" in its name, and it touches code in the same package.

slava77 commented 3 years ago

@nvoytish please take a look to reproduce you will need to setup CMSSW_11_2_ASAN_X_2020-11-23-2300 IB and then there runTheMatrix.py -l 4.37 --ibeos

slava77 commented 3 years ago

@nvoytish please take a look to reproduce you will need to setup CMSSW_11_2_ASAN_X_2020-11-23-2300 IB and then there runTheMatrix.py -l 4.37 --ibeos

@ptcox

ptcox commented 3 years ago

Can you clarify what we are looking for? Does this mean some dynamically assigned memory has been overwritten, or can it mean some simpler thing like a misaligned variable? I do not know what ASAN tests exactly. But the issue might be easier to track down if it could just be some value passed in to findXOnStrip is invalid. (That would seem more likely since the recent updates to the code can affect those.)

makortel commented 3 years ago

The complaint is about a read (or write) beyond an allocated memory block. After a quick peek into the code I'd guess one of these array accesses goes out of bounds https://github.com/cms-sw/cmssw/blob/dd492f74e6c2e9a9dadaa5990245803ec0e76d45/RecoLocalMuon/CSCRecHitD/src/CSCXonStrip_MatchGatti.cc#L549-L570

So the problem could well be

if it could just be some value passed in to findXOnStrip is invalid.

slava77 commented 3 years ago

@ptcox @nvoytish please clarify on the status of resolving this. we should avoid an option of having to revert the last CSC changes.

nvoytish commented 3 years ago

Hi @slava77 ! Unfortunately I was too busy last week. I am on it.

nvoytish commented 3 years ago

@ptcox I managed to find what causes the problem. In a particular event the strip from one chamber (ME11A) is combined with a wire that it physically can not intersect (last wg from ME11B). This causes a mess in Gatti correction calculation. I'll try to understand which change in code lead to this. Sorry for the slow progress!

ptcox commented 3 years ago

That’s good progress! And something like I was expecting. Maybe something I missed in the intrinsic traps in the new code, and we still need some sort of ‘inside’ check.

On Dec 2, 2020, at 11:31, nvoytish notifications@github.com wrote:

 @ptcox I managed to find what causes the problem. In a particular event the strip from one chamber (ME11A) is combined with a wire that it physically can not intersect (last wg from ME11B). This causes a mess in Gatti correction calculation. I'll try to understand which change in code lead to this. Sorry for the slow progress!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

nvoytish commented 3 years ago

I'll keep digging. You were right, when you said that Gatti corrections are "a pain")

nvoytish commented 3 years ago

Hi all!

We are trying to come up with a right fix for the bug. I was working on it today, but couple hours ago after a successful compile I tried to run "cmsenv" command and got: [nvoytish@lxplus723 src]$ cmsenv Unable to find SCRAM version V2_2_9_pre11. Something wrong with the sw installation.

and [nvoytish@lxplus723 ~]$ scram list CMSSW_11_2_ASAN* SCRAM warning: >>>> No SCRAM project CMSSW version CMSSW_11_2_ASAN_X_2020-11-23-2300 available. <<<< You can run "scram list CMSSW" to see the available versions. gives no results. Was this release temporary and is not available any more? Is there another one that will reproduce the failure?

slava77 commented 3 years ago

Was this release temporary and is not available any more? Is there another one that will reproduce the failure?

IBs stay for about 2 weeks; it looks like 11-23 has expired. CMSSW_11_3_ASAN_X_2020-12-04-2300 is a more recent build The same problem in CSC is still there.

nvoytish commented 3 years ago

Hi all!

It turned out that we need to keep the omitted "if" https://github.com/cms-sw/cmssw/commit/7501f0ca9d85f3f5f2e6f20ba2743b1794a48e24#diff-89c0bedb1d5ff71760f4a99102f4c06386cdac66bb8527527d41a7fe28e5dab8L160 We forgot that there is a difference in detid for wires in simulation and real data. For real data all wires are labelled as ME11B. This "if" does not allow for a strip from ME11A to be assembled with a wire that geometrically is outside the ME11A sensitive area. Do I understand correctly, that I need to make a new PR for this change?

makortel commented 3 years ago

Do I understand correctly, that I need to make a new PR for this change?

Yes, thanks!

slava77 commented 3 years ago

+1

I do not find the CSC-related error in https://cmssdt.cern.ch/SDT/html/cmssdt-ib/#/relVal/CMSSW_11_3/2020-12-16-2300?selectedArchs=slc7_amd64_gcc900&selectedFlavors=ASAN_X&selectedStatus=failed anymore (after #32442 was merged on Dec 15)

cmsbuild commented 3 years ago

This issue is fully signed and ready to be closed.