search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.08k stars 4.31k forks source link

Regression of ASAN build #35513

Open VinInn opened 3 years ago

VinInn commented 3 years ago

I have compared running a hlt menu on 1000 events in /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-09-29-2300 vs /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_X_2021-09-29-2300

and results differs

for instance error found are different

ASAN

Severity    # Occurrences   Total Occurrences
--------    -------------   -----------------
Info               174344              174344
FwkInfo            303077              303077
Warning               977                 977
Error                  50                  50
System                  4                   4

Normal build

Severity    # Occurrences   Total Occurrences
--------    -------------   -----------------
Info               175900              175900
FwkInfo            303077              303077
Warning               568                 568
Error                  31                  31
System                  4                   4

and hlt results as well

hltDiff -o output_ori.root -n output_asan.root
....
Found 1000 matching events, out of which 11 have different HLT results

      Events    Accepted      Gained        Lost       Other  Trigger
        1000         141           -          -1           -  HLT_Ele20_WPTight_Gsf_v6
        1000         156           -          -1           -  HLT_Ele15_WPLoose_Gsf_v3
        1000         149           -          -1           -  HLT_Ele17_WPLoose_Gsf_v3
        1000         143           -          -1           -  HLT_Ele20_WPLoose_Gsf_v6
        1000         122           -          -1           -  HLT_Ele27_WPTight_Gsf_v16
        1000         121           -          -1           -  HLT_Ele28_WPTight_Gsf_v1
        1000         113           -          -1           -  HLT_Ele30_WPTight_Gsf_v1
        1000         108           -          -1           -  HLT_Ele32_WPTight_Gsf_v15
        1000         104           -          -1           -  HLT_Ele35_WPTight_Gsf_v9
        1000          90           -          -1           -  HLT_Ele38_WPTight_Gsf_v9
        1000         109           -          -1           -  HLT_Ele32_WPTight_Gsf_L1DoubleEG_v9
        1000         133           -          -1           -  HLT_DiPFJetAve60_HFJEC_v15
        1000          48           -           -          ~1  HLT_PFMET110_PFMHT110_IDTight_v20
        1000          36           -           -          ~1  HLT_PFMET110_PFMHT110_IDTight_CaloBTagDeepCSV_3p1_v8
        1000          51           -           -          ~1  HLT_PFMETTypeOne110_PFMHT110_IDTight_v12
        1000          71           -           -          ~1  HLT_PFMETNoMu110_PFMHTNoMu110_IDTight_v20
        1000           7           -           -          ~1  HLT_Mu12_DoublePFJets40MaxDeta1p6_DoubleCaloBTagDeepCSV_p71_v2
        1000           6           -           -          ~1  HLT_Mu12_DoublePFJets54MaxDeta1p6_DoubleCaloBTagDeepCSV_p71_v2
        1000           5           -           -          ~1  HLT_Mu12_DoublePFJets62MaxDeta1p6_DoubleCaloBTagDeepCSV_p71_v2
        1000          19           -           -          ~2  HLT_DoublePFJets116MaxDeta1p6_DoubleCaloBTagDeepCSV_p71_v2
        1000          14           -           -          ~2  HLT_DoublePFJets128MaxDeta1p6_DoubleCaloBTagDeepCSV_p71_v2
        1000          12           -           -          ~1  HLT_Ele15_Ele8_CaloIdL_TrackIdL_IsoVL_v3
        1000          12           -           -          ~1  HLT_DiJet110_35_Mjj650_PFMET120_v9
        1000           1           -           -          ~1  HLT_TripleJet110_35_35_Mjj650_PFMET120_v9
        1000          19           -           -          ~1  HLT_Ele15_IsoVVVL_PFHT450_CaloBTagDeepCSV_4p5_v8
        1000          12           -           -          ~1  HLT_Ele15_IsoVVVL_PFHT450_PFMET50_v16
        1000          20           -           -          ~1  HLT_Ele15_IsoVVVL_PFHT450_v16
        1000           7           -           -          ~1  HLT_Ele15_IsoVVVL_PFHT600_v20
        1000         193          +1           -           -  HLT_Ele8_CaloIdL_TrackIdL_IsoVL_PFJet30_v16
        1000         183          +1           -           -  HLT_Ele12_CaloIdL_TrackIdL_IsoVL_PFJet30_v18
        1000         178          +1           -           -  HLT_Ele15_CaloIdL_TrackIdL_IsoVL_PFJet30_v3
        1000         822          +1           -           -  MC_PFBTagDeepCSV_v10
        1000         157           -          -1           -  MC_Ele5_WPTight_Gsf_v8
        1000          10           -           -          ~1  MC_Ele15_Ele10_CaloIdL_TrackIdL_IsoVL_DZ_v15
        1000          16           -           -          ~1  HLT_DoubleMediumChargedIsoPFTau35_Trk1_eta2p1_Reg_v12
        1000          15           -           -          ~1  HLT_DoubleMediumChargedIsoPFTau35_Trk1_TightID_eta2p1_Reg_v12
        1000          12           -           -          ~1  HLT_DoubleTightChargedIsoPFTau35_Trk1_eta2p1_Reg_v12
        1000          11           -           -          ~1  HLT_DoubleTightChargedIsoPFTau35_Trk1_TightID_eta2p1_Reg_v12
        1000           8           -           -          ~1  HLT_DoubleTightChargedIsoPFTauHPS35_Trk1_eta2p1_Reg_v1
        1000           7           -           -          ~1  HLT_DoubleMediumChargedIsoPFTauHPS35_Trk1_TightID_eta2p1_Reg_v1
        1000          10           -           -          ~1  HLT_DoubleMediumChargedIsoPFTauHPS35_Trk1_eta2p1_Reg_v4
        1000           5           -           -          ~1  HLT_DoubleTightChargedIsoPFTauHPS35_Trk1_TightID_eta2p1_Reg_v1
        1000           5           -           -          ~1  HLT_DoubleMediumChargedIsoPFTauHPS40_Trk1_eta2p1_Reg_v1
        1000           4           -           -          ~1  HLT_DoubleTightChargedIsoPFTauHPS40_Trk1_eta2p1_Reg_v1
        1000           3           -           -          ~1  HLT_DoubleMediumChargedIsoPFTauHPS40_Trk1_TightID_eta2p1_Reg_v1
        1000           2           -           -          ~1  HLT_DoubleTightChargedIsoPFTauHPS40_Trk1_TightID_eta2p1_Reg_v1
        1000          25           -          -1           -  HLT_QuadPFJet98_83_71_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8
        1000          20           -          -1           -  HLT_QuadPFJet103_88_75_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8
        1000          17           -          -1           -  HLT_QuadPFJet111_90_80_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8
        1000          20           -          -1           -  HLT_QuadPFJet105_88_76_15_DoublePFBTagDeepCSV_1p3_7p7_VBF1_v8

I do not think this was expected

cmsbuild commented 3 years ago

A new Issue was created by @VinInn Vincenzo Innocente.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel commented 3 years ago

assign core,hlt,reconstruction

cmsbuild commented 3 years ago

New categories assigned: core,hlt,reconstruction

@Dr15Jones,@smuzaffar,@slava77,@Martin-Grunewald,@makortel,@jpata,@missirol you have been requested to review this Pull request/Issue and eventually sign? Thanks

makortel commented 3 years ago

I'm not sure if ASAN can impact (e.g. the order of) floating point computations, or if this should be interpreted as a symptom of some code reading from incorrect memory locations (that are still technically valid so ASAN doesn't catch those).

VinInn commented 3 years ago

Maybe we should check offline reco: just to make sure is not a HLT only feature....

VinInn commented 3 years ago

I run reco on 200 events. Reco itself as much less errors than HLT (only few events affected): but is different.

[innocent@patatrack02 asan]$ grep MSG-w asanReco.log | grep -v MemoryCheck
%MSG-w XrdAdaptorInternal:  file_open 04-Oct-2021 15:14:41 CEST pre-events
%MSG-w XrdAdaptorInternal:  file_open 04-Oct-2021 15:14:41 CEST pre-events
%MSG-w XrdAdaptor:  file_open 04-Oct-2021 15:14:43 CEST pre-events
%MSG-w BasicTrajectoryState:  TrackProducer:jetCoreRegionalStepTracks  04-Oct-2021 15:30:16 CEST Run: 1 Event: 1662
%MSG-w BasicTrajectoryState:  TrackProducer:jetCoreRegionalStepTracks  04-Oct-2021 15:30:16 CEST Run: 1 Event: 1662
%MSG-w BasicTrajectoryState:  TrackProducer:jetCoreRegionalStepTracks  04-Oct-2021 15:30:16 CEST Run: 1 Event: 1662
%MSG-w BasicTrajectoryState:  TrackProducer:jetCoreRegionalStepTracks  04-Oct-2021 15:30:16 CEST Run: 1 Event: 1662
%MSG-w BasicTrajectoryState:  TrackProducer:jetCoreRegionalStepTracks  04-Oct-2021 15:30:16 CEST Run: 1 Event: 1662
%MSG-w BasicTrajectoryState:  TrackProducer:jetCoreRegionalStepTracks  04-Oct-2021 15:30:16 CEST Run: 1 Event: 1662
%MSG-w BasicTrajectoryState:  TrackProducer:jetCoreRegionalStepTracks  04-Oct-2021 15:30:16 CEST Run: 1 Event: 1662
%MSG-w BasicTrajectoryState:  TrackProducer:jetCoreRegionalStepTracks  04-Oct-2021 15:30:16 CEST Run: 1 Event: 1662
%MSG-w BasicTrajectoryState:  TrackProducer:jetCoreRegionalStepTracks  04-Oct-2021 15:30:16 CEST Run: 1 Event: 1662
%MSG-w BasicTrajectoryState:  TrackProducer:jetCoreRegionalStepTracks  04-Oct-2021 15:30:16 CEST Run: 1 Event: 1662
%MSG-w TrackNaN:  TrackProducer:jetCoreRegionalStepTracks  04-Oct-2021 15:30:16 CEST Run: 1 Event: 1662
%MSG-w OutOfBounds:  TrackProducer:initialStepTracksPreSplitting  04-Oct-2021 15:32:13 CEST Run: 1 Event: 1670
%MSG-w OutOfBounds:  TrackProducer:initialStepTracks 04-Oct-2021 15:32:34 CEST  Run: 1 Event: 1670
[innocent@patatrack02 asan]$ grep MSG-w oriReco.log | grep -v MemoryCheck
%MSG-w XrdAdaptorInternal:  file_open 04-Oct-2021 16:14:53 CEST pre-events
%MSG-w XrdAdaptorInternal:  file_open 04-Oct-2021 16:14:53 CEST pre-events
%MSG-w XrdAdaptor:  file_open 04-Oct-2021 16:14:54 CEST pre-events
%MSG-w OutOfBounds:  TrackProducer:initialStepTracksPreSplitting  04-Oct-2021 16:19:35 CEST Run: 1 Event: 1670
%MSG-w OutOfBounds:  TrackProducer:initialStepTracks 04-Oct-2021 16:19:36 CEST  Run: 1 Event: 1670
%MSG-w OutOfBounds:  TrackProducer:mixedTripletStepTracks  04-Oct-2021 16:20:39 CEST Run: 1 Event: 1691
%MSG-w PFTrackTransformer:  PFElecTkProducer:pfTrackElec  04-Oct-2021 16:24:15 CEST Run: 1 Event: 1761