search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.08k stars 4.31k forks source link

[12_6_0_pre2] Reco timing increase in Phase2 reconstruction #39482

Closed clacaputo closed 1 year ago

clacaputo commented 2 years ago

We are observing a ~10% increase in reco timing for wf 39634.21 in 12_6_0_pre2. More details can be found in the time diff table

A new version for TICL (TICLv4) has been introduced in pre2 with #38880 

added      +2.37%         0.00 ms/ev ->      1252.35 ms/ev ticlTrackstersCLUE3DHigh

A substantial increase in timing is present in the producers reported hereafter: 

   +1.776026      +0.00%         0.14 ms/ev ->         2.35 ms/ev photonCoreHGC
   +1.461198      +0.01%         0.71 ms/ev ->         4.56 ms/ev photonsHGC
   +1.311622      +4.63%       640.79 ms/ev ->      3082.69 ms/ev ecalDrivenElectronSeeds
   +1.293879      +0.00%         0.05 ms/ev ->         0.24 ms/ev muPFIsoDepositNeutral
   +1.226026      +0.00%         0.00 ms/ev ->         0.00 ms/ev MEtoEDMConverter
   +1.200577      +0.01%         1.00 ms/ev ->         4.01 ms/ev gedPhotonCore
   +1.194280      +0.47%        82.87 ms/ev ->       328.55 ms/ev gsfGeneralInOutOutInConversionTrackMerger
   +1.170723      +0.48%        88.80 ms/ev ->       339.53 ms/ev ecalDrivenGsfElectronsHGC
   +1.140084      +0.10%        19.18 ms/ev ->        70.05 ms/ev ecalDrivenGsfElectronCoresHGC
   +1.134523      +3.18%       640.32 ms/ev ->      2319.05 ms/ev electronGsfTracks
   +1.129216      +0.05%        10.74 ms/ev ->        38.59 ms/ev pixelClusterTagInfos
   +1.119278      +0.00%         0.65 ms/ev ->         2.31 ms/ev photonCore
   +1.097390      +0.58%       126.40 ms/ev ->       433.74 ms/ev pfTrackElec
   +1.092464      +0.02%         3.85 ms/ev ->        13.12 ms/ev ecalDrivenGsfElectronCores
   +1.089632      +0.00%         0.07 ms/ev ->         0.25 ms/ev pfAllNeutralHadrons
   +1.049010      +0.01%         3.28 ms/ev ->        10.51 ms/ev electronMergedSeeds
   +1.025220      +2.65%       664.38 ms/ev ->      2061.90 ms/ev electronCkfTrackCandidates
   +0.969029      +0.19%        53.09 ms/ev ->       152.90 ms/ev allConversions

The most significant increase registered for:

   +1.311622      +4.63%       640.79 ms/ev ->      3082.69 ms/ev ecalDrivenElectronSeeds
   +1.134523      +3.18%       640.32 ms/ev ->      2319.05 ms/ev electronGsfTracks
   +1.025220      +2.65%      664.38 ms/ev ->      2061.90 ms/ev electronCkfTrackCandidates

It is not clear whether #38880 is responsible for the overall increase in timing

FYI @cms-sw/hgcal-dpg-l2

clacaputo commented 2 years ago

assign reconstruction

cmsbuild commented 2 years ago

New categories assigned: reconstruction

@mandrenguyen,@clacaputo you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild commented 2 years ago

A new Issue was created by @clacaputo Claudio Caputo.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

clacaputo commented 2 years ago

assign upgrade

cmsbuild commented 2 years ago

New categories assigned: upgrade

@AdrianoDee,@srimanob you have been requested to review this Pull request/Issue and eventually sign? Thanks

srimanob commented 2 years ago

Hi @cms-sw/reconstruction-l2 @cms-sw/hgcal-dpg-l2

If I understand correctly, it is unclear because you try to compare with workflow .203, right?

Is this line by intention?

ticlIterationsTask = cms.Task(
    ticlCLUE3DHighStepTask
)

https://github.com/cms-sw/cmssw/blob/master/RecoHGCal/TICL/python/iterativeTICL_cff.py#L26-L28

Because, in this case, you already include ticlCLUE3DHighStepTask also in ticl_v3. But this is not what it was in CMSSW_12_6_0_pre1.

From 39434.203, you will get process.ticlIterationsTask = cms.Task(process.ticlCLUE3DHighStepTask, process.ticlEMStepTask, process.ticlHADStepTask, process.ticlTrkEMStepTask, process.ticlTrkStepTask)

But from 39434.0 in CMSSW_12_6_0_pre1 process.ticlIterationsTask = cms.Task(process.ticlEMStepTask, process.ticlHADStepTask, process.ticlTrkEMStepTask, process.ticlTrkStepTask)

So, the increasing from added +2.38% 0.00 ms/ev -> 1269.06 ms/ev ticlTrackstersCLUE3DHigh appears with TICLv4 as expected (to confirmed by HGCAL), but not expected in TICLv3 workflow.

rovere commented 2 years ago 
       added      +2.38%         0.00 ms/ev ->      1269.06 ms/ev ticlTrackstersCLUE3DHigh
     removed      -0.01%         3.23 ms/ev ->         0.00 ms/ev filteredLayerClustersEM
     removed      -0.09%        50.13 ms/ev ->         0.00 ms/ev ticlTrackstersHAD
     removed      -0.01%         2.97 ms/ev ->         0.00 ms/ev ticlSeedingTrk
     removed      -0.01%         3.18 ms/ev ->         0.00 ms/ev filteredLayerClustersTrkEM
     removed      -1.00%       536.23 ms/ev ->         0.00 ms/ev ticlTrackstersTrk
       added      +0.00%         0.00 ms/ev ->         2.64 ms/ev filteredLayerClustersCLUE3DHigh
     removed      -0.00%         1.98 ms/ev ->         0.00 ms/ev filteredLayerClustersHAD
     removed      -1.43%       764.87 ms/ev ->         0.00 ms/ev ticlTrackstersEM
     removed      -0.00%         2.22 ms/ev ->         0.00 ms/ev filteredLayerClustersTrk
     removed      -0.92%       488.76 ms/ev ->         0.00 ms/ev ticlTrackstersTrkEM
   -1.764970      -0.63%       357.53 ms/ev ->        22.32 ms/ev ticlTrackstersMerge

Those numbers are in line with the measurements we made before submitting the PR and result in a net reduction of the purely HGCAL-related reconstruction time, which is good. There are clearly downstream effects, mainly on egamma related objects that are visible. Some comments:

rovere commented 2 years ago

@felicepantaleo @waredjeb FYI

clacaputo commented 2 years ago

Those numbers are in line with the measurements we made before submitting the PR and result in a net reduction of the purely HGCAL-related reconstruction time, which is good. There are clearly downstream effects, mainly on egamma related objects that are visible.

Hi @rovere , these were also my conclusions that I haven't stated properly in the description.

  • pre2, as said, is not optimal from the TICL point of view since it is missing some important improvements (retrained PID and energy regression model, linking) and bug fixes that have been merged after pre2.
  • It is quite likely that some changes and filters will have to be added in input to the objects used to reconstruct electrons and photons. We are already in contact with EGAMMA POG to coordinate efforts in this sense.
  • In this intense period of development some distruptive changes have to be expected, both in terms of net physics outcome and in terms of computing time.

Opening an issue is more effective for collecting inputs from different experts and being able to state whether the effect is somehow expected or not. As long as the distuptive changes are understood, the issue can be closed.

clacaputo commented 2 years ago

I've checked the time differences reverting #38880 in 12_6_0_pre2 on wf 39634.21. The time increase observed in various producers isn't any more there

12_6_0_pre2 VS 12_6_0_pre2[#38880 reverted]

  delta/mean delta/orJob     original                   new       module name
  ---------- ------------     --------                  ----       ------------
     removed      -2.15%      1252.35 ms/ev ->         0.00 ms/ev ticlTrackstersCLUE3DHigh
       added      +0.01%         0.00 ms/ev ->         3.42 ms/ev filteredLayerClustersEM
       added      +0.09%         0.00 ms/ev ->        49.90 ms/ev ticlTrackstersHAD
       added      +0.01%         0.00 ms/ev ->         2.96 ms/ev ticlSeedingTrk
       added      +0.01%         0.00 ms/ev ->         3.36 ms/ev filteredLayerClustersTrkEM
       added      +0.91%         0.00 ms/ev ->       531.30 ms/ev ticlTrackstersTrk
     removed      -0.00%         2.61 ms/ev ->         0.00 ms/ev filteredLayerClustersCLUE3DHigh
       added      +0.00%         0.00 ms/ev ->         2.09 ms/ev filteredLayerClustersHAD
       added      +1.30%         0.00 ms/ev ->       759.57 ms/ev ticlTrackstersEM
       added      +0.00%         0.00 ms/ev ->         2.43 ms/ev filteredLayerClustersTrk
       added      +0.83%         0.00 ms/ev ->       484.30 ms/ev ticlTrackstersTrkEM
   -1.793285      -0.00%         2.35 ms/ev ->         0.13 ms/ev photonCoreHGC
   +1.764094      +0.56%        22.03 ms/ev ->       351.45 ms/ev ticlTrackstersMerge
   -1.498126      -0.01%         4.56 ms/ev ->         0.65 ms/ev photonsHGC
   +1.405357      +0.00%         0.00 ms/ev ->         0.01 ms/ev lowPtTripletStepTrackingRegions
   -1.297696      -4.16%      3082.69 ms/ev ->       656.51 ms/ev ecalDrivenElectronSeeds
   -1.273164      -0.00%         0.24 ms/ev ->         0.05 ms/ev muPFIsoDepositNeutral
   -1.197286      -0.00%         0.00 ms/ev ->         0.00 ms/ev MEtoEDMConverter
   -1.181157      -0.01%         4.01 ms/ev ->         1.03 ms/ev gedPhotonCore
   -1.177637      -0.43%       339.53 ms/ev ->        87.87 ms/ev ecalDrivenGsfElectronsHGC
   -1.140899      -0.41%       328.55 ms/ev ->        89.86 ms/ev gsfGeneralInOutOutInConversionTrackMerger
   -1.136033      -0.00%         2.31 ms/ev ->         0.64 ms/ev photonCore
   -1.135163      -2.88%      2319.05 ms/ev ->       639.71 ms/ev electronGsfTracks
   -1.133885      -0.09%        70.05 ms/ev ->        19.36 ms/ev ecalDrivenGsfElectronCoresHGC
   -1.111499      -0.53%       433.74 ms/ev ->       123.86 ms/ev pfTrackElec
   -1.109924      -0.05%        38.59 ms/ev ->        11.04 ms/ev pixelClusterTagInfos
   -1.086457      -0.02%        13.12 ms/ev ->         3.88 ms/ev ecalDrivenGsfElectronCores
   -1.060788      -0.01%        10.51 ms/ev ->         3.22 ms/ev electronMergedSeeds
   -1.055590      -2.44%      2061.90 ms/ev ->       637.28 ms/ev electronCkfTrackCandidates
   -1.053960      -0.00%         0.25 ms/ev ->         0.08 ms/ev pfAllNeutralHadrons
   -0.962520      -0.17%       152.90 ms/ev ->        53.54 ms/ev allConversions
   -0.816884      -0.08%        77.63 ms/ev ->        32.61 ms/ev ecalDrivenGsfElectrons
   -0.803420      -0.00%         1.22 ms/ev ->         0.52 ms/ev pfAllNeutralHadronsAndPhotons
   +0.787274      +0.00%         1.74 ms/ev ->         4.00 ms/ev lowPtGsfElectronID
   -0.770464      -0.15%       154.21 ms/ev ->        68.44 ms/ev pfTrack
   -0.703786      -0.22%       247.02 ms/ev ->       118.42 ms/ev particleFlowSuperClusterHGCal
   -0.639401      -0.00%         1.46 ms/ev ->         0.75 ms/ev egmPhotonIsolationCITK
   -0.624002      -0.00%         1.38 ms/ev ->         0.72 ms/ev egmElectronIsolationCITK
   -0.582571      -0.14%       176.91 ms/ev ->        97.10 ms/ev particleFlowClusterHGCal
   -0.556057      -0.00%         1.02 ms/ev ->         0.58 ms/ev muPFIsoDepositGamma
   -0.525310      -0.00%         1.06 ms/ev ->         0.62 ms/ev pfNoPileUpCandidates
   -0.502092      -0.00%         1.74 ms/ev ->         1.04 ms/ev pfTICL
   -0.421234      -0.00%         7.64 ms/ev ->         4.98 ms/ev pfChargeBJetTags
   -0.354184      -0.00%         0.71 ms/ev ->         0.50 ms/ev pfAllPhotons
   -0.310548      -0.00%         0.85 ms/ev ->         0.62 ms/ev softPFElectronsTagInfos
   -0.308219      -0.19%       410.17 ms/ev ->       300.63 ms/ev combinatoricRecoTaus
   -0.271064      -0.00%         0.81 ms/ev ->         0.62 ms/ev ak4JetTracksAssociatorExplicit
   -0.234987      -0.00%        13.84 ms/ev ->        10.93 ms/ev recoTauAK4PFJets08Region
   -0.221675      -0.00%         2.91 ms/ev ->         2.33 ms/ev ak4PFJetsLegacyHPSPiZeros
   -0.214002      -0.00%         0.99 ms/ev ->         0.80 ms/ev muons
   -0.201995      -0.01%        27.60 ms/ev ->        22.53 ms/ev hpsPFTauProducerSansRefs
   -0.194024      -0.01%        24.34 ms/ev ->        20.04 ms/ev ak4PFJetsCHS
   -0.148422      -0.00%        11.00 ms/ev ->         9.48 ms/ev particleFlow
   -0.138529      -0.00%         7.00 ms/ev ->         6.09 ms/ev particleFlowTmp
   -0.138328      -0.01%        35.61 ms/ev ->        31.00 ms/ev pfRecoTauTagInfoProducer
   -0.136885      -0.01%        38.70 ms/ev ->        33.74 ms/ev ak4PFJets
   -0.115567      -0.06%       298.29 ms/ev ->       265.70 ms/ev hpsPFTauBasicDiscriminators
   -0.114583      -0.06%       297.28 ms/ev ->       265.06 ms/ev hpsPFTauBasicDiscriminatorsdR03
   +0.104576      +0.00%         5.37 ms/ev ->         5.96 ms/ev mtdTrackingRecHits
   +0.095464      +0.00%        15.85 ms/ev ->        17.44 ms/ev hgcalTrackCollection
   -0.080590      -0.02%       172.63 ms/ev ->       159.25 ms/ev pfImpactParameterTagInfos
   -0.079566      -0.65%      4976.78 ms/ev ->      4595.95 ms/ev AODSIMoutput
   -0.066484      -0.00%         7.87 ms/ev ->         7.36 ms/ev pfCombinedCvsBJetTags
   +0.066478      +0.00%        31.41 ms/ev ->        33.57 ms/ev uncleanedOnlyPfTrack
   -0.050996      -0.00%         5.19 ms/ev ->         4.93 ms/ev ak8PFJetsPuppiSoftDrop
  ---------- ------------     --------                  ----       ------------
Job total:  58.3244 s/ev ==> 52.0233 s/ev

12_6_0_pre1 VS 12_6_0_pre2[#38880 reverted]

  delta/mean delta/orJob     original                   new       module name
  ---------- ------------     --------                  ----       ------------
   +1.391380      +0.00%         0.00 ms/ev ->         0.01 ms/ev lowPtTripletStepTrackingRegions
   -0.819397      -0.01%         5.10 ms/ev ->         2.13 ms/ev uncleanedOnlyConversionTrackCandidates
   +0.692005      +0.00%         1.94 ms/ev ->         4.00 ms/ev lowPtGsfElectronID
   +0.412522      +0.00%         0.84 ms/ev ->         1.28 ms/ev uncleanedOnlyElectronGsfTracks
   -0.290003      -0.00%         1.57 ms/ev ->         1.18 ms/ev muonSeededTrackCandidatesOutIn
   -0.272193      -0.00%         0.80 ms/ev ->         0.61 ms/ev muonShowerInformation
   +0.253636      +0.01%        11.40 ms/ev ->        14.72 ms/ev vertexMerger
   -0.201684      -0.00%         3.99 ms/ev ->         3.26 ms/ev dt4DCosmicSegments
   -0.197955      -0.00%         8.78 ms/ev ->         7.20 ms/ev muonSeededTracksInOut
   -0.173452      -0.00%         6.46 ms/ev ->         5.43 ms/ev muonSeededSeedsInOut
   -0.166626      -0.01%        18.45 ms/ev ->        15.61 ms/ev mergedDuplicateTracks
   +0.162325      +0.04%       117.10 ms/ev ->       137.79 ms/ev preDuplicateMergingGeneralTracks
   -0.160099      -0.07%       245.00 ms/ev ->       208.68 ms/ev detachedQuadStepTracks
   -0.154893      -0.37%      1371.53 ms/ev ->      1174.36 ms/ev initialStepTracks
   -0.149650      -0.00%         5.45 ms/ev ->         4.69 ms/ev particleFlowClusterHCAL
   +0.137970      +0.00%         5.19 ms/ev ->         5.96 ms/ev mtdTrackingRecHits
   -0.134376      -0.19%       796.18 ms/ev ->       695.93 ms/ev lowPtQuadStepTracks
   -0.113889      -0.00%        14.48 ms/ev ->        12.92 ms/ev muonSeededTrackCandidatesInOut
   -0.102879      -0.03%       167.58 ms/ev ->       151.18 ms/ev lowPtTripletStepTracks
   -0.094105      -0.10%       596.81 ms/ev ->       543.17 ms/ev highPtTripletStepTracks
   -0.085426      -0.04%       228.79 ms/ev ->       210.05 ms/ev puppiNoLep
   +0.084519      +0.02%        94.96 ms/ev ->       103.34 ms/ev inclusiveVertexFinder
   -0.081669      -0.03%       228.12 ms/ev ->       210.22 ms/ev puppi
   +0.080959      +0.01%        82.87 ms/ev ->        89.86 ms/ev gsfGeneralInOutOutInConversionTrackMerger
   -0.079715      -0.04%       253.04 ms/ev ->       233.64 ms/ev detachedQuadStepTrackCandidates
   -0.079686      -0.00%        23.20 ms/ev ->        21.43 ms/ev gedPhotonsTmp
   -0.077899      -0.01%        62.81 ms/ev ->        58.10 ms/ev pixelPairStepTracks
   -0.075231      -0.01%        63.41 ms/ev ->        58.81 ms/ev CSCHaloData
   -0.070623      -0.01%        47.20 ms/ev ->        43.98 ms/ev ak4TrackJets
   -0.066629      -0.04%       321.35 ms/ev ->       300.63 ms/ev combinatoricRecoTaus
   -0.066588      -0.00%         6.80 ms/ev ->         6.36 ms/ev muonSimClassifier
   -0.062931      -0.07%       595.72 ms/ev ->       559.38 ms/ev tpClusterProducer
   +0.062282      +0.03%       233.33 ms/ev ->       248.32 ms/ev conversionTrackCandidates
   -0.061322      -0.05%       431.66 ms/ev ->       405.98 ms/ev candidateVertexArbitratorCvsL
   +0.061137      +0.00%        37.91 ms/ev ->        40.30 ms/ev tofPID
   +0.060958      +0.00%         6.16 ms/ev ->         6.55 ms/ev muonsFromCosmics1Leg
   -0.060458      -0.04%       361.00 ms/ev ->       339.82 ms/ev candidateVertexArbitrator
   +0.059907      +0.01%       111.15 ms/ev ->       118.01 ms/ev inclusiveCandidateVertexFinder
   +0.059373      +0.01%       111.83 ms/ev ->       118.68 ms/ev inclusiveCandidateVertexFinderCvsL
   -0.059191      -0.02%       176.19 ms/ev ->       166.06 ms/ev pixelPairStepTrackCandidates
   +0.058012      +0.00%        38.35 ms/ev ->        40.64 ms/ev tofPID4DnoPID
   +0.056783      +0.00%        16.47 ms/ev ->        17.44 ms/ev hgcalTrackCollection
   -0.056477      -0.21%      2052.04 ms/ev ->      1939.33 ms/ev initialStepTrackCandidates
   -0.055573      -0.25%      2433.31 ms/ev ->      2301.74 ms/ev lowPtQuadStepTrackCandidates
  ---------- ------------     --------                  ----       ------------
Job total:  52.7852 s/ev ==> 52.0233 s/ev
swagata87 commented 2 years ago

pre2, as said, is not optimal from the TICL point of view since it is missing some important improvements (retrained PID and energy regression model, linking) and bug fixes that have been merged after pre2.

yes let's check the reco timing again in 12_6_0_pre3 when it is cut.. maybe part of the timing increase will be mitigated by the improvements and bug-fixes that missed pre2 but will be there in pre3.

It is quite likely that some changes and filters will have to be added in input to the objects used to reconstruct electrons and photons. We are already in contact with EGAMMA POG to coordinate efforts in this sense.

if there is still a residual timing increase in pre3, we would need to think about the filters to be put in before accepting a trackster for egamma reco. Of course it would need some detailed studies to ensure that the filters will reduce the timing (and fakes) without hurting the efficiency and overall physics performance.

In this intense period of development some distruptive changes have to be expected, both in terms of net physics outcome and in terms of computing time.

yep! agreed.. and given Run3 rush, we might not be able to fix things in phase2 side super-quickly right now. But egamma takes note of it.

AdrianoDee commented 2 years ago

+upgrade solved by https://github.com/cms-sw/cmssw/pull/39810

clacaputo commented 1 year ago

wf 21034.21, CPU time:

Job total: 58.6912 s/ev ==> 51.8199 s/ev

TimeDiff

The same excluding the first 1 events
  delta/mean delta/orJob     original                   new       module name
  ---------- ------------     --------                  ----       ------------
   -1.781799      -0.00%         1.94 ms/ev ->         0.11 ms/ev photonCoreHGC
   -1.602525      -4.96%      3274.77 ms/ev ->       361.31 ms/ev ecalDrivenElectronSeeds
   -1.552555      -0.50%       338.22 ms/ev ->        42.60 ms/ev ecalDrivenGsfElectronsHGC
   -1.505418      -0.00%         2.70 ms/ev ->         0.38 ms/ev photonCore
   -1.493406      -3.09%      2123.75 ms/ev ->       307.97 ms/ev electronCkfTrackCandidates
   -1.487308      -0.01%         4.23 ms/ev ->         0.62 ms/ev gedPhotonCore
   -1.484925      -0.10%        67.09 ms/ev ->         9.92 ms/ev ecalDrivenGsfElectronCoresHGC
   -1.481800      -0.02%        11.83 ms/ev ->         1.76 ms/ev electronMergedSeeds
   -1.481571      -0.43%       294.28 ms/ev ->        43.82 ms/ev gsfGeneralInOutOutInConversionTrackMerger
   -1.468475      -0.00%         0.11 ms/ev ->         0.02 ms/ev pfDisplacedTrackerVertex
   -1.464545      -3.27%      2271.78 ms/ev ->       351.11 ms/ev electronGsfTracks
   -1.410864      -0.02%        13.52 ms/ev ->         2.33 ms/ev ecalDrivenGsfElectronCores
   -1.398589      -0.61%       434.36 ms/ev ->        76.86 ms/ev pfTrackElec
   -1.295927      -0.00%         3.43 ms/ev ->         0.73 ms/ev photonsHGC
   -1.230152      -0.33%       255.50 ms/ev ->        60.89 ms/ev particleFlowSuperClusterHGCal
   -1.132376      -0.17%       141.16 ms/ev ->        39.10 ms/ev allConversions
   -1.061376      -0.00%         0.15 ms/ev ->         0.05 ms/ev gsfConversionTrackProducer
   -1.039618      -0.10%        82.32 ms/ev ->        26.01 ms/ev ecalDrivenGsfElectrons
   -1.022161      -0.20%       176.65 ms/ev ->        57.16 ms/ev particleFlowClusterHGCal
   -0.937514      -0.16%       145.92 ms/ev ->        52.78 ms/ev pfTrack
   -0.721728      -0.01%         8.70 ms/ev ->         4.09 ms/ev uncleanedOnlyConversionTrackCandidates
   -0.476624      -0.41%       620.16 ms/ev ->       381.46 ms/ev ak4JetTracksAssociatorAtVertexJPT
   -0.414931      -0.00%         1.86 ms/ev ->         1.22 ms/ev uncleanedOnlyElectronSeeds
   -0.211398      -0.00%         0.68 ms/ev ->         0.55 ms/ev muonSeededTrackCandidatesOutInDisplaced
   +0.206109      +0.00%         6.11 ms/ev ->         7.51 ms/ev muonsFromCosmics1Leg
   +0.144275      +0.01%        43.46 ms/ev ->        50.22 ms/ev globalMuons
   +0.131503      +0.14%       588.60 ms/ev ->       671.45 ms/ev particleFlowClusterHGCalFromSimCl
   +0.120601      +0.01%        37.85 ms/ev ->        42.71 ms/ev tevMuons
   +0.120225      +0.03%       144.52 ms/ev ->       163.01 ms/ev cosmicMuons1Leg
   +0.108396      +0.01%        26.59 ms/ev ->        29.63 ms/ev displacedMuons1stStep
   +0.093493      +0.00%        21.46 ms/ev ->        23.56 ms/ev gedPhotonsTmp
   +0.087699      +0.06%       356.98 ms/ev ->       389.73 ms/ev generalV0Candidates
   -0.086261      -0.00%         6.52 ms/ev ->         5.98 ms/ev ak8PFJetsPuppiSoftDrop
   +0.085403      +0.00%        27.30 ms/ev ->        29.74 ms/ev gedGsfElectronsTmp
   +0.081303      +0.00%        20.40 ms/ev ->        22.13 ms/ev lowPtGsfElePfGsfTracks
   +0.079848      +0.00%        19.04 ms/ev ->        20.62 ms/ev detachedQuadStep
   -0.079239      -0.66%      5082.29 ms/ev ->      4694.92 ms/ev AODSIMoutput
   +0.078510      +0.05%       350.22 ms/ev ->       378.83 ms/ev candidateVertexArbitrator
   +0.076115      +0.00%        17.90 ms/ev ->        19.32 ms/ev pfNoPileUpIso
   +0.075966      +0.09%       683.81 ms/ev ->       737.81 ms/ev highPtTripletStepHitTriplets
   +0.075886      +0.40%      3003.35 ms/ev ->      3240.25 ms/ev detachedQuadStepHitQuadruplets
   +0.074438      +0.00%        10.86 ms/ev ->        11.70 ms/ev muonsFromCosmics
   +0.074169      +0.01%        44.74 ms/ev ->        48.18 ms/ev pfNoPileUpJME
   +0.074068      +0.16%      1204.56 ms/ev ->      1297.21 ms/ev particleFlowDisplacedVertex
   +0.074067      +0.00%        37.14 ms/ev ->        40.00 ms/ev refittedStandAloneMuons
   +0.073487      +0.10%       743.43 ms/ev ->       800.15 ms/ev lowPtQuadStepHitQuadruplets
   +0.072760      +0.08%       656.64 ms/ev ->       706.22 ms/ev initialStepHitQuadruplets
   +0.072751      +0.05%       384.66 ms/ev ->       413.70 ms/ev lowPtTripletStepHitTriplets
   +0.072719      +0.00%         6.70 ms/ev ->         7.20 ms/ev muonSeededTracksInOut
   +0.072393      +0.09%       692.35 ms/ev ->       744.36 ms/ev particleFlowDisplacedVertexCandidate
   +0.072064      +0.00%        24.83 ms/ev ->        26.69 ms/ev siPixelClusterShapeCache
   +0.069368      +0.23%      1899.24 ms/ev ->      2035.72 ms/ev earlyGeneralTracks
   +0.069168      +0.07%       590.56 ms/ev ->       632.87 ms/ev duplicateTrackCandidates
   +0.067997      +0.02%       158.49 ms/ev ->       169.64 ms/ev JetPlusTrackZSPCorJetAntiKt4
   +0.066492      +0.00%        39.76 ms/ev ->        42.50 ms/ev lowPtGsfEleGsfTracks
   +0.065957      +0.00%        14.28 ms/ev ->        15.26 ms/ev vertexMerger
   +0.064881      +0.00%        27.31 ms/ev ->        29.14 ms/ev pixelPairStepSeeds
   +0.063632      +0.02%       134.76 ms/ev ->       143.62 ms/ev generalTracks
   +0.060970      +0.04%       336.37 ms/ev ->       357.52 ms/ev trackVertexArbitrator
   +0.060795      +0.01%        67.80 ms/ev ->        72.05 ms/ev firstStepPrimaryVertices
   +0.060047      +0.04%       415.32 ms/ev ->       441.04 ms/ev candidateVertexArbitratorCvsL
   -0.059563      -0.01%        63.87 ms/ev ->        60.18 ms/ev CSCHaloData
   +0.058185      +0.00%        47.37 ms/ev ->        50.21 ms/ev ak4TrackJets
   +0.057881      +0.00%        39.14 ms/ev ->        41.47 ms/ev tofPID
   +0.057757      +0.01%       117.97 ms/ev ->       124.98 ms/ev preDuplicateMergingGeneralTracks
   +0.057748      +0.02%       225.93 ms/ev ->       239.36 ms/ev puppi
   +0.057490      +0.00%         9.46 ms/ev ->        10.02 ms/ev pfChargeBJetTags
   +0.057461      +0.03%       289.40 ms/ev ->       306.52 ms/ev hpsPFTauBasicDiscriminators
   +0.056426      +0.03%       288.09 ms/ev ->       304.82 ms/ev hpsPFTauBasicDiscriminatorsdR03
   +0.055195      +0.01%        76.93 ms/ev ->        81.29 ms/ev lowPtQuadStepHitDoublets
   +0.055028      +0.00%        26.27 ms/ev ->        27.75 ms/ev cosmicsVeto
   +0.054261      +0.02%       211.84 ms/ev ->       223.65 ms/ev pfPileUpIso
   +0.054189      +0.02%       208.01 ms/ev ->       219.60 ms/ev pfPileUpJME
   +0.053962      +0.02%       227.03 ms/ev ->       239.63 ms/ev puppiNoLep
   +0.053907      +0.01%        80.47 ms/ev ->        84.93 ms/ev ak4PFJetsRecoTauChargedHadrons
   +0.053493      +0.00%        39.39 ms/ev ->        41.55 ms/ev tofPID4DnoPID
   +0.053152      +0.00%        32.29 ms/ev ->        34.05 ms/ev standAloneMuons
   +0.052518      +0.00%        24.45 ms/ev ->        25.77 ms/ev uncleanedOnlyAllConversions
   +0.051530      +0.00%        55.41 ms/ev ->        58.34 ms/ev highPtTripletStepSeeds
   +0.051124      +0.00%        28.15 ms/ev ->        29.63 ms/ev pfSecondaryVertexTagInfos
   +0.050243      +0.01%       129.63 ms/ev ->       136.31 ms/ev detachedQuadStepHitDoublets
  ---------- ------------     --------                  ----       ------------
clacaputo commented 1 year ago

+reconstruction

cmsbuild commented 1 year ago

This issue is fully signed and ready to be closed.