search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.07k stars 4.28k forks source link

heap-buffer-overflow HcalDDDSimConstants::getPhiCons() #39480

Closed dan131riley closed 1 year ago

dan131riley commented 1 year ago

Log from 39434.911, also seen in 38634.0:

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/raw/el8_amd64_gcc11/CMSSW_12_6_ASAN_X_2022-09-21-1100/pyRelValMatrixLogs/run/39434.911_TTbar_14TeV+2026D88_DD4hep+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal/step1_TTbar_14TeV+2026D88_DD4hep+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal.log

=================================================================
==7210==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60c0015bbd98 at pc 0x2b3b5ad6af5f bp 0x2b3b887243a0 sp 0x2b3b88724398
READ of size 8 at 0x60c0015bbd98 thread T7
    #0 0x2b3b5ad6af5e in HcalDDDSimConstants::getPhiCons(int const&, int const&) const (/cvmfs/cms-ib.cern.ch/nweek-02751/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_ASAN_X_2022-09-21-1100/lib/el8_amd64_gcc11/libGeometryHcalCommonData.so+0x6df5e)
    #1 0x2b3b5ad9ecfc in HcalNumberingFromDDD::unitID(int, double, double, int, int) const (/cvmfs/cms-ib.cern.ch/nweek-02751/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_ASAN_X_2022-09-21-1100/lib/el8_amd64_gcc11/libGeometryHcalCommonData.so+0xa1cfc)
    #2 0x2b3b5ad9f238 in HcalNumberingFromDDD::unitID(int, ROOT::Math::DisplacementVector3D<ROOT::Math::Cartesian3D<double>, ROOT::Math::DefaultCoordinateSystemTag> const&, int, int) const (/cvmfs/cms-ib.cern.ch/nweek-02751/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_ASAN_X_2022-09-21-1100/lib/el8_amd64_gcc11/libGeometryHcalCommonData.so+0xa2238)
    #3 0x2b3b70ebc288 in HCalSD::setDetUnitId(int, CLHEP::Hep3Vector const&, int, int) (/cvmfs/cms-ib.cern.ch/nweek-02751/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_ASAN_X_2022-09-21-1100/biglib/el8_amd64_gcc11/pluginSimulation.so+0x711288)
    #4 0x2b3b70e63106 in CaloSD::ProcessHits(G4Step*, G4TouchableHistory*) (/cvmfs/cms-ib.cern.ch/nweek-02751/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_ASAN_X_2022-09-21-1100/biglib/el8_amd64_gcc11/pluginSimulation.so+0x6b8106)
    #5 0x2b3b72bb4a36 in G4SteppingManager::Stepping() (/cvmfs/cms-ib.cern.ch/nweek-02751/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_ASAN_X_2022-09-21-1100/biglib/el8_amd64_gcc11/pluginSimulation.so+0x2409a36)
    #6 0x2b3b7268e16a in G4TrackingManager::ProcessOneTrack(G4Track*) (/cvmfs/cms-ib.cern.ch/nweek-02751/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_ASAN_X_2022-09-21-1100/biglib/el8_amd64_gcc11/pluginSimulation.so+0x1ee316a)
    #7 0x2b3b72370de9 in G4EventManager::DoProcessing(G4Event*) (/cvmfs/cms-ib.cern.ch/nweek-02751/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_ASAN_X_2022-09-21-1100/biglib/el8_amd64_gcc11/pluginSimulation.so+0x1bc5de9)

0x60c0015bbd98 is located 88 bytes to the right of 128-byte region [0x60c0015bbcc0,0x60c0015bbd40)
allocated by thread T4 here:
    #0 0x2b3b2e114f37 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x2b3b5ad9ffbe in std::vector<double, std::allocator<double> >::operator=(std::vector<double, std::allocator<double> > const&) [clone .isra.0] (/cvmfs/cms-ib.cern.ch/nweek-02751/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_ASAN_X_2022-09-21-1100/lib/el8_amd64_gcc11/libGeometryHcalCommonData.so+0xa2fbe)
    #2 0x2b3b5ada505c in HcalParametersFromDD::build(cms::DDCompactView const&, HcalParameters&) (/cvmfs/cms-ib.cern.ch/nweek-02751/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_ASAN_X_2022-09-21-1100/lib/el8_amd64_gcc11/libGeometryHcalCommonData.so+0xa805c)
    #3 0x2b3b5be80d0e in HcalParametersESModule::produce(HcalParametersRcd const&) (/cvmfs/cms-ib.cern.ch/nweek-02751/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_ASAN_X_2022-09-21-1100/lib/el8_amd64_gcc11/pluginGeometryHcalCommonDataPlugins.so+0xe3d0e)
cmsbuild commented 1 year ago

A new Issue was created by @dan131riley Dan Riley.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel commented 1 year ago

assign geometry

cmsbuild commented 1 year ago

New categories assigned: geometry

@mdhildreth,@ianna,@Dr15Jones,@makortel,@bsunanda,@civanch you have been requested to review this Pull request/Issue and eventually sign? Thanks

makortel commented 1 year ago

Looks similar to https://github.com/cms-sw/cmssw/issues/39445 even if the stack trace is different.

dan131riley commented 1 year ago

Looks similar to #39445 even if the stack trace is different.

It's plausible they are related

Dr15Jones commented 1 year ago

So I was able to replicate the problem with a debug build under gdb. The crash I see happens on line 173

https://github.com/cms-sw/cmssw/blob/8faa02169a1e1a0f11df7e927eab9d5179a58015/Geometry/HcalCommonData/src/HcalDDDRecConstants.cc#L171-L173

but the problem originates on line 171 as the value of ieta is 28 and the length of ietaMap is only 16.

For the problem I see, the values are being deduced by looking at the HcalID of values read from the pileup where the raw value of the id is testId=604008449.

Dr15Jones commented 1 year ago

So the values used to fill ietaMap come from this ES module

>>> print(process.hcalParameters.dumpPython())
cms.ESProducer("HcalParametersESModule",
    appendToDataLabel = cms.string(''),
    fromDD4hep = cms.bool(True)
)

which gets its data from DD4hep

The values pulled from the loop filling the structure are

 nEta 16 i 0 hpar->etagroup[i] 1
 nEta 16 i 1 hpar->etagroup[i] 1
 nEta 16 i 2 hpar->etagroup[i] 1
 nEta 16 i 3 hpar->etagroup[i] 1
 nEta 16 i 4 hpar->etagroup[i] 1
 nEta 16 i 5 hpar->etagroup[i] 1
 nEta 16 i 6 hpar->etagroup[i] 1
 nEta 16 i 7 hpar->etagroup[i] 1
 nEta 16 i 8 hpar->etagroup[i] 1
 nEta 16 i 9 hpar->etagroup[i] 1
 nEta 16 i 10 hpar->etagroup[i] 1
 nEta 16 i 11 hpar->etagroup[i] 1
 nEta 16 i 12 hpar->etagroup[i] 1
 nEta 16 i 13 hpar->etagroup[i] 1
 nEta 16 i 14 hpar->etagroup[i] 1
 nEta 16 i 15 hpar->etagroup[i] 1

The DD4Hep description appears to come from this module

>>> print(process.DDDetectorESProducer.dumpPython())
cms.ESSource("DDDetectorESProducer",
    appendToDataLabel = cms.string(''),
    confGeomXMLFiles = cms.FileInPath('Geometry/CMSCommonData/data/dd4hep/cmsExtendedGeometry2026D88.xml')
)

Is this the correct geometry file for this pileup?

bsunanda commented 1 year ago

Hi

I believe they are using a pileup file from runs 1,2, or 3 for phase2 where the ieta value is restricted to 16. I shall make a protection to make a more realistic diagnostic. The ieta value for this channel is 28 where the maximum ieta vaue should be 16 (there is no HE in run4). Thanks Chris for providing the diagnosis.

Sunanda

From: Chris Jones @.*** Sent: 28 October 2022 23:23 To: cms-sw/cmssw Cc: Sunanda Banerjee; Mention Subject: Re: [cms-sw/cmssw] heap-buffer-overflow HcalDDDSimConstants::getPhiCons() (Issue #39480)

So the values used to fill ietaMap come from this ES module

print(process.hcalParameters.dumpPython()) cms.ESProducer("HcalParametersESModule", appendToDataLabel = cms.string(''), fromDD4hep = cms.bool(True) )

which gets its data from DD4hep

The values pulled from the loop filling the structure are

nEta 16 i 0 hpar->etagroup[i] 1 nEta 16 i 1 hpar->etagroup[i] 1 nEta 16 i 2 hpar->etagroup[i] 1 nEta 16 i 3 hpar->etagroup[i] 1 nEta 16 i 4 hpar->etagroup[i] 1 nEta 16 i 5 hpar->etagroup[i] 1 nEta 16 i 6 hpar->etagroup[i] 1 nEta 16 i 7 hpar->etagroup[i] 1 nEta 16 i 8 hpar->etagroup[i] 1 nEta 16 i 9 hpar->etagroup[i] 1 nEta 16 i 10 hpar->etagroup[i] 1 nEta 16 i 11 hpar->etagroup[i] 1 nEta 16 i 12 hpar->etagroup[i] 1 nEta 16 i 13 hpar->etagroup[i] 1 nEta 16 i 14 hpar->etagroup[i] 1 nEta 16 i 15 hpar->etagroup[i] 1

The DD4Hep description appears to come from this module

print(process.DDDetectorESProducer.dumpPython()) cms.ESSource("DDDetectorESProducer", appendToDataLabel = cms.string(''), confGeomXMLFiles = cms.FileInPath('Geometry/CMSCommonData/data/dd4hep/cmsExtendedGeometry2026D88.xml') )

Is this the correct geometry file for this pileup?

— Reply to this email directly, view it on GitHubhttps://github.com/cms-sw/cmssw/issues/39480#issuecomment-1295480157, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABGMZORK45GZHVEC2NJ2KE3WFQ74FANCNFSM6AAAAAAQTCGRMM. You are receiving this because you were mentioned.Message ID: @.***>

makortel commented 1 year ago

I believe they are using a pileup file from runs 1,2, or 3 for phase2 where the ieta value is restricted to 16

Umm, I'd expect runTheMatrix workflows to have consistent pileup files (@cms-sw/pdmv-l2).

The workflows 20834.911 and 23634.0 (that are the ones failing after the re-numbering of phase 2 workflows) do not have pileup. (checking workflows 39434.911 and 38634.0, that were mentioned in the issue description, from 12_5_X shows they don't have pileup either)

makortel commented 1 year ago

After https://github.com/cms-sw/cmssw/pull/39920 workflow 20834.911 step 2 fails with an exception

----- Begin Fatal Exception 01-Nov-2022 05:30:17 CET-----------------------
An exception of category 'HcalDDDRecConstants' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 2
   [1] Running path 'FEVTDEBUGHLToutput_step'
   [2] Prefetching for module PoolOutputModule/'FEVTDEBUGHLToutput'
   [3] Calling method for module MixingModule/'mix'
Exception Message:
getHCID: receives an eta value 28 outside the limit (1:16)
----- End Fatal Exception -------------------------------------------------

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc10/CMSSW_12_6_X_2022-10-31-2300/pyRelValMatrixLogs/run/20834.911_TTbar_14TeV+2026D88_DD4hep+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal+ALCAPhase2/step2_TTbar_14TeV+2026D88_DD4hep+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal+ALCAPhase2.log#/

(as I noted above, this workflow does not include pileup)

Dr15Jones commented 1 year ago

The configuration of the mixing module in the job that is failing is

>>> print(process.mix.dumpPython())
cms.EDProducer("MixingModule",
    LabelPlayback = cms.string(''),
    bunchspace = cms.int32(450),
    digitizers = cms.PSet(
        calotruth = cms.PSet(
            HepMCProductLabel = cms.InputTag("generatorSmeared"),
            MaxPseudoRapidity = cms.double(5.0),
            MinEnergy = cms.double(0.5),
            accumulatorType = cms.string('CaloTruthAccumulator'),
            allowDifferentSimHitProcesses = cms.bool(False),
            doHGCAL = cms.bool(True),
            genParticleCollection = cms.InputTag("genParticles"),
            maximumPreviousBunchCrossing = cms.uint32(0),
            maximumSubsequentBunchCrossing = cms.uint32(0),
            premixStage1 = cms.bool(False),
            simHitCollections = cms.PSet(
                hgc = cms.VInputTag(cms.InputTag("g4SimHits","HGCHitsEE"), cms.InputTag("g4SimHits","HGCHitsHEfront"), cms.InputTag("g4SimHits","HGCHitsHEback"))
            ),
            simTrackCollection = cms.InputTag("g4SimHits"),
            simVertexCollection = cms.InputTag("g4SimHits")
        ),
        ecal = cms.PSet(
            ConstantTerm = cms.double(0.003),
            EBCorrNoiseMatrixG01 = cms.vdouble(
                1.0, 0.73354, 0.64442, 0.58851, 0.55425,
                0.53082, 0.51916, 0.51097, 0.50732, 0.50409
            ),
            EBCorrNoiseMatrixG06 = cms.vdouble(
                1.0, 0.70946, 0.58021, 0.49846, 0.45006,
                0.41366, 0.39699, 0.38478, 0.37847, 0.37055
            ),
            EBCorrNoiseMatrixG12 = cms.vdouble(
                1.0, 0.71073, 0.55721, 0.46089, 0.40449,
                0.35931, 0.33924, 0.32439, 0.31581, 0.30481
            ),
            EBdigiCollection = cms.string(''),
            EBs25notContainment = cms.double(0.9675),
            EECorrNoiseMatrixG01 = cms.vdouble(
                1.0, 0.72698, 0.62048, 0.55691, 0.51848,
                0.49147, 0.47813, 0.47007, 0.46621, 0.46265
            ),
            EECorrNoiseMatrixG06 = cms.vdouble(
                1.0, 0.71217, 0.47464, 0.34056, 0.26282,
                0.20287, 0.17734, 0.16256, 0.15618, 0.14443
            ),
            EECorrNoiseMatrixG12 = cms.vdouble(
                1.0, 0.71373, 0.44825, 0.30152, 0.21609,
                0.14786, 0.11772, 0.10165, 0.09465, 0.08098
            ),
            EEdigiCollection = cms.string(''),
            EEs25notContainment = cms.double(0.968),
            ESdigiCollection = cms.string(''),
            EcalPreMixStage1 = cms.bool(False),
            EcalPreMixStage2 = cms.bool(False),
            UseLCcorrection = cms.untracked.bool(True),
            accumulatorType = cms.string('EcalDigiProducer'),
            apdAddToBarrel = cms.bool(False),
            apdDigiTag = cms.string('APD'),
            apdDoPEStats = cms.bool(True),
            apdNonlParms = cms.vdouble(
                1.48, -3.75, 1.81, 1.26, 2.0,
                45, 1.0
            ),
            apdSeparateDigi = cms.bool(True),
            apdShapeTau = cms.double(40.5),
            apdShapeTstart = cms.double(74.5),
            apdSimToPEHigh = cms.double(88200000.0),
            apdSimToPELow = cms.double(2450000.0),
            apdTimeOffWidth = cms.double(0.8),
            apdTimeOffset = cms.double(-13.5),
            applyConstantTerm = cms.bool(True),
            binOfMaximum = cms.int32(6),
            cosmicsPhase = cms.bool(False),
            cosmicsShift = cms.double(0.0),
            doEB = cms.bool(True),
            doEE = cms.bool(False),
            doENoise = cms.bool(True),
            doES = cms.bool(False),
            doESNoise = cms.bool(True),
            doFast = cms.bool(True),
            doPhotostatistics = cms.bool(True),
            hitsProducer = cms.string('g4SimHits'),
            makeDigiSimLinks = cms.untracked.bool(False),
            photoelectronsToAnalogBarrel = cms.double(0.000444444),
            photoelectronsToAnalogEndcap = cms.double(0.000555555),
            samplingFactor = cms.double(1.0),
            simHitToPhotoelectronsBarrel = cms.double(2250.0),
            simHitToPhotoelectronsEndcap = cms.double(1800.0),
            syncPhase = cms.bool(True),
            timeDependent = cms.bool(False),
            timePhase = cms.double(0.0)
        ),
        ecalTime = cms.PSet(
            EBtimeDigiCollection = cms.string('EBTimeDigi'),
            EEtimeDigiCollection = cms.string('EETimeDigi'),
            accumulatorType = cms.string('EcalTimeDigiProducer'),
            hitsProducerEB = cms.InputTag("g4SimHits","EcalHitsEB"),
            hitsProducerEE = cms.InputTag("g4SimHits","EcalHitsEE"),
            timeLayerBarrel = cms.int32(7),
            timeLayerEndcap = cms.int32(3)
        ),
        fastTimingLayer = cms.PSet(
            accumulatorType = cms.string('MTDDigiProducer'),
            barrelDigitizer = cms.PSet(
                DeviceSimulation = cms.PSet(
                    LightCollectionEff = cms.double(0.25),
                    LightCollectionSlopeL = cms.double(0.075),
                    LightCollectionSlopeR = cms.double(0.075),
                    LightYield = cms.double(40000.0),
                    PhotonDetectionEff = cms.double(0.2),
                    bxTime = cms.double(25)
                ),
                ElectronicsSimulation = cms.PSet(
                    ChannelTimeOffset = cms.double(0.0),
                    CorrelationCoefficient = cms.double(1.0),
                    DarkCountRate = cms.double(10.0),
                    EnergyThreshold = cms.double(4.0),
                    Npe_to_V = cms.double(0.0064),
                    Npe_to_pC = cms.double(0.016),
                    ReferencePulseNpe = cms.double(100.0),
                    ScintillatorDecayTime = cms.double(40.0),
                    ScintillatorRiseTime = cms.double(1.1),
                    SigmaClock = cms.double(0.015),
                    SigmaElectronicNoise = cms.double(1.0),
                    SinglePhotonTimeResolution = cms.double(0.06),
                    SmearTimeForOOTtails = cms.bool(True),
                    TestBeamMIPTimeRes = cms.double(4.293),
                    TimeThreshold1 = cms.double(20.0),
                    TimeThreshold2 = cms.double(50.0),
                    adcNbits = cms.uint32(10),
                    adcSaturation_MIP = cms.double(600.0),
                    adcThreshold_MIP = cms.double(0.064),
                    bxTime = cms.double(25),
                    smearChannelTimeOffset = cms.double(0.0),
                    tdcNbits = cms.uint32(10),
                    toaLSB_ns = cms.double(0.02)
                ),
                digiCollectionTag = cms.string('FTLBarrel'),
                digitizerName = cms.string('BTLDigitizer'),
                inputSimHits = cms.InputTag("g4SimHits","FastTimerHitsBarrel"),
                maxSimHitsAccTime = cms.uint32(100),
                premixStage1 = cms.bool(False),
                premixStage1MaxCharge = cms.double(1000000.0),
                premixStage1MinCharge = cms.double(0.0001)
            ),
            endcapDigitizer = cms.PSet(
                DeviceSimulation = cms.PSet(
                    bxTime = cms.double(25),
                    meVPerMIP = cms.double(0.015),
                    tofDelay = cms.double(1)
                ),
                ElectronicsSimulation = cms.PSet(
                    FluenceVsRadius = cms.string('1.937*TMath::Power(x,-1.706)'),
                    IntegratedLuminosity = cms.double(1000.0),
                    LGADGainVsFluence = cms.string('TMath::Min(15.,30.-x)'),
                    TimeResolution2 = cms.string('0.0225/x'),
                    adcNbits = cms.uint32(8),
                    adcSaturation_MIP = cms.double(25),
                    adcThreshold_MIP = cms.double(0.025),
                    bxTime = cms.double(25),
                    tdcNbits = cms.uint32(11),
                    toaLSB_ns = cms.double(0.013)
                ),
                digiCollectionTag = cms.string('FTLEndcap'),
                digitizerName = cms.string('ETLDigitizer'),
                inputSimHits = cms.InputTag("g4SimHits","FastTimerHitsEndcap"),
                maxSimHitsAccTime = cms.uint32(100),
                premixStage1 = cms.bool(False),
                premixStage1MaxCharge = cms.double(1000000.0),
                premixStage1MinCharge = cms.double(0.0001)
            ),
            makeDigiSimLinks = cms.bool(False),
            verbosity = cms.untracked.uint32(0)
        ),
        hcal = cms.PSet(
            DelivLuminosity = cms.double(0),
            HBDarkening = cms.bool(False),
            HEDarkening = cms.bool(False),
            HFDarkening = cms.bool(False),
            HFRecalParameterBlock = cms.PSet(
                HFdepthOneParameterA = cms.vdouble(
                    0.004123, 0.00602, 0.008201, 0.010489, 0.013379,
                    0.016997, 0.021464, 0.027371, 0.034195, 0.044807,
                    0.058939, 0.125497
                ),
                HFdepthOneParameterB = cms.vdouble(
                    -4e-06, -2e-06, 0.0, 4e-06, 1.5e-05,
                    2.6e-05, 6.3e-05, 8.4e-05, 0.00016, 0.000107,
                    0.000425, 0.000209
                ),
                HFdepthTwoParameterA = cms.vdouble(
                    0.002861, 0.004168, 0.0064, 0.008388, 0.011601,
                    0.014425, 0.018633, 0.023232, 0.028274, 0.035447,
                    0.051579, 0.086593
                ),
                HFdepthTwoParameterB = cms.vdouble(
                    -2e-06, -0.0, -7e-06, -6e-06, -2e-06,
                    1e-06, 1.9e-05, 3.1e-05, 6.7e-05, 1.2e-05,
                    0.000157, -3e-06
                )
            ),
            HcalPreMixStage1 = cms.bool(False),
            HcalPreMixStage2 = cms.bool(False),
            TestNumbering = cms.bool(True),
            accumulatorType = cms.string('HcalDigiProducer'),
            debugCaloSamples = cms.bool(False),
            doEmpty = cms.bool(True),
            doHFWindow = cms.bool(False),
            doIonFeedback = cms.bool(True),
            doNeutralDensityFilter = cms.bool(True),
            doNoise = cms.bool(True),
            doThermalNoise = cms.bool(True),
            doTimeSlew = cms.bool(True),
            hb = cms.PSet(
                binOfMaximum = cms.int32(6),
                delayQIE = cms.int32(-999),
                doPhotoStatistics = cms.bool(True),
                doSiPMSmearing = cms.bool(True),
                firstRing = cms.int32(1),
                readoutFrameSize = cms.int32(10),
                samplingFactors = cms.vdouble(
                    125.44, 125.54, 125.32, 125.13, 124.46,
                    125.01, 125.22, 125.48, 124.45, 125.9,
                    125.83, 127.01, 126.82, 129.73, 131.83,
                    143.52
                ),
                simHitToPhotoelectrons = cms.double(2000.0),
                sipmTau = cms.double(10.0),
                syncPhase = cms.bool(True),
                threshold_currentTDC = cms.double(18.7),
                timePhase = cms.double(6.0),
                timeSmearing = cms.bool(True)
            ),
            he = cms.PSet(
                binOfMaximum = cms.int32(6),
                delayQIE = cms.int32(-999),
                doPhotoStatistics = cms.bool(True),
                doSiPMSmearing = cms.bool(True),
                firstRing = cms.int32(16),
                readoutFrameSize = cms.int32(10),
                samplingFactors = cms.vdouble(
                    210.55, 197.93, 186.12, 189.64, 189.63,
                    189.96, 190.03, 190.11, 190.18, 190.25,
                    190.32, 190.4, 190.47, 190.54, 190.61,
                    190.69, 190.83, 190.94, 190.94, 190.94,
                    190.94, 190.94, 190.94, 190.94, 190.94,
                    190.94, 190.94, 190.94, 190.94, 190.94,
                    190.94, 190.94, 190.94, 190.94, 190.94,
                    190.94, 190.94, 190.94, 190.94, 190.94,
                    190.94, 190.94, 190.94, 190.94, 190.94,
                    190.94, 190.94, 190.94, 190.94, 190.94,
                    190.94, 190.94, 190.94, 190.94, 190.94,
                    190.94, 190.94, 190.94, 190.94, 190.94,
                    190.94, 190.94, 190.94, 190.94, 190.94,
                    190.94, 190.94, 190.94, 190.94, 190.94,
                    190.94, 190.94, 190.94, 190.94, 190.94,
                    190.94, 190.94, 190.94, 190.94, 190.94,
                    190.94, 190.94, 190.94, 190.94, 190.94
                ),
                simHitToPhotoelectrons = cms.double(2000.0),
                sipmTau = cms.double(10.0),
                syncPhase = cms.bool(True),
                threshold_currentTDC = cms.double(18.7),
                timePhase = cms.double(6.0),
                timeSmearing = cms.bool(True)
            ),
            hf1 = cms.PSet(
                binOfMaximum = cms.int32(2),
                delayQIE = cms.int32(-999),
                doPhotoStatistics = cms.bool(True),
                doSiPMSmearing = cms.bool(False),
                photoelectronsToAnalog = cms.double(2.79),
                readoutFrameSize = cms.int32(3),
                samplingFactor = cms.double(0.37),
                simHitToPhotoelectrons = cms.double(6.0),
                sipmTau = cms.double(0.0),
                syncPhase = cms.bool(True),
                threshold_currentTDC = cms.double(3.0),
                timePhase = cms.double(9.0)
            ),
            hf2 = cms.PSet(
                binOfMaximum = cms.int32(2),
                delayQIE = cms.int32(-999),
                doPhotoStatistics = cms.bool(True),
                doSiPMSmearing = cms.bool(False),
                photoelectronsToAnalog = cms.double(1.843),
                readoutFrameSize = cms.int32(3),
                samplingFactor = cms.double(0.37),
                simHitToPhotoelectrons = cms.double(6.0),
                sipmTau = cms.double(0.0),
                syncPhase = cms.bool(True),
                threshold_currentTDC = cms.double(3.0),
                timePhase = cms.double(8.0)
            ),
            hitsProducer = cms.string('g4SimHits'),
            ho = cms.PSet(
                binOfMaximum = cms.int32(5),
                delayQIE = cms.int32(-999),
                doPhotoStatistics = cms.bool(True),
                doSiPMSmearing = cms.bool(False),
                firstRing = cms.int32(1),
                readoutFrameSize = cms.int32(10),
                samplingFactors = cms.vdouble(
                    231.0, 231.0, 231.0, 231.0, 360.0,
                    360.0, 360.0, 360.0, 360.0, 360.0,
                    360.0, 360.0, 360.0, 360.0, 360.0
                ),
                siPMCode = cms.int32(1),
                simHitToPhotoelectrons = cms.double(4000.0),
                sipmTau = cms.double(5.0),
                syncPhase = cms.bool(True),
                threshold_currentTDC = cms.double(-999.0),
                timePhase = cms.double(5.0),
                timeSmearing = cms.bool(False)
            ),
            hoHamamatsu = cms.PSet(
                binOfMaximum = cms.int32(5),
                delayQIE = cms.int32(-999),
                doPhotoStatistics = cms.bool(True),
                doSiPMSmearing = cms.bool(False),
                firstRing = cms.int32(1),
                readoutFrameSize = cms.int32(10),
                samplingFactors = cms.vdouble(
                    231.0, 231.0, 231.0, 231.0, 360.0,
                    360.0, 360.0, 360.0, 360.0, 360.0,
                    360.0, 360.0, 360.0, 360.0, 360.0
                ),
                siPMCode = cms.int32(2),
                simHitToPhotoelectrons = cms.double(4000.0),
                sipmTau = cms.double(5.0),
                syncPhase = cms.bool(True),
                threshold_currentTDC = cms.double(-999.0),
                timePhase = cms.double(5.0),
                timeSmearing = cms.bool(False)
            ),
            hoZecotek = cms.PSet(
                binOfMaximum = cms.int32(5),
                delayQIE = cms.int32(-999),
                doPhotoStatistics = cms.bool(True),
                doSiPMSmearing = cms.bool(False),
                firstRing = cms.int32(1),
                readoutFrameSize = cms.int32(10),
                samplingFactors = cms.vdouble(
                    231.0, 231.0, 231.0, 231.0, 360.0,
                    360.0, 360.0, 360.0, 360.0, 360.0,
                    360.0, 360.0, 360.0, 360.0, 360.0
                ),
                siPMCode = cms.int32(2),
                simHitToPhotoelectrons = cms.double(4000.0),
                sipmTau = cms.double(5.0),
                syncPhase = cms.bool(True),
                threshold_currentTDC = cms.double(-999.0),
                timePhase = cms.double(5.0),
                timeSmearing = cms.bool(False)
            ),
            ignoreGeantTime = cms.bool(False),
            injectTestHits = cms.bool(False),
            injectTestHitsCells = cms.vint32(),
            injectTestHitsEnergy = cms.vdouble(),
            injectTestHitsTime = cms.vdouble(),
            killHE = cms.bool(True),
            makeDigiSimLinks = cms.untracked.bool(False),
            minFCToDelay = cms.double(5.0),
            zdc = cms.PSet(
                binOfMaximum = cms.int32(5),
                delayQIE = cms.int32(-999),
                doPhotoStatistics = cms.bool(True),
                doSiPMSmearing = cms.bool(False),
                photoelectronsToAnalog = cms.double(1.843),
                readoutFrameSize = cms.int32(10),
                samplingFactor = cms.double(1.0),
                simHitToPhotoelectrons = cms.double(6.0),
                sipmTau = cms.double(0.0),
                syncPhase = cms.bool(True),
                threshold_currentTDC = cms.double(-999.0),
                timePhase = cms.double(-4.0)
            )
        ),
        hgceeDigitizer = cms.PSet(
            NoiseGeneration_Method = cms.bool(True),
            accumulatorType = cms.string('HGCDigiProducer'),
            bxTime = cms.double(25),
            digiCfg = cms.PSet(
                cceParams = cms.PSet(
                    refToPSet_ = cms.string('HGCAL_cceParams_toUse')
                ),
                chargeCollectionEfficiencies = cms.PSet(
                    refToPSet_ = cms.string('HGCAL_chargeCollectionEfficiencies')
                ),
                doTimeSamples = cms.bool(False),
                feCfg = cms.PSet(
                    adcNbits = cms.uint32(10),
                    adcPulse = cms.vdouble(
                        0.0, 0.017, 0.817, 0.163, 0.003,
                        0.0
                    ),
                    adcSaturation_fC = cms.double(100),
                    adcThreshold_fC = cms.double(0.672),
                    fwVersion = cms.uint32(2),
                    jitterConstant_ns = cms.vdouble(0.02, 0.02, 0.02),
                    jitterNoise_ns = cms.vdouble(5.0, 5.0, 5.0),
                    pulseAvgT = cms.vdouble(
                        0.0, 23.42298, 13.16733, 6.41062, 5.03946,
                        4.532
                    ),
                    targetMIPvalue_ADC = cms.uint32(10),
                    tdcChargeDrainParameterisation = cms.vdouble(
                        -919.13, 365.36, -14.1, 0.2, -21.85,
                        49.39, 22.21, 0.8, -0.28, 27.14,
                        43.95, 3.89048
                    ),
                    tdcForToAOnset_fC = cms.vdouble(12.0, 12.0, 12.0),
                    tdcNbits = cms.uint32(12),
                    tdcOnset_fC = cms.double(60),
                    tdcResolutionInPs = cms.double(0.001),
                    tdcSaturation_fC = cms.double(10000),
                    toaLSB_ns = cms.double(0.0244),
                    toaMode = cms.uint32(1)
                ),
                ileakParam = cms.PSet(
                    refToPSet_ = cms.string('HGCAL_ileakParam_toUse')
                ),
                keV2fC = cms.double(0.044259),
                noise_fC = cms.PSet(
                    refToPSet_ = cms.string('HGCAL_noise_fC')
                ),
                thresholdFollowsMIP = cms.bool(True)
            ),
            digiCollection = cms.string('HGCDigisEE'),
            digitizationType = cms.uint32(0),
            digitizer = cms.string('HGCEEDigitizer'),
            eVPerEleHolePair = cms.double(3.62),
            hitCollection = cms.string('HGCHitsEE'),
            hitsProducer = cms.string('g4SimHits'),
            makeDigiSimLinks = cms.bool(False),
            maxSimHitsAccTime = cms.uint32(100),
            premixStage1 = cms.bool(False),
            premixStage1MaxCharge = cms.double(1000000.0),
            premixStage1MinCharge = cms.double(0),
            tofDelay = cms.double(-9),
            useAllChannels = cms.bool(True),
            verbosity = cms.untracked.uint32(0)
        ),
        hgchebackDigitizer = cms.PSet(
            NoiseGeneration_Method = cms.bool(True),
            accumulatorType = cms.string('HGCDigiProducer'),
            bxTime = cms.double(25),
            digiCfg = cms.PSet(
                algo = cms.uint32(2),
                doTimeSamples = cms.bool(False),
                feCfg = cms.PSet(
                    adcNbits = cms.uint32(10),
                    adcPulse = cms.vdouble(
                        0.0, 0.017, 0.817, 0.163, 0.003,
                        0.0
                    ),
                    adcSaturation_fC = cms.double(68.75),
                    adcThreshold_fC = cms.double(0.5),
                    fwVersion = cms.uint32(2),
                    jitterConstant_ns = cms.vdouble(0.02, 0.02, 0.02),
                    jitterNoise_ns = cms.vdouble(5.0, 5.0, 5.0),
                    pulseAvgT = cms.vdouble(
                        0.0, 23.42298, 13.16733, 6.41062, 5.03946,
                        4.532
                    ),
                    targetMIPvalue_ADC = cms.uint32(15),
                    tdcChargeDrainParameterisation = cms.vdouble(
                        -919.13, 365.36, -14.1, 0.2, -21.85,
                        49.39, 22.21, 0.8, -0.28, 27.14,
                        43.95, 3.89048
                    ),
                    tdcForToAOnset_fC = cms.vdouble(12.0, 12.0, 12.0),
                    tdcNbits = cms.uint32(12),
                    tdcOnset_fC = cms.double(55),
                    tdcResolutionInPs = cms.double(0.001),
                    tdcSaturation_fC = cms.double(1000),
                    toaLSB_ns = cms.double(0.0244),
                    toaMode = cms.uint32(1)
                ),
                keV2MIP = cms.double(0.0014814814814814814),
                nPEperMIP = cms.double(21.0),
                nTotalPE = cms.double(7500),
                noise = cms.PSet(
                    refToPSet_ = cms.string('HGCAL_noise_heback')
                ),
                sdPixels = cms.double(1e-06),
                thresholdFollowsMIP = cms.bool(True)
            ),
            digiCollection = cms.string('HGCDigisHEback'),
            digitizationType = cms.uint32(1),
            digitizer = cms.string('HGCHEbackDigitizer'),
            hitCollection = cms.string('HGCHitsHEback'),
            hitsProducer = cms.string('g4SimHits'),
            makeDigiSimLinks = cms.bool(False),
            maxSimHitsAccTime = cms.uint32(100),
            premixStage1 = cms.bool(False),
            premixStage1MaxCharge = cms.double(1000000.0),
            premixStage1MinCharge = cms.double(0),
            tofDelay = cms.double(-14),
            useAllChannels = cms.bool(True),
            verbosity = cms.untracked.uint32(0)
        ),
        hgchefrontDigitizer = cms.PSet(
            NoiseGeneration_Method = cms.bool(True),
            accumulatorType = cms.string('HGCDigiProducer'),
            bxTime = cms.double(25),
            digiCfg = cms.PSet(
                cceParams = cms.PSet(
                    refToPSet_ = cms.string('HGCAL_cceParams_toUse')
                ),
                chargeCollectionEfficiencies = cms.PSet(
                    refToPSet_ = cms.string('HGCAL_chargeCollectionEfficiencies')
                ),
                doTimeSamples = cms.bool(False),
                feCfg = cms.PSet(
                    adcNbits = cms.uint32(10),
                    adcPulse = cms.vdouble(
                        0.0, 0.017, 0.817, 0.163, 0.003,
                        0.0
                    ),
                    adcSaturation_fC = cms.double(100),
                    adcThreshold_fC = cms.double(0.672),
                    fwVersion = cms.uint32(2),
                    jitterConstant_ns = cms.vdouble(0.02, 0.02, 0.02),
                    jitterNoise_ns = cms.vdouble(5.0, 5.0, 5.0),
                    pulseAvgT = cms.vdouble(
                        0.0, 23.42298, 13.16733, 6.41062, 5.03946,
                        4.532
                    ),
                    targetMIPvalue_ADC = cms.uint32(10),
                    tdcChargeDrainParameterisation = cms.vdouble(
                        -919.13, 365.36, -14.1, 0.2, -21.85,
                        49.39, 22.21, 0.8, -0.28, 27.14,
                        43.95, 3.89048
                    ),
                    tdcForToAOnset_fC = cms.vdouble(12.0, 12.0, 12.0),
                    tdcNbits = cms.uint32(12),
                    tdcOnset_fC = cms.double(60),
                    tdcResolutionInPs = cms.double(0.001),
                    tdcSaturation_fC = cms.double(10000),
                    toaLSB_ns = cms.double(0.0244),
                    toaMode = cms.uint32(1)
                ),
                ileakParam = cms.PSet(
                    refToPSet_ = cms.string('HGCAL_ileakParam_toUse')
                ),
                keV2fC = cms.double(0.044259),
                noise_fC = cms.PSet(
                    refToPSet_ = cms.string('HGCAL_noise_fC')
                ),
                thresholdFollowsMIP = cms.bool(True)
            ),
            digiCollection = cms.string('HGCDigisHEfront'),
            digitizationType = cms.uint32(0),
            digitizer = cms.string('HGCHEfrontDigitizer'),
            hitCollection = cms.string('HGCHitsHEfront'),
            hitsProducer = cms.string('g4SimHits'),
            makeDigiSimLinks = cms.bool(False),
            maxSimHitsAccTime = cms.uint32(100),
            premixStage1 = cms.bool(False),
            premixStage1MaxCharge = cms.double(1000000.0),
            premixStage1MinCharge = cms.double(0),
            tofDelay = cms.double(-11),
            useAllChannels = cms.bool(True),
            verbosity = cms.untracked.uint32(0)
        ),
        mergedtruth = cms.PSet(
            HepMCProductLabel = cms.InputTag("generatorSmeared"),
            accumulatorType = cms.string('TrackingTruthAccumulator'),
            allowDifferentSimHitProcesses = cms.bool(False),
            alwaysAddAncestors = cms.bool(True),
            createInitialVertexCollection = cms.bool(True),
            createMergedBremsstrahlung = cms.bool(True),
            createUnmergedCollection = cms.bool(True),
            genParticleCollection = cms.InputTag("genParticles"),
            ignoreTracksOutsideVolume = cms.bool(False),
            maximumPreviousBunchCrossing = cms.uint32(9999),
            maximumSubsequentBunchCrossing = cms.uint32(9999),
            removeDeadModules = cms.bool(False),
            select = cms.PSet(
                chargedOnlyTP = cms.bool(True),
                intimeOnlyTP = cms.bool(False),
                lipTP = cms.double(1000),
                maxRapidityTP = cms.double(5.0),
                minHitTP = cms.int32(0),
                minRapidityTP = cms.double(-5.0),
                pdgIdTP = cms.vint32(),
                ptMaxTP = cms.double(1e+100),
                ptMinTP = cms.double(0.1),
                signalOnlyTP = cms.bool(False),
                stableOnlyTP = cms.bool(False),
                tipTP = cms.double(1000)
            ),
            simHitCollections = cms.PSet(
                muon = cms.VInputTag(cms.InputTag("g4SimHits","MuonDTHits"), cms.InputTag("g4SimHits","MuonCSCHits"), cms.InputTag("g4SimHits","MuonRPCHits"), cms.InputTag("g4SimHits","MuonGEMHits")),
                pixel = cms.VInputTag(cms.InputTag("g4SimHits","TrackerHitsPixelBarrelLowTof"), cms.InputTag("g4SimHits","TrackerHitsPixelBarrelHighTof"), cms.InputTag("g4SimHits","TrackerHitsPixelEndcapLowTof"), cms.InputTag("g4SimHits","TrackerHitsPixelEndcapHighTof")),
                tracker = cms.VInputTag()
            ),
            simTrackCollection = cms.InputTag("g4SimHits"),
            simVertexCollection = cms.InputTag("g4SimHits"),
            vertexDistanceCut = cms.double(0.003),
            volumeRadius = cms.double(120.0),
            volumeZ = cms.double(300.0)
        ),
        pixel = cms.PSet(
            AlgorithmCommon = cms.PSet(
                DeltaProductionCut = cms.double(0.03),
                makeDigiSimLinks = cms.untracked.bool(True)
            ),
            GeometryType = cms.string('idealForDigi'),
            PSPDigitizerAlgorithm = cms.PSet(
                AdcFullScale = cms.int32(255),
                AddBiasRailInefficiency = cms.bool(False),
                AddInefficiency = cms.bool(False),
                AddNoise = cms.bool(True),
                AddNoisyPixels = cms.bool(True),
                AddThresholdSmearing = cms.bool(True),
                AddXTalk = cms.bool(True),
                Alpha2Order = cms.bool(True),
                CellsToKill = cms.VPSet(),
                ClusterWidth = cms.double(3),
                DeadModules = cms.VPSet(),
                DeadModules_DB = cms.bool(False),
                EfficiencyFactors_Barrel = cms.vdouble(
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999, 0.999, 0.999, 0.999, 0.999
                ),
                EfficiencyFactors_Endcap = cms.vdouble(
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999
                ),
                ElectronPerAdc = cms.double(135.0),
                HIPThresholdInElectrons_Barrel = cms.double(10000000000.0),
                HIPThresholdInElectrons_Endcap = cms.double(10000000000.0),
                Inefficiency_DB = cms.bool(False),
                InterstripCoupling = cms.double(0.05),
                KillModules = cms.bool(False),
                LorentzAngle_DB = cms.bool(True),
                NoiseInElectrons = cms.double(200),
                Phase2ReadoutMode = cms.int32(0),
                ReadoutNoiseInElec = cms.double(200.0),
                SigmaCoeff = cms.double(1.8),
                SigmaZero = cms.double(0.00037),
                TanLorentzAnglePerTesla_Barrel = cms.double(0.07),
                TanLorentzAnglePerTesla_Endcap = cms.double(0.07),
                ThresholdInElectrons_Barrel = cms.double(6300.0),
                ThresholdInElectrons_Endcap = cms.double(6300.0),
                ThresholdSmearing_Barrel = cms.double(630.0),
                ThresholdSmearing_Endcap = cms.double(630.0),
                TofLowerCut = cms.double(-12.5),
                TofUpperCut = cms.double(12.5)
            ),
            PSSDigitizerAlgorithm = cms.PSet(
                AdcFullScale = cms.int32(255),
                AddInefficiency = cms.bool(False),
                AddNoise = cms.bool(True),
                AddNoisyPixels = cms.bool(True),
                AddThresholdSmearing = cms.bool(True),
                AddXTalk = cms.bool(True),
                Alpha2Order = cms.bool(True),
                CellsToKill = cms.VPSet(),
                ClusterWidth = cms.double(3),
                DeadModules = cms.VPSet(),
                DeadModules_DB = cms.bool(False),
                EfficiencyFactors_Barrel = cms.vdouble(
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999, 0.999, 0.999, 0.999, 0.999
                ),
                EfficiencyFactors_Endcap = cms.vdouble(
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999
                ),
                ElectronPerAdc = cms.double(135.0),
                HIPThresholdInElectrons_Barrel = cms.double(21000.0),
                HIPThresholdInElectrons_Endcap = cms.double(21000.0),
                Inefficiency_DB = cms.bool(False),
                InterstripCoupling = cms.double(0.05),
                KillModules = cms.bool(False),
                LorentzAngle_DB = cms.bool(True),
                NoiseInElectrons = cms.double(700),
                Phase2ReadoutMode = cms.int32(0),
                ReadoutNoiseInElec = cms.double(700.0),
                SigmaCoeff = cms.double(1.8),
                SigmaZero = cms.double(0.00037),
                TanLorentzAnglePerTesla_Barrel = cms.double(0.07),
                TanLorentzAnglePerTesla_Endcap = cms.double(0.07),
                ThresholdInElectrons_Barrel = cms.double(6300.0),
                ThresholdInElectrons_Endcap = cms.double(6300.0),
                ThresholdSmearing_Barrel = cms.double(630.0),
                ThresholdSmearing_Endcap = cms.double(630.0),
                TofLowerCut = cms.double(-12.5),
                TofUpperCut = cms.double(12.5)
            ),
            Pixel3DDigitizerAlgorithm = cms.PSet(
                AdcFullScale = cms.int32(15),
                AddInefficiency = cms.bool(False),
                AddNoise = cms.bool(False),
                AddNoisyPixels = cms.bool(False),
                AddThresholdSmearing = cms.bool(False),
                AddXTalk = cms.bool(False),
                Alpha2Order = cms.bool(True),
                ApplyTimewalk = cms.bool(False),
                CellsToKill = cms.VPSet(),
                ClusterWidth = cms.double(3),
                DeadModules = cms.VPSet(),
                DeadModules_DB = cms.bool(False),
                EfficiencyFactors_Barrel = cms.vdouble(
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999, 0.999, 0.999, 0.999, 0.999
                ),
                EfficiencyFactors_Endcap = cms.vdouble(
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999
                ),
                ElectronPerAdc = cms.double(1500.0),
                Even_column_interchannelCoupling_next_column = cms.double(0.0),
                Even_row_interchannelCoupling_next_row = cms.double(0.0),
                HIPThresholdInElectrons_Barrel = cms.double(10000000000.0),
                HIPThresholdInElectrons_Endcap = cms.double(10000000000.0),
                Inefficiency_DB = cms.bool(False),
                InterstripCoupling = cms.double(0.0),
                KillModules = cms.bool(False),
                LorentzAngle_DB = cms.bool(True),
                NPColumnGap = cms.double(46.0),
                NPColumnRadius = cms.double(4.0),
                NoiseInElectrons = cms.double(0.0),
                Odd_column_interchannelCoupling_next_column = cms.double(0.0),
                Odd_row_interchannelCoupling_next_row = cms.double(0.2),
                OhmicColumnRadius = cms.double(4.0),
                Phase2ReadoutMode = cms.int32(3),
                ReadoutNoiseInElec = cms.double(0.0),
                SigmaCoeff = cms.double(1.8),
                SigmaZero = cms.double(0.00037),
                TanLorentzAnglePerTesla_Barrel = cms.double(0.106),
                TanLorentzAnglePerTesla_Endcap = cms.double(0.106),
                ThresholdInElectrons_Barrel = cms.double(1000.0),
                ThresholdInElectrons_Endcap = cms.double(1000.0),
                ThresholdSmearing_Barrel = cms.double(0.0),
                ThresholdSmearing_Endcap = cms.double(0.0),
                TimewalkModel = cms.PSet(
                    Curves = cms.VPSet(
                        cms.PSet(
                            charge = cms.vdouble(
                                1000, 1025, 1050, 1100, 1200,
                                1500, 2000, 6000, 10000, 15000,
                                20000, 30000
                            ),
                            delay = cms.vdouble(
                                26.8, 23.73, 21.92, 19.46, 16.52,
                                12.15, 8.88, 3.03, 1.69, 0.95,
                                0.56, 0.19
                            )
                        ),
                        cms.PSet(
                            charge = cms.vdouble(
                                1200, 1225, 1250, 1500, 2000,
                                6000, 10000, 15000, 20000, 30000
                            ),
                            delay = cms.vdouble(
                                26.28, 23.5, 21.79, 14.92, 10.27,
                                3.33, 1.86, 1.07, 0.66, 0.27
                            )
                        ),
                        cms.PSet(
                            charge = cms.vdouble(
                                1500, 1525, 1550, 1600, 2000,
                                6000, 10000, 15000, 20000, 30000
                            ),
                            delay = cms.vdouble(
                                25.36, 23.05, 21.6, 19.56, 12.94,
                                3.79, 2.14, 1.26, 0.81, 0.39
                            )
                        ),
                        cms.PSet(
                            charge = cms.vdouble(
                                3000, 3025, 3050, 3100, 3500,
                                6000, 10000, 15000, 20000, 30000
                            ),
                            delay = cms.vdouble(
                                25.63, 23.63, 22.35, 20.65, 14.92,
                                6.7, 3.68, 2.29, 1.62, 1.02
                            )
                        )
                    ),
                    ThresholdValues = cms.vdouble(1000, 1200, 1500, 3000)
                ),
                TofLowerCut = cms.double(-12.5),
                TofUpperCut = cms.double(12.5)
            ),
            PixelBrickedDigitizerAlgorithm = cms.PSet(
                AdcFullScale = cms.int32(15),
                AddInefficiency = cms.bool(False),
                AddNoise = cms.bool(False),
                AddNoisyPixels = cms.bool(False),
                AddThresholdSmearing = cms.bool(False),
                AddXTalk = cms.bool(False),
                Alpha2Order = cms.bool(True),
                ApplyTimewalk = cms.bool(False),
                CellsToKill = cms.VPSet(),
                ClusterWidth = cms.double(3),
                DeadModules = cms.VPSet(),
                DeadModules_DB = cms.bool(False),
                EfficiencyFactors_Barrel = cms.vdouble(
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999, 0.999, 0.999, 0.999, 0.999
                ),
                EfficiencyFactors_Endcap = cms.vdouble(
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999
                ),
                ElectronPerAdc = cms.double(1500.0),
                Even_column_interchannelCoupling_next_column = cms.double(0.0),
                Even_row_interchannelCoupling_next_row = cms.double(0.0),
                HIPThresholdInElectrons_Barrel = cms.double(10000000000.0),
                HIPThresholdInElectrons_Endcap = cms.double(10000000000.0),
                Inefficiency_DB = cms.bool(False),
                InterstripCoupling = cms.double(0.0),
                KillModules = cms.bool(False),
                LorentzAngle_DB = cms.bool(True),
                NoiseInElectrons = cms.double(0.0),
                Odd_column_interchannelCoupling_next_column = cms.double(0.0),
                Odd_row_interchannelCoupling_next_row = cms.double(0.2),
                Phase2ReadoutMode = cms.int32(3),
                ReadoutNoiseInElec = cms.double(0.0),
                SigmaCoeff = cms.double(0),
                SigmaZero = cms.double(0.00037),
                TanLorentzAnglePerTesla_Barrel = cms.double(0.106),
                TanLorentzAnglePerTesla_Endcap = cms.double(0.106),
                ThresholdInElectrons_Barrel = cms.double(1000.0),
                ThresholdInElectrons_Endcap = cms.double(1000.0),
                ThresholdSmearing_Barrel = cms.double(0.0),
                ThresholdSmearing_Endcap = cms.double(0.0),
                TimewalkModel = cms.PSet(
                    Curves = cms.VPSet(
                        cms.PSet(
                            charge = cms.vdouble(
                                1000, 1025, 1050, 1100, 1200,
                                1500, 2000, 6000, 10000, 15000,
                                20000, 30000
                            ),
                            delay = cms.vdouble(
                                26.8, 23.73, 21.92, 19.46, 16.52,
                                12.15, 8.88, 3.03, 1.69, 0.95,
                                0.56, 0.19
                            )
                        ),
                        cms.PSet(
                            charge = cms.vdouble(
                                1200, 1225, 1250, 1500, 2000,
                                6000, 10000, 15000, 20000, 30000
                            ),
                            delay = cms.vdouble(
                                26.28, 23.5, 21.79, 14.92, 10.27,
                                3.33, 1.86, 1.07, 0.66, 0.27
                            )
                        ),
                        cms.PSet(
                            charge = cms.vdouble(
                                1500, 1525, 1550, 1600, 2000,
                                6000, 10000, 15000, 20000, 30000
                            ),
                            delay = cms.vdouble(
                                25.36, 23.05, 21.6, 19.56, 12.94,
                                3.79, 2.14, 1.26, 0.81, 0.39
                            )
                        ),
                        cms.PSet(
                            charge = cms.vdouble(
                                3000, 3025, 3050, 3100, 3500,
                                6000, 10000, 15000, 20000, 30000
                            ),
                            delay = cms.vdouble(
                                25.63, 23.63, 22.35, 20.65, 14.92,
                                6.7, 3.68, 2.29, 1.62, 1.02
                            )
                        )
                    ),
                    ThresholdValues = cms.vdouble(1000, 1200, 1500, 3000)
                ),
                TofLowerCut = cms.double(-12.5),
                TofUpperCut = cms.double(12.5)
            ),
            PixelDigitizerAlgorithm = cms.PSet(
                AdcFullScale = cms.int32(15),
                AddInefficiency = cms.bool(False),
                AddNoise = cms.bool(False),
                AddNoisyPixels = cms.bool(False),
                AddThresholdSmearing = cms.bool(False),
                AddXTalk = cms.bool(False),
                Alpha2Order = cms.bool(True),
                ApplyTimewalk = cms.bool(False),
                CellsToKill = cms.VPSet(),
                ClusterWidth = cms.double(3),
                DeadModules = cms.VPSet(),
                DeadModules_DB = cms.bool(False),
                EfficiencyFactors_Barrel = cms.vdouble(
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999, 0.999, 0.999, 0.999, 0.999
                ),
                EfficiencyFactors_Endcap = cms.vdouble(
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999
                ),
                ElectronPerAdc = cms.double(1500.0),
                Even_column_interchannelCoupling_next_column = cms.double(0.0),
                Even_row_interchannelCoupling_next_row = cms.double(0.0),
                HIPThresholdInElectrons_Barrel = cms.double(10000000000.0),
                HIPThresholdInElectrons_Endcap = cms.double(10000000000.0),
                Inefficiency_DB = cms.bool(False),
                InterstripCoupling = cms.double(0.0),
                KillModules = cms.bool(False),
                LorentzAngle_DB = cms.bool(True),
                NoiseInElectrons = cms.double(0.0),
                Odd_column_interchannelCoupling_next_column = cms.double(0.0),
                Odd_row_interchannelCoupling_next_row = cms.double(0.2),
                Phase2ReadoutMode = cms.int32(3),
                ReadoutNoiseInElec = cms.double(0.0),
                SigmaCoeff = cms.double(0),
                SigmaZero = cms.double(0.00037),
                TanLorentzAnglePerTesla_Barrel = cms.double(0.106),
                TanLorentzAnglePerTesla_Endcap = cms.double(0.106),
                ThresholdInElectrons_Barrel = cms.double(1000.0),
                ThresholdInElectrons_Endcap = cms.double(1000.0),
                ThresholdSmearing_Barrel = cms.double(0.0),
                ThresholdSmearing_Endcap = cms.double(0.0),
                TimewalkModel = cms.PSet(
                    Curves = cms.VPSet(
                        cms.PSet(
                            charge = cms.vdouble(
                                1000, 1025, 1050, 1100, 1200,
                                1500, 2000, 6000, 10000, 15000,
                                20000, 30000
                            ),
                            delay = cms.vdouble(
                                26.8, 23.73, 21.92, 19.46, 16.52,
                                12.15, 8.88, 3.03, 1.69, 0.95,
                                0.56, 0.19
                            )
                        ),
                        cms.PSet(
                            charge = cms.vdouble(
                                1200, 1225, 1250, 1500, 2000,
                                6000, 10000, 15000, 20000, 30000
                            ),
                            delay = cms.vdouble(
                                26.28, 23.5, 21.79, 14.92, 10.27,
                                3.33, 1.86, 1.07, 0.66, 0.27
                            )
                        ),
                        cms.PSet(
                            charge = cms.vdouble(
                                1500, 1525, 1550, 1600, 2000,
                                6000, 10000, 15000, 20000, 30000
                            ),
                            delay = cms.vdouble(
                                25.36, 23.05, 21.6, 19.56, 12.94,
                                3.79, 2.14, 1.26, 0.81, 0.39
                            )
                        ),
                        cms.PSet(
                            charge = cms.vdouble(
                                3000, 3025, 3050, 3100, 3500,
                                6000, 10000, 15000, 20000, 30000
                            ),
                            delay = cms.vdouble(
                                25.63, 23.63, 22.35, 20.65, 14.92,
                                6.7, 3.68, 2.29, 1.62, 1.02
                            )
                        )
                    ),
                    ThresholdValues = cms.vdouble(1000, 1200, 1500, 3000)
                ),
                TofLowerCut = cms.double(-12.5),
                TofUpperCut = cms.double(12.5)
            ),
            ROUList = cms.vstring(
                'TrackerHitsPixelBarrelLowTof',
                'TrackerHitsPixelBarrelHighTof',
                'TrackerHitsPixelEndcapLowTof',
                'TrackerHitsPixelEndcapHighTof'
            ),
            SSDigitizerAlgorithm = cms.PSet(
                AdcFullScale = cms.int32(255),
                AddInefficiency = cms.bool(False),
                AddNoise = cms.bool(True),
                AddNoisyPixels = cms.bool(True),
                AddThresholdSmearing = cms.bool(True),
                AddXTalk = cms.bool(True),
                Alpha2Order = cms.bool(True),
                CBCDeadTime = cms.double(0.0),
                CellsToKill = cms.VPSet(),
                ClusterWidth = cms.double(3),
                DeadModules = cms.VPSet(),
                DeadModules_DB = cms.bool(False),
                EfficiencyFactors_Barrel = cms.vdouble(
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999, 0.999, 0.999, 0.999, 0.999
                ),
                EfficiencyFactors_Endcap = cms.vdouble(
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999, 0.999, 0.999, 0.999, 0.999,
                    0.999
                ),
                ElectronPerAdc = cms.double(135.0),
                HIPThresholdInElectrons_Barrel = cms.double(10000000000.0),
                HIPThresholdInElectrons_Endcap = cms.double(10000000000.0),
                HitDetectionMode = cms.int32(0),
                Inefficiency_DB = cms.bool(False),
                InterstripCoupling = cms.double(0.05),
                KillModules = cms.bool(False),
                LorentzAngle_DB = cms.bool(True),
                NoiseInElectrons = cms.double(1000),
                Phase2ReadoutMode = cms.int32(0),
                PulseShapeParameters = cms.vdouble(
                    -3.0, 16.043703, 99.999857, 40.57165, 2.0,
                    1.2459094
                ),
                ReadoutNoiseInElec = cms.double(1000.0),
                SigmaCoeff = cms.double(1.8),
                SigmaZero = cms.double(0.00037),
                TanLorentzAnglePerTesla_Barrel = cms.double(0.07),
                TanLorentzAnglePerTesla_Endcap = cms.double(0.07),
                ThresholdInElectrons_Barrel = cms.double(5800.0),
                ThresholdInElectrons_Endcap = cms.double(5800.0),
                ThresholdSmearing_Barrel = cms.double(580.0),
                ThresholdSmearing_Endcap = cms.double(580.0),
                TofLowerCut = cms.double(-12.5),
                TofUpperCut = cms.double(12.5)
            ),
            accumulatorType = cms.string('Phase2TrackerDigitizer'),
            hitsProducer = cms.string('g4SimHits'),
            isOTreadoutAnalog = cms.bool(False),
            premixStage1 = cms.bool(False)
        ),
        puVtx = cms.PSet(
            accumulatorType = cms.string('PileupVertexAccumulator'),
            hitsProducer = cms.string('generator'),
            makeDigiSimLinks = cms.untracked.bool(False),
            saveVtxTimes = cms.bool(True),
            vtxFallbackTag = cms.InputTag("generator"),
            vtxTag = cms.InputTag("generatorSmeared")
        )
    ),
    maxBunch = cms.int32(3),
    minBunch = cms.int32(-5),
    mixObjects = cms.PSet(
        mixCH = cms.PSet(
            crossingFrames = cms.untracked.vstring(),
            input = cms.VInputTag(
                cms.InputTag("g4SimHits","CastorFI"), cms.InputTag("g4SimHits","EcalHitsEB"), cms.InputTag("g4SimHits","EcalHitsEE"), cms.InputTag("g4SimHits","EcalHitsES"), cms.InputTag("g4SimHits","HcalHits"),
                cms.InputTag("g4SimHits","ZDCHITS"), cms.InputTag("g4SimHits","HGCHitsEE"), cms.InputTag("g4SimHits","HGCHitsHEfront"), cms.InputTag("g4SimHits","HGCHitsHEback")
            ),
            subdets = cms.vstring(
                'CastorFI',
                'EcalHitsEB',
                'EcalHitsEE',
                'EcalHitsES',
                'HcalHits',
                'ZDCHITS',
                'HGCHitsEE',
                'HGCHitsHEfront',
                'HGCHitsHEback'
            ),
            type = cms.string('PCaloHit')
        ),
        mixHepMC = cms.PSet(
            input = cms.VInputTag(cms.InputTag("generatorSmeared"), cms.InputTag("generator")),
            makeCrossingFrame = cms.untracked.bool(True),
            type = cms.string('HepMCProduct')
        ),
        mixSH = cms.PSet(
            crossingFrames = cms.untracked.vstring(
                'MuonCSCHits',
                'MuonDTHits',
                'MuonRPCHits',
                'MuonGEMHits',
                'FastTimerHitsBarrel',
                'FastTimerHitsEndcap'
            ),
            input = cms.VInputTag(
                cms.InputTag("g4SimHits","MuonCSCHits"), cms.InputTag("g4SimHits","MuonDTHits"), cms.InputTag("g4SimHits","MuonRPCHits"), cms.InputTag("g4SimHits","TrackerHitsPixelBarrelHighTof"), cms.InputTag("g4SimHits","TrackerHitsPixelBarrelLowTof"),
                cms.InputTag("g4SimHits","TrackerHitsPixelEndcapHighTof"), cms.InputTag("g4SimHits","TrackerHitsPixelEndcapLowTof"), cms.InputTag("g4SimHits","TrackerHitsTECHighTof"), cms.InputTag("g4SimHits","TrackerHitsTECLowTof"), cms.InputTag("g4SimHits","TrackerHitsTIBHighTof"),
                cms.InputTag("g4SimHits","TrackerHitsTIBLowTof"), cms.InputTag("g4SimHits","TrackerHitsTIDHighTof"), cms.InputTag("g4SimHits","TrackerHitsTIDLowTof"), cms.InputTag("g4SimHits","TrackerHitsTOBHighTof"), cms.InputTag("g4SimHits","TrackerHitsTOBLowTof"),
                cms.InputTag("g4SimHits","MuonGEMHits"), cms.InputTag("g4SimHits","FastTimerHitsBarrel"), cms.InputTag("g4SimHits","FastTimerHitsEndcap")
            ),
            pcrossingFrames = cms.untracked.vstring(),
            subdets = cms.vstring(
                'MuonCSCHits',
                'MuonDTHits',
                'MuonRPCHits',
                'TrackerHitsPixelBarrelHighTof',
                'TrackerHitsPixelBarrelLowTof',
                'TrackerHitsPixelEndcapHighTof',
                'TrackerHitsPixelEndcapLowTof',
                'TrackerHitsTECHighTof',
                'TrackerHitsTECLowTof',
                'TrackerHitsTIBHighTof',
                'TrackerHitsTIBLowTof',
                'TrackerHitsTIDHighTof',
                'TrackerHitsTIDLowTof',
                'TrackerHitsTOBHighTof',
                'TrackerHitsTOBLowTof',
                'MuonGEMHits',
                'FastTimerHitsBarrel',
                'FastTimerHitsEndcap'
            ),
            type = cms.string('PSimHit')
        ),
        mixTracks = cms.PSet(
            input = cms.VInputTag(cms.InputTag("g4SimHits")),
            makeCrossingFrame = cms.untracked.bool(False),
            type = cms.string('SimTrack')
        ),
        mixVertices = cms.PSet(
            input = cms.VInputTag(cms.InputTag("g4SimHits")),
            makeCrossingFrame = cms.untracked.bool(False),
            type = cms.string('SimVertex')
        )
    ),
    mixProdStep1 = cms.bool(False),
    mixProdStep2 = cms.bool(False),
    playback = cms.untracked.bool(False),
    useCurrentProcessOnly = cms.bool(False)
)
Dr15Jones commented 1 year ago

So in this workflow, step1 does the GEN+SIM step and then step 2 (where the problem is seen) does the digi step. It looks like the SIM step thinks the geometry of the detector is different from what the SIM step thinks. I checked the parameters passed to cmsDriver.py in both steps, and they seem consistent

step 1

# with command line options: TTbar_14TeV_TuneCP5_cfi -s GEN,SIM -n 10 --conditions auto:phase2_realistic_T21 --beamspot HLLHC14TeV --datatier GEN-SIM --eventcontent FEVTDEBUG --geometry DD4hepExtended2026D88 --era Phase2C17I13M9 --procModifiers dd4hep --relval 9000,100 --fileout file:step1.root --nThreads 4

step 2

# with command line options: step2 -s DIGI:pdigi_valid,L1TrackTrigger,L1,DIGI2RAW,HLT:@fake2 --conditions auto:phase2_realistic_T21 --datatier GEN-SIM-DIGI-RAW -n 10 --eventcontent FEVTDEBUGHLT --geometry DD4hepExtended2026D88 --era Phase2C17I13M9 --procModifiers dd4hep --filein file:step1.root --fileout file:step2.root --nThreads 4
ianna commented 1 year ago

IMHO, we must move this and all other constants as records to DB and pick them up from a GT.

srimanob commented 1 year ago

I believe they are using a pileup file from runs 1,2, or 3 for phase2 where the ieta value is restricted to 16

Umm, I'd expect runTheMatrix workflows to have consistent pileup files (@cms-sw/pdmv-l2).

The pileup of Phase-2 is controlled by https://github.com/cms-sw/cmssw/blob/master/Configuration/PyReleaseValidation/python/relval_steps.py#L3730-L3733 i.e. defaultDataSets['2026D88']='CMSSW_12_3_0_pre5-123X_mcRun4_realistic_v4_2026D88noPU-v'

So, if you are running runTheMatrix with 12_6_0_pre4, it will pick MinBias GS from CMSSW_12_3_0_pre5, /RelValMinBias_14TeV/CMSSW_12_3_0_pre5-123X_mcRun4_realistic_v4_2026D88noPU-v1/GEN-SIM You can check from runTheMatrix.py --what upgrade -l 21034.0 --wm init

cmsDriver.py step2 -s DIGI:pdigi_valid,L1TrackTrigger,L1,DIGI2RAW,HLT:@fake2 --conditions auto:phase2_realistic_T21 --datatier GEN-SIM-DIGI-RAW -n 10 --eventcontent FEVTDEBUGHLT --geometry Extended2026D88 --era Phase2C17I13M9 --pileup AVE_200_BX_25ns --pileup_input das:/RelValMinBias_14TeV/CMSSW_12_3_0_pre5-123X_mcRun4_realistic_v4_2026D88noPU-v1/GEN-SIM --io DigiTriggerPU_2026D88PU.io --python DigiTriggerPU_2026D88PU.py --no_exec --filein file:step1.root --fileout file:step2.root > step2_TTbar_14TeV+2026D88PU+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTriggerPU+RecoGlobalPU+HARVESTGlobalPU.log 2>&1

If GS from 12_3_0_pre5 is different than 12_6_0_pre4, which I assume they are different, then we need to update the lines mention above.

Should this be updated, and see if we still have issue?

makortel commented 1 year ago

Should this be updated, and see if we still have issue?

The failing workflows do not use pileup, so I don't see how their update would help. (written that, there may be other reasons to justify the update)

srimanob commented 1 year ago

Hi @makortel OK.

I think for wf point-of-view, i.e. to get consistent geometry of pileup wf, this maybe good to update.

makortel commented 1 year ago

Could the heap-buffer-overlow in HcalDDDSimConstants::getPhiCons() occurring in SIM step, reported in the issue description, be a smoking gun for the root cause? I'm wondering if that could lead to corrupt output that then causes the trouble in DIGI?

bsunanda commented 1 year ago

This exception happens in the SIM step (step1) and not in the DIGI step (step2). So clearly eta assignment gets wrong even at the time of assigning it - does it happen for all CMSSW architectures and only for Phase2 dd4hep case?

srimanob commented 1 year ago

Hi @bsunanda @perrotta

I think my conclusion in last ORP was wrong if I read original report by @dan131riley https://github.com/cms-sw/cmssw/issues/39480#issue-1382477184 and also comment from @makortel https://github.com/cms-sw/cmssw/issues/39480#issuecomment-1297062002. The issue also sees in DDD workflow, i.e. in 23634.0 (it is NoPU D95 wf, 2026D95+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal+ALCAPhase2)

From https://github.com/cms-sw/cmssw/issues/39445#issuecomment-1255068447 reports that in normal IB, the issue appears randomly. One should try to run on ASAN IB.

perrotta commented 1 year ago

urgent (marking urgent the issues affecting relvals in the IBs)

makortel commented 1 year ago

@bsunanda

The exception (introduced in https://github.com/cms-sw/cmssw/pull/39920) appears to be occurring only in workflow 20834.911 step 2, i.e. Phase 2 D88 DD4Hep without pileup, and pretty consistently across all IB flavors.

The ASAN failure (mentioned in the description of this issue) appears to be occurring in the following workflows in a random fashion (i.e. each of the workflow fails in some ASAN IBs, but not in all of them)

(note that an execution not triggering the ASAN failure does not mean all would be well, it could easily be that the ASAN's instrumentation just fails to catch a read from incorrect address).

bsunanda commented 1 year ago

Dear All,

Most likely I found the bug. But this bug has been there from the very first days of CMSSW and I believe it has affected all workflows from Run1 onward. It only gave wrong results and did not cause the crash till we removed HE. I am trying to make a PR out of this. Regards Sunanda

From: Matti Kortelainen @.*** Sent: 02 November 2022 18:37 To: cms-sw/cmsswDear Cc: Sunanda Banerjee; Mention Subject: Re: [cms-sw/cmssw] heap-buffer-overflow HcalDDDSimConstants::getPhiCons() (Issue #39480)

@bsunandahttps://github.com/bsunanda

The exception (introduced in #39920https://github.com/cms-sw/cmssw/pull/39920) appears to be occurring only in workflow 20834.911 step 2, i.e. Phase 2 D88 DD4Hep without pileup, and pretty consistently across all IB flavors.

The ASAN failure (mentioned in the description of this issue) appears to be occurring in the following workflows in a random fashion (i.e. each of the workflow fails in some ASAN IBs, but not in all of them)

(note that an execution not triggering the ASAN failure does not mean all would be well, it could easily be that the ASAN's instrumentation just fails to catch a read from incorrect address).

— Reply to this email directly, view it on GitHubhttps://github.com/cms-sw/cmssw/issues/39480#issuecomment-1300997912, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABGMZOUNUQGGZFPJDV5FGFLWGKRHDANCNFSM6AAAAAAQTCGRMM. You are receiving this because you were mentioned.Message ID: @.***>

makortel commented 1 year ago

This problem got fixed by https://github.com/cms-sw/cmssw/pull/39967

makortel commented 1 year ago

@cmsbuild, please close