search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.09k stars 4.33k forks source link

Non-reproducibility in TriggerResults in 24834.0, 24834.911, and 25034.999 #43790

Open makortel opened 10 months ago

makortel commented 10 months ago

It seems like there is a recent non-reproducibility in the TriggerResults comparisons of the step2 of the workflows 24834.0, 24834.911, and 25034.999 (all of them use D98). Here is an example from https://github.com/cms-sw/cmssw/pull/43758#issuecomment-1910555392 image

Here is another occurrence https://github.com/cms-sw/cmssw/pull/43738#issuecomment-1900186136

makortel commented 10 months ago

assign upgrade, hlt

cmsbuild commented 10 months ago

New categories assigned: upgrade,hlt

@Martin-Grunewald,@mmusich,@srimanob,@subirsarkar you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild commented 10 months ago

cms-bot internal usage

cmsbuild commented 10 months ago

A new Issue was created by @makortel Matti Kortelainen.

@antoniovilela, @smuzaffar, @rappoccio, @makortel, @sextonkennedy, @Dr15Jones can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel commented 10 months ago

Visible e.g. in

but not in CMSSW_14_0_X_2024-01-22-1100 (https://github.com/cms-sw/cmssw/pull/43737#issuecomment-1905215487)

mmusich commented 10 months ago

Given the pattern of merges between CMSSW_14_0_X_2024-01-22-100 (not visible) and CMSSW_14_0_X_2024-01-23-2300 (visible), see the gitHub diff of commits merged, it looks plausible that the non-reproducibility could come from https://github.com/cms-sw/cmssw/pull/43738 (merged in CMSSW_14_0_X_2024-01-23-2300). Incidentally it was also noticed in the PR tests themselves: https://github.com/cms-sw/cmssw/pull/43738#issuecomment-1900186136. @SohamBhattacharya @rovere FYI.

makortel commented 9 months ago

Is anyone looking into this non-reproducibility?

dan131riley commented 9 months ago

Possibly related to #43723? The fix for that issue in #43852 was just merged, so will be interesting to see if it improves the reproducibility.

makortel commented 9 months ago

Possibly related to #43723? The fix for that issue in #43852 was just merged, so will be interesting to see if it improves the reproducibility.

The random differences are still there.

makortel commented 7 months ago

Is anyone looking into this non-reproducibility?

mmusich commented 1 month ago

Is anyone looking into this non-reproducibility?

while I don't think anyone looked at this explicitly, I don't seem to be able to find recent instances (at least since ~ August 2024). Perhaps it got fixed in one of the recent phase-2 HLT menus updates?

mmusich commented 2 weeks ago

+hlt