Technical improvements

[kmohrman]

This branch includes various technical improvements to the topcoffea code (mainly in response to the Issues in the Technical updates to topcoffea project). It should not merged yet, as the updates are still in progress.

[codecov[bot]]

Codecov Report

Merging #354 (fbbdf99) into master (08b21e1) will increase coverage by 3.07%. The diff coverage is 48.20%.

@@            Coverage Diff             @@
##           master     #354      +/-   ##
==========================================
+ Coverage   30.28%   33.36%   +3.07%     
==========================================
  Files          47       39       -8     
  Lines        7600     6461    -1139     
==========================================
- Hits         2302     2156     -146     
+ Misses       5298     4305     -993     

Flag	Coverage Δ
unittests	33.36% <48.20%> (+3.07%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
analysis/topEFT/datacards_post_processing.py	0.00% <0.00%> (ø)
analysis/topEFT/get_datacard_yields.py	0.00% <0.00%> (ø)
analysis/topEFT/get_yield_json.py	91.66% <ø> (ø)
.../topEFT/make_1d_quad_plots_from_template_histos.py	0.00% <0.00%> (ø)
analysis/topEFT/make_cr_and_sr_plots.py	0.00% <0.00%> (ø)
analysis/topEFT/make_jsons.py	0.00% <0.00%> (ø)
analysis/topEFT/make_skim_jsons.py	0.00% <0.00%> (ø)
analysis/topEFT/missing_parton.py	0.00% <0.00%> (ø)
analysis/topEFT/parse_datacard_templtes.py	0.00% <0.00%> (ø)
analysis/topEFT/run_sow.py	0.00% <0.00%> (ø)
... and 19 more

... and 4 files with indirect coverage changes

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[kmohrman]

Note that we have pinned the numpy version in the environment.yml file (to numpy=1.23.5) as a workaround to the numpy.bool error that we otherwise get (from one of the coffea files).

This should be revisited at some point (e.g. once coffea has fixed the numpy.bool instance, or once conda has figured out how to make a consistent environment that does not lead to this error) to remove the numpy version pin.

See Issue #368.

[kmohrman]

The technical_improvements branch has been run through the full workflow. I made the pkl file and datacards on the topcoffea technical_improvements branch, and made the workspace on @hnelson's updated branch on the EFTFit repo (PR here). I also made a CR pkl file and made the CR plots from that as well (just to cross check the normalization) and there were no issues with that.

The fits (profiled and frozen, unblind) look consistent to me. For the vast majority of the WCs, the differences are less than a percent. But I don't think there's any reason to believe the few O(1%) differences that do exist are due to changes on the technical_improvements brach. We don't expect identical results here because I ran with fewer scan points (so that they'd finish faster) so this means the interpolation for the 2sigma number will be different, plus the randomness in the processor can result in very small (sub percent) fluctuations in the ranges.