Integrate the trigger emulation to our MC workflow

[yipengsun]

We need to integrate the trigger emulation from https://github.com/umd-lhcb/TrackerOnlyEmu.

So far, I'd like to clarify these concepts:

• Trigger Matched (tm) vs. Non-Trigger-Matched (ntm)
• Training inputs to the XGB

[yipengsun]

I was able to re-run Manuel's studies in trigger-emu-debug branch.

[yipengsun]

This is done. Closed.

[yipengsun]

I forgot to report an important observation when I try to reproduce Manuel's study.

• Manuel's study is preserved in the trigger-emu-debug branch of this repo. Follow the README of that branch and you reproduce Manuel's result
• I couldn't reproduce Manuel's BDT results, so I tried to check my code and try to go back to the old code
• I discovered no bug
  • Previously the BDT input has to be generated by another script
  • I copied that old script, and generated the BDT input for my new code, it still didn't converge
  • I then tried to use the old input generated in Manuel's study, and I reproduced the plots
• I realized that this is due to the difference on how the BDT input is generated
  • In Manuel's study, the BDT input is generated on the unsplit ntuple, before splitting into training and validation samples
  • In the new approach, since now the new script can handle raw inputs, we FIRST split the ntuple, THEN prepare the training sample for BDT input
  • What I observed is that although the distribution of the key variable d0_et_emu_no_bdt have very similar distributions, the values for the same event are very different
    • This is likely due to different number of events during the d0_et_emu_no_bdt generation; even though the random seed is the same.
• This shows that the BDT convergence is finicky
• The XGB doesn't suffer from this issue, so we don't really care
• For now I just reproduce Manuel's plot w/ the old input ntuple, committed along with the code
  • The relevant code is at: https://github.com/umd-lhcb/lhcb-ntuples-gen/tree/master/studies/trigger_emulation-l0_hadron_tos_debug
  • Don't forget to get the annexed old ntuple!

Finally, a plot showing the reproduction:

And the over-trained BDT:

FYI @manuelfs

[yipengsun]

We still couldn't fully reproduce Manuel's results with the latest code, as I have to rely on his old input ntuple, but we won't spend any more time on this as we don't care about BDT.