Proposal: Simplifying systematics calculations

[ntadej]

As we are currently observing some strange issues with disappearing containers, I dived a bit into the how xAH handles containers for systematics and how are scale factors applied at the end of the calculations.

If my knowledge is correct only *Calibrator algorithms actually change quantities on physics objects. All others add decorators based on these quantities, which are the same for systematics not affecting the objects (e.g. electron SFs will be the same for all muon systematics). Calibrators already make shallow copies of the initial objects so these containers should be safe for different SFs.

There is a lot of deep copying going around in the *EfficiencyCorrector algorithms which is probably not necessary. Also we should expect all input containers to be available and not just ignore if they are missing.

My proposal:

• writing systematics names into metadata (or another appropriate location), not per event
• decoupling of nominal and systematic scale factors (to be decided if vector also contains nominal SF)
• remove all deep copying of containers, using ConstDataVectors is enough to sort and filter the objects
• remove quiet checks if input container is missing - they should all be there or something is wrong
• Scale factors changes:
  • Nominal container will have all SFs calculated (nominal and systematics)
  • Systematics containers that actually affect these objects e.g. electron container with electron calibration systematics will have nominal SFs calculated.
  • All other containers should use the SFs from nominal tree as we are using shallow copies initially in the calibrator.
  • write out systematics SFs only for nominal tree and keep branches in other trees empty (or maybe even not include them, to avoid confusion)

I would really like the input from core developers, but this would simplify things a lot. I can also volunteer to make the initial implementation.

[kratsg]

writing systematics names into metadata (or another appropriate location), not per event

This is potentially possible, but you need to write it into a location that non-xAH algorithms can access. Nominally, this is the point of the TStore. We don't have another location that is shared amongst all algorithms. We can do it for xAH::Algorithms internally only, but this breaks the flexibility of xAH with non-xAH algorithms.

decoupling of nominal and systematic scale factors (to be decided if vector also contains nominal SF)

seems like a good idea (do we not do this already?)

remove all deep copying of containers, using ConstDataVectors is enough to sort and filter the objects

Yes. We'll probably need this. Deep-copying is needed in very specific cases and I suspect CDV wasn't available at the time when we were doing deep-copying back in 2015.

remove quiet checks if input container is missing - they should all be there or something is wrong

Yep. This should probably be done first.

[ntadej]

This is potentially possible, but you need to write it into a location that non-xAH algorithms can access. Nominally, this is the point of the TStore. We don't have another location that is shared amongst all algorithms. We can do it for xAH::Algorithms internally only, but this breaks the flexibility of xAH with non-xAH algorithms.

Sorry for not being clear enough. This was meant for output files. So systematics should not be written in the tree on event basis, but only once in metadata file. This would noticeably reduce the output file.

seems like a good idea (do we not do this already?)

Currently SFs are vectors which have more than one elements when systematics are enabled. I propose we have separate nominal number and vector of sys SFs.

[ntadej]

Event rejection for a single systematics should also be handled better. Probably using (shallow) copies of xAOD::EventInfo decorated with passed status for each tree.