TensorFlow/Keras Toolbox

[pfackeldey]

Dear CMS-ML community,

I would like to bring up the following proposal:

Feature description

Machine-learning applications in CMS-analysis are mainly implemented with TensorFlow/Keras nowadays. Such a common baseline allows the support of a "toolbox", containing useful callbacks, metrics, loss-functions, ...

Possible solution

Simple predefined Keras custom Callbacks, Metrics, ... using e.g.:

• Callback: https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/Callback
• Metric: https://www.tensorflow.org/api_docs/python/tf/keras/metrics/Metric

Best, Peter

[riga]

Hey @pfackeldey ! Thanks for the suggestion. Do you have anything particular in mind, like certain callbacks or CMS/HEP specific metrics and losses?

[pfackeldey]

Hi,

I could imagine callbacks for common plotting problems, which can be displayed e.g. after each epoch and logged by TensorBoard: Confusion matrices, Input variable distributions, Loss, ROC (preferable already in HEP-styling) Also more technical things, such as GPU usage Callbacks.

For metrics and losses I have nothing in particular in mind, but I can imagine that many analysts are experimenting and this knowledge might get lost if it is not condensed in a central place. (One metric I could imagine is the significance of the signal node of a classification problem.)

[riga]

If you have a - say - callback available as a gist, I can create a minimal structure around it in the package and set things up. As a second step, we can add tools one-by-one and see if the structure holds.

[pfackeldey]

Hope this snippet can serve as a starting point for a GPU resource-logging callback: https://gist.github.com/pfackeldey/e3c5edfc3ea81bd1b82bc6e1fd1ab3af