Extend logging API to save metrics and images

[goodmattg]

We should probably be using Ignite built-in functionality to do logging: https://pytorch.org/ignite/contrib/handlers.html#ignite.contrib.handlers.trains_logger.OutputHandler

Is Ignite flexible enough? Should we write a custom logger? Formally define a simple, but flexible interface to log assorted metrics and engine outputs. As of now we have the temporary structure below which says to run a eval_engine on a DataLoader loader and then log arbitrary fields in the metrics dictionary produced by eval_engine.

[ChickenTarm]

Can the logging api only log metrics? how do you log outputs like saving images?

[goodmattg]

The logging API can also handle engine outputs, but from the docs it seems like they only anticipated numerical. I can look for an easy way to do that as custom.

[ChickenTarm]

Right I guess you could define images to be a custom metric and have that metric function do the saving of images and use the hydra folders for the path

[goodmattg]

I am modifying what this task covers. Logging metrics to a logfile and stdout work fine. This now covers:

• logging the metrics to metrics specific numeric files. E.g. "accuracy" saved to a binary file "accuracy.pt" that can load to numpy and then matplotlib
• logging images to folders
  • i.e. if user says to log field "img1" and we typecheck "img1" as an image, save to it's own img file in folder "img1" labeled by either epoch or iteration depending on how called