Epoch summaries on tensorboard

[sebpuetz]

Add epoch-averaged (train and dev) values of the summarized metrics to tensorboard. After a few epochs it's hard to tell anything from the per-batch graphs.

[twuebi]

I'd like to keep the per-batch summaries and add the per-epoch as additional summaries.

Something along these lines:

 epoch_loss_placeholder = tf.placeholder(name="epoch_loss_placeholder",dtype=tf.float32,shape=[])
 epoch_acc_placeholder =  tf.placeholder(name="epoch_acc_placeholder",dtype=tf.float32,shape=[])
 val_epoch = tf.Variable(
     0,
     trainable=False,
     dtype=tf.int64,
     name="val_epoch")
     val_epoch = tf.convert_to_tensor(val_epoch)
 with tf.control_dependencies([val_epoch.assign_add(1)]):
     epoch_val_summaries = [
         tf.contrib.summary.scalar(name="epoch_loss",
                                   tensor=epoch_loss_placeholder,
                                   step=val_epoch,
                                   family="val"),
         tf.contrib.summary.scalar(name="epoch_acc",
                                   tensor=epoch_acc_placeholder,
                                   step=val_step,
                                   family="val")]

 ...

 with tf.compat.v1.variable_scope("summaries"):
     self.train_summaries = tf.group(train_summaries, name="train")
     self.val_summaries = tf.group(val_summaries, name="val")
     self.epoch_val_summaries = tf.group(epoch_val_summaries, name="epoch_val")

IMO the per-batch graphs can be informative to find out what's going on when things don't work. For instance, by looking at the per batch gradient norms or spikes in the loss / accuracy.

[danieldk]

At this point graph compatibility is everything. I guess the ramification here would be that we need three additional Optional ops in TaggerGraph, right?

[twuebi]

If graph compatibility means to load an old model with a newly written graph, then we need 4 optional ops since the variable val_epoch will be missing when calling the restore op of a new graph. So it'd also need to be a placeholder.

If it only means being able to load both graphs on the rust side, then it's three options.

All in all, it may be a good idea to rewrite the graph after training for inference where a stable interface is needed such that all of these compatibility problems would only apply to training graphs.