Multiple metrics during training

[nitrif]

Hi,

How exactly is it possible to evaluate multiple metrics after each minibatch in python cntk? AFAIK train_minibatch and test_minibatch return only one.

Also, I am interested in performing metric accumulation (not averaging) during the epoch, mostly updating metrics nominator and denominal terms, where the final metric calculation would be performed at the end of the epoch. What is the most convenient way to implement this?

Thanks, Nikola

[cha-zhang]

This is a known issue. We will attempt to fix this in our next iteration.

[nitrif]

Thanks.

Is it possible to do this in cross-validation phase?

Like instead of calling: metric = trainer.test_minibatch(data) maybe call: out = trainer.model.eval(data[list(input_map)[0]]) and then calculate all metrics needed?

I tried this, but found several score mismatches. Also I had to reduce minibatch_size since the second approach apparently uses more GPU memory?

[cha-zhang]

Yes, you can. I'm not sure about the more GPU memory part. It should not be.

One example you can look into is this: https://github.com/Microsoft/FERPlus/blob/master/src/train.py

[ghost]

This would be extremely useful such as in situations where one would like to keep track of both Top-1 and Top-5 accuracy during training and testing.

[Arkoprabho]

It is possible to have multiple metrics with Keras as a front end to CNTK. Can someone direct how to do this using CNTK alone?

[MaxCarlson]

Is multiple metrics possible yet? I have a network that has two output heads and I'd like to be able to view the accuracy for both in tensorboard. If this isn't supported by the trainer class yet, is there another way both metrics could be calculated?

[aschmiri]

This would be really helpful.