How to plot multiple validation sets

[offchan42]

❓ Questions/Help/Support

Suppose I'm classifying cats and dogs. I want to test my accuracy against different image types e.g. bright or dark images. Then I'm going to need a bright validation set and dark validation set. How do I show the metric of multiple validation sets separately in the plot?

This feature is very important for me to understand and get more feedback from the model, instead of a single score from one validation set that doesn't give you much information to analyze and improve the model.

[stared]

@off99555 I edited comment, at it is a question, not a feature request.

That said, you can accomplish that using groups option:

from time import sleep
import numpy as np

from livelossplot import PlotLosses

groups = {'acccuracy': ['acc', 'val_acc'], 'log-loss': ['loss', 'val_loss', 'val2_loss', 'val3_loss']}
plotlosses = PlotLosses(groups=groups)

for i in range(10):
    plotlosses.update({
        'acc': 1 - np.random.rand() / (i + 2.),
        'val_acc': 1 - np.random.rand() / (i + 0.5),
        'loss': 1. / (i + 2.),
        'val_loss': 1. / (i + 0.5),
        'val2_loss': 1. / (i + 1.5),
        'val3_loss': 1. / (i + 2.5)
    })
    plotlosses.send()
    sleep(.1)

Alternatively, instead of groups use group_pattern

group_patterns = [
    (r'^(?!val\d?(_|-))(.*)', 'training'),
    (r'^(val(_|-))(.*)', 'validation'),
    (r'^(val2(_|-))(.*)', 'validation B'),
    (r'^(val3(_|-))(.*)', 'validation C'),
]
plotlosses = PlotLosses(group_patterns=group_patterns)

[offchan42]

I'm using PlotLossesKerasTF. Do you have an example of how to provide 2 validation sets in the model.fit() arguments?

[stared]

Well, if the question is about Keras itself, this as on StackOverflow (see e.g. Using multiple validation sets with keras).

Personally, I haven't used Keras is such a scenario (for more complicated stuff I use PyTorch, in which there is full freedom to log whatever I want).

[offchan42]

How can AdditionalValidationSets callback be used with PlotLossesKerasTF? Because I don't know how PlotLossesKerasTF works.