model evaluation on random samples during training is not straightforward

[hvgazula]

For example, if I want to evaluate random samples at the end of each epoch. I envision using a custom callback as follows. However, the problem with this approach is the nobrainer dataset object cannot be indexed as it does not have a __len__ method and hence we cannot randomly sample the test examples.

If we pass the list of test files instead, it entails calling nib.load, one-hot encoding the labels, and then creating the dataset object which can then be sent to model.evaluate.

# Custom callback for testing the model on a random sample from the test dataset
class TestRandomSampleCallback(tf.keras.callbacks.Callback):
    def __init__(self, test_dataset):
        super(TestRandomSampleCallback, self).__init__()
        self.test_dataset = test_dataset

    def on_epoch_end(self, epoch, logs=None):
        print("\nTesting model on a random sample after epoch {}...".format(epoch + 1))
        # Select a random sample from the test dataset
        random_index = random.randint(0, len(self.test_dataset) - 1)
        random_sample = self.test_dataset[random_index]
        test_loss, *test_metrics = self.model.evaluate(random_sample[0], random_sample[1], verbose=0)
        print("Test Loss: {:.4f}".format(test_loss))
        for i, metric_name in enumerate(self.model.metrics_names):
            print("Test {}: {:.4f}".format(metric_name, test_metrics[i]))

[hvgazula]

Since we are already checkpointing the model after every epoch, I suggest we should consider evaluation after training finishes. @satra thoughts?

[satra]

aren't you passing dataset.dataset to keras/tensorflow?

also aren't you giving two different datasets for training and evaluation? currently the last n samples are used for eval, but randomizing is a function of adjusting the order of files given to from_files when creating the records.

[hvgazula]

addressed saving predictions on test images as png files here https://github.com/neuronets/nobrainer_training_scripts/commit/b9c18dadffaedc2c9bd2f12ccc4ab9485a9d1bb6