For the data loader, we used num_samples // batch_size to define the number of batches per dataset, as before TensorFlow would raise an error when the generator ends.
In particular, for paired loader, we used the number of images as the number of samples. It is ok when the images are not labeled, or having one label.
However, when each image has multiple samples, the number of samples should be sum of number of labels. Which means that the number of steps calculated is smaller.
Then, for validation, it caused a big problem when each image has many labels. For example, given 9 images, if each image has 4 labels. There should be 36 samples, but now we estimated 9 samples. With a batch size of 4, we will actually define two steps of validation, which should be 9 instead.
Eventually, this causes the printed metrics inconsistent with the model inference.
So, we should
define the number of samples correctly for paired loader
check unpaired and grouped loader
print step info using logger before training/prediction :)
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Subject of the issue
For the data loader, we used
num_samples // batch_size
to define the number of batches per dataset, as before TensorFlow would raise an error when the generator ends.https://github.com/DeepRegNet/DeepReg/blob/d3edf264b8685b47f1bdd9bb73aca79b1a72790b/deepreg/util.py#L52
In particular, for paired loader, we used the number of images as the number of samples. It is ok when the images are not labeled, or having one label.
https://github.com/DeepRegNet/DeepReg/blob/d3edf264b8685b47f1bdd9bb73aca79b1a72790b/deepreg/dataset/loader/paired_loader.py#L77
However, when each image has multiple samples, the number of samples should be sum of number of labels. Which means that the number of steps calculated is smaller.
Then, for validation, it caused a big problem when each image has many labels. For example, given 9 images, if each image has 4 labels. There should be 36 samples, but now we estimated 9 samples. With a batch size of 4, we will actually define two steps of validation, which should be 9 instead.
Eventually, this causes the printed metrics inconsistent with the model inference.
So, we should
logger
before training/prediction :)