During the training of our own datasets, how should the train_kimg, report_kimg, and save_kimg flags change from the default values? Is train_kimg to represent the total number of images in our dataset, both labeled and unlabeled?
The help message for train_kimg states 'Training duration in kibi-samples', but I don't quite understand what the kibi-samples are.
train_kimg is the total duration of the training (default 65536 * 1024 images)
report_kimg is how frequently the test/validation accuracy is measured (default 64 * 1024 images)
save_kimg is how often a tensorflow checkpoint is saved to disk (default 64 * 1024 images), so that's in case your cloud computer is turned off and you need to restart from a previous checkpoint. The checkpoints keep copies of the network parameters.
Note: None of these really are related to the dataset size, the dataset data is just set on repeat during training (so it keeps reading and shuffling in a loop).
During the training of our own datasets, how should the train_kimg, report_kimg, and save_kimg flags change from the default values? Is train_kimg to represent the total number of images in our dataset, both labeled and unlabeled?
The help message for train_kimg states 'Training duration in kibi-samples', but I don't quite understand what the kibi-samples are.
Thank you for your help!