Meaning of train_kimg, report_img, and save_kimg flags

[varunnair18]

During the training of our own datasets, how should the train_kimg, report_kimg, and save_kimg flags change from the default values? Is train_kimg to represent the total number of images in our dataset, both labeled and unlabeled?

The help message for train_kimg states 'Training duration in kibi-samples', but I don't quite understand what the kibi-samples are.

Thank you for your help!

[david-berthelot]

1 kibi = 1024 images

• train_kimg is the total duration of the training (default 65536 * 1024 images)
• report_kimg is how frequently the test/validation accuracy is measured (default 64 * 1024 images)
• save_kimg is how often a tensorflow checkpoint is saved to disk (default 64 * 1024 images), so that's in case your cloud computer is turned off and you need to restart from a previous checkpoint. The checkpoints keep copies of the network parameters.

[david-berthelot]

Note: None of these really are related to the dataset size, the dataset data is just set on repeat during training (so it keeps reading and shuffling in a loop).

[varunnair18]

Got it, thank you!