Stackoverflow validation dataset

[ZSL98]

It seems that the validation set pick 10000 examples from the test dataset randomly at the beginning of iterations. So does it mean that the validation set is variable over experiments and invariable over different rounds. To conduct experiments over different hyperparameters, should I fix the seed in create_tf_dataset_from_all_clients since 10000 is rather small and there can be different skewness in partial samplings, making comparisons inconvincible? Thank you~

[zcharles8]

@ZSL98 you are correct. There are no guarantees that ClientData.create_tf_dataset_from_all_clients().take(x) will result in the same examples each time. Using the random seed argument will result in more reproducible results.

Generally, we recommend using the training set to tune hyperparameters, and basing comparisons on the entire test set. The validation set should only be a kind of sanity check throughout training. Great question.