Currently, validation is randomly sampled from the validation dataset. Removing the randomness, or resetting the dataloader each time? (PL has a function for this) could work best. Alternatively, creating a hand-picked validation set could also be helpful that covers the most challenging and diverse set. This would require a bit more work, but could use something like #42 to select where the models fail the most would be a much more informative set. Could also then change to store the validation samples as a much smaller webdataset, which would be faster to load.
Currently, validation is randomly sampled from the validation dataset. Removing the randomness, or resetting the dataloader each time? (PL has a function for this) could work best. Alternatively, creating a hand-picked validation set could also be helpful that covers the most challenging and diverse set. This would require a bit more work, but could use something like #42 to select where the models fail the most would be a much more informative set. Could also then change to store the validation samples as a much smaller webdataset, which would be faster to load.