I want to be able to train models with random initial weights and train/valid/test splits, but test predictions on a deterministic ordering of the test set for a given train/valid/test seed, as we are interested in comparing the predictions between models at each specific data sample. Amazingly I couldn't find a torch sampler able to do this, without fixing the global random seed (the SequentialSampler, which I thought would do this, simply returns indices based on the length of the list of indices, it doesn't return a non-shuffled list of the indices themselves, which seems totally useless). So have added one called SequenceSampler.
I want to be able to train models with random initial weights and train/valid/test splits, but test predictions on a deterministic ordering of the test set for a given train/valid/test seed, as we are interested in comparing the predictions between models at each specific data sample. Amazingly I couldn't find a torch sampler able to do this, without fixing the global random seed (the SequentialSampler, which I thought would do this, simply returns indices based on the length of the list of indices, it doesn't return a non-shuffled list of the indices themselves, which seems totally useless). So have added one called
SequenceSampler
.