Open dvirginz opened 4 years ago
@dvirginz Can you add link to the forum post.
https://discuss.pytorch.org/t/compatibility-of-subset-dataset-with-disabled-batch-sampling/78793
Quoting @ptrblck's suggested workaround:
As a workaround you could use a SubsetRandomSampler and pass the shuffled indices to it. Inside the Dataset.getitem you might need to create a single index tensor via:
index = torch.stack(index) x = self.data[index]
since the SubsetRandomSampler will pass a list of tensors to the Dataset for the BatchSampler approach.
The workaround is only partial, as most of the time with remote datasets the random sampler is a bad approach (sequential queries are much cheaper). I ended implementing a subset sequential sampler:) But yes:)
I'm facing the exact same issue. I guess it can be fixed by storing the Subset indices as a tensor/numpy array instead of a plain Python list (using .numpy()
instead of .tolist()
here)
I think there is a compatibility issue with disabled batch sampling and subset dataset The use-case - define custom batch sampling, and split the dataset using PyTorch split utility function Here's a minimal working example
And when iterating the subset datasets this is the error
As the self.indices of the
subset
object is a simple python list.Please refer to the forum post, where @ptrblck approves reproducing the bug and offering a workaround.
cc @SsnL