concat dataset - Githubissues

dotnet / TorchSharp

A .NET library that provides access to the library that powers PyTorch.

MIT License

1.41k stars 182 forks source link

concat dataset #1411

Open yueyinqiu opened 1 week ago

yueyinqiu commented 1 week ago

add support for torch.utils.data.ConcatDataset
- a new interface IDataset<out T> is added
  - now Dataset<T> implements IDataset<T>
- more overloads of DataLoader() has been added, to accept IDataset datasets
  - those overloads directly return a DataLoader<T, S>, rather than its subclasses
- datasets supported by DataLoader<T, S> have been relaxed to IDataset<T>
parameter of collate functions in DataLoader<T, S> has been relaxed to IReadOnlyList

yueyinqiu commented 1 week ago

As mentioned here https://github.com/dotnet/TorchSharp/pull/1357#issuecomment-2478038515, the current version has no API changes (except the type of dataset and collate_fn in DataLoader<T, S>).

However we don't actually need Dataset<T>, Dataset and IterableDataset. Do we have any plan to remove them later? They may also occupy the position of some other classes in PyTorch, like #1353.

(But I find it hard to use [Obsolete] on them since some methods are using them as the return type.)