Closed lkhphuc closed 2 years ago
@lkhphuc I don't know if this is the best way to resolve this issue. I think we need a proper HuggingFaceDatasetAdapter
similar to the tensorflow adapter, that can handle a general setting. Specifically, huggingface datasets behave different when they are on streaming
mode or not.
That would indeed be better.
Currently it's not possible to create a
eg.data.DataLoader
from huggingface'sdataset
. HF'sdataset
check the indices values of the typeint, slice, range str, Iterable
.Converting the indices from from np.int64 to python's int fix this.