search

brando90 / ML4HOList

0 stars 0 forks source link

Data Pipeline Construction #4

Closed Seeker-YML closed 3 years ago

Seeker-YML commented 4 years ago

All about building Data Pipeline when we are using PyTorch ...

Seeker-YML commented 4 years ago

torch.utils.data.Dataset and torch.utils.data.DataLoader OR TorchText?

Generally speaking, torch.utils.data.DataLoader can help PyTorch Users to build their custom dataset and read those data parallel (efficiently), but it requires users to construct the way to read the data (please refer here https://pytorch.org/tutorials/beginner/data_loading_tutorial.html). But what Torchtext provides are some normal methods&functions to preprocess and read the data for NLP Task.