search

tunz / transformer-pytorch

Transformer implementation in PyTorch.
https://tunz.kr/post/4
MIT License
464 stars 102 forks source link

data loading #6

Closed anxuthu closed 3 years ago

anxuthu commented 3 years ago

Hello! This is a very nice work!

I am looking into your codes and did not figure out why only the first training file ([0]) was loaded in line 146 of translation.py

examples_train = torch.load(data_paths[0])

Thanks!

tunz commented 3 years ago

IIRC, it splits data, and loads one at a time because of memory limitation. The code is just loading the first chunk, and the iterator will pick the next one after completing each one.

It's my old memory, so I could be wrong :)