loading the whole dataset to memory

[faizwhb]

If I understand salgan code correctly, why does it load the whole dataset into the memory and then picks a batch from the dataset?

it would be better to just load file names into the memory and then load a batch into the memory.

[junting]

Hi, We are doing this because the dataset is small, so it allows us to load the entire dataset into memory, so we do not suffer form the IO bottleneck. But if you prefers to be more memory efficient you can always create you own data loading queue.

Best,