search

twitter-archive / torch-dataset

An extensible and high performance method of reading, sampling and processing data for Torch
Apache License 2.0
76 stars 24 forks source link

Is there any limitation in the memory when using torch-dataset? #26

Closed chienlinhuang1116 closed 8 years ago

chienlinhuang1116 commented 8 years ago

Hi, we saved our data to torch senor format 'train.t7' and loaded it using torch-dataset.

local trainingDataset = Dataset('train.t7', {partition = 1, partitions = 1})

If the size of 'train.t7' is 500GB or more, is there any limitation in the memory when using torch-dataset? Is there any issue when we define 'batchSize' or use 'torch-ipc'?

Thank you.

zakattacktwitter commented 8 years ago

This is the same as https://github.com/twitter/torch-ipc/issues/23 so I am answering there, closing this one. Thanks.