The code does not work with huge pickle dataset

[saeedizadi]

Hello, You load the whole train_data into the RAM at once and then shuffle them for mini-batch selection. Now, my data is so much huge so that it is twice larger than my RAM. Do you know any alternative in theano/lasagne to handle this problem in your code?

Thanks Saeed

[junting]

Hello @saeedizadi

You can replace the train_data = pickle.load(f) with train_data = np.load(TRAIN_DATA_DIR, mmap_mode='r') .