search

roatienza / Deep-Learning-Experiments

Videos, notes and experiments to understand deep learning
MIT License
1.1k stars 760 forks source link

memory Error #22

Closed newmluser closed 5 years ago

newmluser commented 5 years ago

Hi, When I increase the size of the file I see I am getting memory Error.

Looks

content = np.array(content)

is not able to get more data into the buffer (100MB Text file). is there a way (or) any suggestions to make this same program work for large amount of data

roatienza commented 5 years ago

If you need to load huge data from filesystem, I recommend Data Generator. Pls see this tutorial:

https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly