search

NVIDIA / caffe

Caffe: a fast open framework for deep learning.
http://caffe.berkeleyvision.org/
Other
672 stars 263 forks source link

Cache and shuffling are now disabled. #485

Closed huan-fu closed 6 years ago

huan-fu commented 6 years ago

Hi anyone,

When I trained my deep model for some iterations, I got some warning about the data cache because of memory limitation. See the sentence below for the warning.

W0321 10:26:16.957023 93510 data_reader.cpp:197] Data Reader cached 1148000 records so far but it can't continue because it used more than half of swap buffer. Free swap memory left: 36476575744 of total 73014435840. Cache and shuffling are now disabled.

Will the cache and shuffling restart after some further iterations during the training?

Thanks for your help.

caffe version: caffe-0.16.6 data type: LMDB

drnikolaev commented 6 years ago

@huanfu1029 see my reply to #486