search

Fangyh09 / Image2LMDB

Convert image folder to lmdb, adapted from Efficient-PyTorch
66 stars 10 forks source link

MemoryError #3

Closed Dami111 closed 2 years ago

Dami111 commented 3 years ago

In my dataset (datasize:500000 images), I get MemoryError when dealing with 85000 images. What could be the reasons? Thanks in advance.

[80000/500000] [85000/500000] File "/usr/local/lib64/python3.6/site-packages/torch/utils/data/dataloader.py", line 804, in next idx, data = self._get_data() File "/usr/local/lib64/python3.6/site-packages/torch/utils/data/dataloader.py", line 771, in _get_data success, data = self._try_get_data() File "/usr/local/lib64/python3.6/site-packages/torch/utils/data/dataloader.py", line 724, in _try_get_data data = self.data_queue.get(timeout=timeout) File "/usr/lib64/python3.6/multiprocessing/queues.py", line 113, in get return _ForkingPickler.loads(res) File "/usr/local/lib64/python3.6/site-packages/torch/multiprocessing/reductions.py", line 290, in rebuild_storage_fd shared_cache[fd_id(fd)] = StorageWeakRef(storage) File "/usr/local/lib64/python3.6/site-packages/torch/multiprocessing/reductions.py", line 49, in setitem dict.setitem(self, key, storage_ref) MemoryError

My dataloader keeps the same as the github.

Fangyh09 commented 3 years ago

Could share some data and codes? So it could be reproduced.

buzhangjiuzhou commented 3 years ago

I've noticed that during my process of creating lmdb file, the memory usage is continously growing, while thanks to my large memory, my process finished well.

Fangyh09 commented 2 years ago

Agree on @buzhangjiuzhou. Will close this issue.