getting memory error on Tesla K80

[sujit420]

getting error on loading: Traceback (most recent call last): File "main.py", line 33, in train_dset = VQAFeatureDataset('train', dictionary) File "/home/sujitmishra/bottom-up-attention-vqa/dataset.py", line 120, in init self.features = np.array(hf.get('image_features')) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "/home/sujitmishra/py2/local/lib/python2.7/site-packages/h5py/_hl/dataset.py", line 690, in array arr = numpy.empty(self.shape, dtype=self.dtype if dtype is None else dtype) MemoryError

How much gpu does it need for training?

[ZhuFengdaaa]

The problem is you need more Memory and swap space, which must be added up to at least 50G. Correctness: total memory must be added up to at least 80G.

[sujit420]

Thanks for your response @ZhuFengdaaa . I increased my swap space with total of more than 50G. Getting other issue now: Traceback (most recent call last): File "main.py", line 45, in train(model, train_loader, eval_loader, args.epochs, args.output) File "/home/sujitmishra/bottom-up-attention-vqa/train.py", line 36, in train for i, (v, b, q, a) in enumerate(train_loader): File "/home/sujitmishra/py2/local/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 417, in iter return DataLoaderIter(self) File "/home/sujitmishra/py2/local/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 234, in init w.start() File "/usr/lib/python2.7/multiprocessing/process.py", line 130, in start self._popen = Popen(self) File "/usr/lib/python2.7/multiprocessing/forking.py", line 121, in init self.pid = os.fork() OSError: [Errno 12] Cannot allocate memory

How much memory does it need to train? Or do we have pertained model for doing evaluation/inference?

[ZhuFengdaaa]

I use htop to check the training process, and the virtual memory cost is 77.2G. You can try that again. I encountered exactly the same bug OSError: [Errno 12] Cannot allocate memory. It can be solved if you have enough virtual memory.

I have 40G physical memory so I just created 50G swap. You might need more.

[sujit420]

Thanks a lot @ZhuFengdaaa . Training is pretty slow, but its running.. Will ask you if I encounter further errors.

[DaddyWesker]

So, if i got it right, i need to create more virtual memory? Can you tell me how to do that and as i see i need >50 gb virtual memory?

[YuanEZhou]

I encountered the same problem and fixed it by modifying the dataset.py file like following: dataset.py.txt