search

zoogzog / chexnet

Implementation of the CheXNet network (PyTorch)
211 stars 94 forks source link

Memory error while training #12

Open hmchuong opened 6 years ago

hmchuong commented 6 years ago

I used 2xTeslaK80(12GB each) for training from scratch with batchSize=16 but always got out of memory at the end of the first epoch. Have you trained with the same batchSize on your machine?

QinyuHu commented 6 years ago

I feel like it is because of the way he passed loss to scheduler. Instead of only adding values during develop, he added tensor, which accumulated too many memory. You can just unpack the value. losstensorMean += losstensor to losstensorMean += losstensor.data[0]

hmchuong commented 6 years ago

when we change to that, the losstensorMean and lossVal will receive the same value, isn't it?

taijizhao commented 5 years ago

@hmchuong hi hmchuong, I trained on 2 1080ti and also met the problem. Meanwhile, the GPU memory usage isn't too high. Did you solve the issue? Thanks!

yan-michael commented 5 years ago

The problem is still ongoing, any suggestions?? Adding [0] or .item() does not work either, even for batch size of 2.