search

rdmenezes / visual-rbm

Automatically exported from code.google.com/p/visual-rbm
1 stars 0 forks source link

VisualRBM causes Radeon HD 6490M crash #10

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 

VisualRBM causes Radeon HD 6490M crash,
I get message "Display driver stopped responding and has recovered"

After some investigation I changed registry settings 
in HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\GraphicsDrivers
which are responsible for timeouts and recovery of graphic card 
but with no luck.

I tried many settings combinations.
Changing drivers for graphic card did not help either.

Any ideas how to solve the problem ?

It seems that VisualRBM settings have no effect on crashing.

I use latest VisualRBM, Windows 7 64.
It happens with MNIST data as well as with other data.

Original issue reported on code.google.com by klu...@jabster.pl on 17 Feb 2014 at 4:21

GoogleCodeExporter commented 9 years ago 
When does the crash occur?  Does it happen immediately after starting training 
or does VisualRBM go for a couple epochs and then crash?  Does the crash happen 
on older versions of VisualRBM?  If you run VisualRBM from the command-line do 
any error messages print out?

Original comment by pospes...@gmail.com on 17 Feb 2014 at 7:24

GoogleCodeExporter commented 9 years ago 
It happens after few epochs, sometimes 10, sometimes 100,
in older version as well as in new one.

I use it also in my code, it happens in:

atlasResult=atlas.Next(training_example);
cd.Train(training_example);

I cannot debug because it stops/crashes there.

Is it related to amount of data - my training data set is around 350MB ?

One epoch takes up to 2 minutes, I changed registry timeout settings up to 
120min 
(default is 2 sec) so that windows does not think the card has crashed 
if not responded for that time.

Original comment by klu...@jabster.pl on 18 Feb 2014 at 11:59

GoogleCodeExporter commented 9 years ago 
Can you track how long each epoch takes (ie, does each successive epoch take 
longer and longer)?  Can you track your GPU temps during training?  My best 
guess is your laptop's GPU is overheating, but I really have no idea.

Original comment by pospes...@gmail.com on 18 Feb 2014 at 8:22