search

matthewearl / deep-anpr

Using neural networks to build an automatic number plate recognition system
MIT License
1.85k stars 697 forks source link

The training part is too slow #3

Closed SunAriesCN closed 8 years ago

SunAriesCN commented 8 years ago

I have run the extractbgs.py and gen.py easily just in a short time, following your instruction. However, your train.py has been training for mostly a week (without GPU laptop). It's a really long time I think for just 1000 samples, do you have any idea for this problem, or tell me how long you got your model before? I want to know whether the program is working well?

WangJinxinNEU commented 8 years ago

Could you tell me what is the symbol when the train is finished?

matthewearl commented 8 years ago

In my tests the training has converged (ie. hit 100% in the test set) after around 100,000 batches. You should see the training begin to converge much sooner than that. The training program does not exit on its own --- you must send a keyboard interrupt when you are satisfied with the training accuracy, at which point a weights.npz file will be produced and the program will exit.

@SunAriesCN training may just be extremely slow on your system (a GPU system is highly recommended). Alternatively the training may have diverged, although I haven't seen this occur myself. If I can find a suitable place to host the 250MB weights file I'll upload my weights.npz for people to play with without having to go to the lengths of doing the training themselves (any suggestions?).

It's possible the model and/or training parameters can be tweaked to improve training time (and potentially reduce model size as well).

WangJinxinNEU commented 8 years ago

ctrl+z is okay to finish the train process?

SunAriesCN commented 8 years ago

I think ctrl+c may be.

matthewearl commented 8 years ago

I've updated the README with information about how to terminate training, and expectations about running time. I'm closing this issue now as the slowness isn't so much a bug as it is an artifact of the neural network design. If you are seeing the network not converge at all please raise another issue to track this (be sure to attach logs).

SunAriesCN commented 8 years ago

OK, thanks!