licence detector training with GPU

[Ruveyda]

I want to train my custom dataset for licence plate detection. I have 2000 image for this. First, I trained my labelling dataset with CPU.

optimizer : Adam, learning rate : 0.01 , batch size = 64, iteration: 300000

In this case, according my calculations, my training period would take 21 days. I just trainned 2 days and when I tested the model could detect licence plates but with some mistakes

Then, I decided to train with gpu because of the training period is too long.

I uninstall tensorflow and I install tensorflow-gpu(1.13), Cuda(10.0) and Cudnn(7.4) and I added these codes to train_detector.py :

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.InteractiveSession(config=config)

In this time, when I use the same values (Adam, lr=0.01, itr=100000, bs=64), loss value decreases very fast every iteration (loss begins with 210 and ends 0.000000). The training ends in four-five hours. But when I tested, my model could not detect any licence plate.

I try to change values. I did learning rate = 0.000001, iteration =3000 so that the loss is 0.7, training ends 10-20 minutes, but model detects very wrong. (and I know this number is very bad for learning rate)

My GPU : GeForce RTX 2080

How can I optimize these numbers, where is my mistake?

Thank you.

[ThorKing01]

@Ruveyda I have something wrong about the license-plate-detection.py after I installed tensorflow-gpu(1.13), Cuda(10.0) and cudnn(7.4). Traceback (most recent call last): File "license-plate-detection.py", line 47, in <module> UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. Could you tell me how did you resolve the bug? Thank you

[wildnrg]

I have the same problem when I try with tensorflow-gpu

[ThorKing01]

I have the same problem when I try with tensorflow-gpu

The problem is so far from here. maybe you should check your cuDNN's version, I had sloved the problem after I installed cuDNN 7.4.2.

[ThorKing01]

I have the same problem when I try with tensorflow-gpu And maybe you should add the code to limit you video memery。 config=tf.ConfigProto(); config.gpu_options.per_process_gpu_memory_fraction=0.5 set_session(tf.Session(config=config))