search

matthewearl / deep-anpr

Using neural networks to build an automatic number plate recognition system
MIT License
1.83k stars 703 forks source link

number plate recognition system vs number plate detection system ? #28

Open tianqig opened 7 years ago

tianqig commented 7 years ago

This is a very interesting work based on deep learning, but the title is misleading, it is NOT a number plate recognition because no individual character detection and recognition are performed at all, instead it only locate number plate from given images.

Let me know if my understanding is incorrect.

Best regards

sourabh2k15 commented 7 years ago

No it does both localization and recognition as per my understanding. It is trained using images generated using gen.py . The names of the images are numplate_1.jpg, so 1/0 indicate if plate is present in the image, and the numplate number is used to train the network to recognize digits.

tianqig commented 7 years ago

Maybe I miss something, but I don't see you have explicit steps for segmenting individual characters, and then recognise the segmented characters. It would be good to have images where segmented characters are displayed with bounding boxes, plus recognition results of these characters. Another way is to calculate plate number recognition performance with your both training and testing images. Thanks for your interesting work.

On Sat, Apr 1, 2017 at 1:13 PM, sourabh2k15 notifications@github.com wrote:

No it does both localization and recognition as per my understanding. It is trained using images generated using gen.py . The names of the images are _1.jpg, so 1/0 indicate if plate is present in the image, and the numplate number is used to train the network to recognize digits.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/matthewearl/deep-anpr/issues/28#issuecomment-290896033, or mute the thread https://github.com/notifications/unsubscribe-auth/AFXX1dU01Ifk00oIp9OgeEHInL5CCLxmks5rrdzpgaJpZM4Mvtyu .

sourabh2k15 commented 7 years ago

I am an undergraduate student working on my final year project of automatic number plate recognition. So far I have written python scripts using opencv that seperate out license plate from the full image, then segment individual characters. The part left is to recognize the individual characters. All this was a tedious process and even then the system doesn't work for all the images you put in it randomly. It works for some. Then I stumbled upon this awesome work that uses CNN to detect and recognize license plates

sourabh2k15 commented 7 years ago

It is a neural network , so you do not have intermediate steps , you just have a set of weights as an output that can be used by detect.py that takes in an image and creates a bouding box around the number plate and puts the actual read number plate above the box . It does so in under 2 seconds and from what I have tested it gives stunningly accurate results for the recognition part ( like 100%) . Also a note I would like to add: it is really time consuming and resource intensive to train this network on our normal pcs , instead you could download the weights.npz ( 250 mb file) and use it right away. The problem with this ? well the weights work for UK plates which have 7 chars long. I have modified the code to train it for 10 chars long ( indian number plates ) , it is training now ( but awfully slow ). Do let me know if I can help you in any way.

Thanks.

ZackTack commented 7 years ago

@sourabh2k15 Hi, I am also an undergraduate who working on license plate detection. However, my pc is not sufficient enough to train the network. Do you know anywhere I can download the weight file directly (weights.npz) ?

sourabh2k15 commented 7 years ago

I have pretrained weights for UK number plates ( mediafire link already available in one of the issues in this repo ) and Indian number plates ( trained it for last 3 days , can upload it somewhere ) . If you need to detect some other license plate, you will have to change gen.py, generate the appropriate dataset and run train.py again , which will take a long time

ZackTack commented 7 years ago

Found it. Thx

ghost commented 7 years ago

What do you need to change in gen.py or other files to fit a plate system from another country? (In french with XX - NNN - XX (letters - digits - letters)) Thank's

sourabh2k15 commented 7 years ago

You need to edit code generating part , which generates random number plates def generate_code():

On Fri, Apr 7, 2017 at 2:32 PM, novsub notifications@github.com wrote:

What do you need to change in gen.py or other files to fit a plate system from another country? (In french with XX - NNN - XX (letters - digits - letters)) Thank's

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/matthewearl/deep-anpr/issues/28#issuecomment-292482282, or mute the thread https://github.com/notifications/unsubscribe-auth/AH09QqxR9Ito3TIIQLBMeE0AimPHqVKzks5rtfuUgaJpZM4Mvtyu .

sourabh2k15 commented 7 years ago

Also we found a better solution using edge detection for localization and tesseract for recognition

On Fri, Apr 7, 2017 at 8:02 PM, sourab reddy sourab.reddy2k14@gmail.com wrote:

You need to edit code generating part , which generates random number plates def generate_code():

On Fri, Apr 7, 2017 at 2:32 PM, novsub notifications@github.com wrote:

What do you need to change in gen.py or other files to fit a plate system from another country? (In french with XX - NNN - XX (letters - digits - letters)) Thank's

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/matthewearl/deep-anpr/issues/28#issuecomment-292482282, or mute the thread https://github.com/notifications/unsubscribe-auth/AH09QqxR9Ito3TIIQLBMeE0AimPHqVKzks5rtfuUgaJpZM4Mvtyu .

ghost commented 7 years ago

How do we know that we converge and what is the batches?

rphly commented 7 years ago

@sourabh2k15 is it just me, or when I add extra combinations for plates:

def generate_code():
    choices = [1,2]
    choice = random.choice(choices)
    if choice == 1:
        return "{}{}{}{}{}{}{}{}".format(
            random.choice(common.LETTERS),
            random.choice(common.LETTERS),
            random.choice(common.LETTERS),
            random.choice(common.DIGITS),
            random.choice(common.DIGITS),
            random.choice(common.DIGITS),
            random.choice(common.DIGITS),
            random.choice(common.LETTERS))
    elif choice == 2:
        return "{}{}{}{}{}{}{}".format(
            random.choice(common.LETTERS),
            random.choice(common.LETTERS),
            random.choice(common.DIGITS),
            random.choice(common.DIGITS),
            random.choice(common.DIGITS),
            random.choice(common.DIGITS),
            random.choice(common.LETTERS))

This works and generates images but images are incorrectly labelled 1/0 when generated. This is the definition right (?):

1 == number plate present
0 == number plate cut off / not present

But...

Some images are labelled foo_0.png when the number plate is completely in view.

Anyone else face this issue?

sourabh2k15 commented 7 years ago

Sometimes in the training data the number plate will be completely visible, but it will be labelled 0 because the size is not according to the requirements. You can go through matt's blog post to understand the whole working, wherein he has mentioned that if the number plate is too big or too small it won't be detected ( exact measures to be found in that blog post )

On Fri, Apr 14, 2017 at 12:00 PM, Raphael Yee notifications@github.com wrote:

@sourabh2k15 https://github.com/sourabh2k15 is it just me, or when I add extra combinations for plates:

def generate_code(): choices = [1,2] choice = random.choice(choices) if choice == 1: return "{}{}{}{}{}{}{}{}".format( random.choice(common.LETTERS), random.choice(common.LETTERS), random.choice(common.LETTERS), random.choice(common.DIGITS), random.choice(common.DIGITS), random.choice(common.DIGITS), random.choice(common.DIGITS), random.choice(common.LETTERS)) elif choice == 2: return "{}{}{}{}{}{}{}".format( random.choice(common.LETTERS), random.choice(common.LETTERS), random.choice(common.DIGITS), random.choice(common.DIGITS), random.choice(common.DIGITS), random.choice(common.DIGITS), random.choice(common.LETTERS))

This works and generates images but images are incorrectly labelled 1/0 when generated. This is the definition right (?):

1 == number plate present 0 == number plate cut off / not present

But...

Some images are labelled foo_0.png when the number plate is completely in view.

Anyone else face this issue?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/matthewearl/deep-anpr/issues/28#issuecomment-294099810, or mute the thread https://github.com/notifications/unsubscribe-auth/AH09QsTdajp9ToEU66JAP08ziAob9Q43ks5rvxJ-gaJpZM4Mvtyu .

rphly commented 7 years ago

@sourabh2k15 Ah yes! pardon me I totally forgot about the sizing requirements. Is it wise though to label the image 0 even if the sizing is small? My understanding is that should the bounding box zoom in smaller and smaller, as long as it detects a number plate - like object, it should reflect in detection.

Only if say the number plate is cut off should it fail to detect.

sourabh2k15 commented 7 years ago

You can edit function generate_im() in gen.py and set min_scale and max_scale to 0 and 1 respectively. Haven't tried it , let me know if this works.

rphly commented 7 years ago

@sourabh2k15 Ahh okay, I'm having another issue:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1021, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1003, in _run_fn
    status, run_metadata)
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 469, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: logits and labels must be same size: logits_size=[350,36] labels_size=[400,36]
     [[Node: SoftmaxCrossEntropyWithLogits = SoftmaxCrossEntropyWithLogits[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](Reshape_3, Reshape_4)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train.py", line 270, in <module>
    initial_weights=initial_weights)
  File "train.py", line 242, in train
    do_batch()
  File "train.py", line 222, in do_batch
    feed_dict={x: batch_xs, y_: batch_ys})
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 766, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 964, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1014, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1034, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: logits and labels must be same size: logits_size=[350,36] labels_size=[400,36]
     [[Node: SoftmaxCrossEntropyWithLogits = SoftmaxCrossEntropyWithLogits[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](Reshape_3, Reshape_4)]]

Caused by op 'SoftmaxCrossEntropyWithLogits', defined at:
  File "train.py", line 270, in <module>
    initial_weights=initial_weights)
  File "train.py", line 172, in train
    digits_loss, presence_loss, loss = get_loss(y, y_)
  File "train.py", line 131, in get_loss
    tf.reshape(y_[:, 1:],[-1, len(common.CHARS)]))
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/ops/nn_ops.py", line 1449, in softmax_cross_entropy_with_logits
    precise_logits, labels, name=name)
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 2265, in _softmax_cross_entropy_with_logits
    features=features, labels=labels, name=name)
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1128, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): logits and labels must be same size: logits_size=[350,36] labels_size=[400,36]
     [[Node: SoftmaxCrossEntropyWithLogits = SoftmaxCrossEntropyWithLogits[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](Reshape_3, Reshape_4)]]

I modified gen.py to generate 8-digit number plates, and adjusted the input sizes in train.py.

sourabh2k15 commented 7 years ago

you will also have to adjust sizes in model.py ( change every 7 to 8) and also modify the read_data function in train.py , it takes in file names in test directory and splits them up into number plate and presence 1/0 . So right now it is

code = fname.split("/")[1][9:16] // change 16 to 17 to read the extra 8th character in file name code = fname.split("/")[1][17] // change 17 to 18 to read presence

rphly commented 7 years ago

@sourabh2k15 Thank you, what a lifesaver! I actually wrote code above to randomly create images with differing numberplate length. But now that I think about it, my input will be of different sizes, is there a method where I can "pad" the smaller inputs with 0s ?

Now I'm getting this error:

Traceback (most recent call last):
  File "train.py", line 270, in <module>
    initial_weights=initial_weights)
  File "train.py", line 235, in train
    test_xs, test_ys = unzip(list(read_data("test/*.png"))[:50])
  File "train.py", line 69, in read_data
    yield im, code_to_vec(p, code)
  File "train.py", line 59, in code_to_vec
    c = numpy.vstack([char_to_vec(c) for c in code])
  File "train.py", line 59, in <listcomp>
    c = numpy.vstack([char_to_vec(c) for c in code])
  File "train.py", line 56, in char_to_vec
    y[common.CHARS.index(c)] = 1.0
ValueError: substring not found
sourabh2k15 commented 7 years ago

Yeah I saw your code from an earlier comment, randomly generating number plates of length 7 and 8. I read in one of the other issue comments that this is a desirable feature , but support hasn't been provided yet as the original maintainer seems to have abandoned this project for some months now ( last github activity in jan :/ ) . The way the neural net is designed, you can train it on a single length only.

Now I can suggest you 2 solutions:

1) if you dig in the issues you will find a link to another github repo where the guy has trained the neural net to recognize variable length digits , borrowing code from this repo

2) I have used an edge detection algorithm that accurately cuts out the number plate section from the image, which is then feeded to tesseract ocr to get number plate text . This system works but needs a bit of fine tuning , you could try if it is working for your images.

On Sat, Apr 15, 2017 at 3:07 PM, Raphael Yee notifications@github.com wrote:

@sourabh2k15 https://github.com/sourabh2k15 Thank you, what a lifesaver! I actually wrote code above to randomly create images with differing numberplate length. But now that I think about it, my input will be of different sizes, is there a method where I can "pad" the smaller inputs with 0s ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/matthewearl/deep-anpr/issues/28#issuecomment-294282812, or mute the thread https://github.com/notifications/unsubscribe-auth/AH09QgdUcW4xhikO_pVnLeOY1Sv7Fu26ks5rwI-8gaJpZM4Mvtyu .

ashish1405 commented 6 years ago

@sourabh2k15 Can you please share the Indian number plates pretrained weights. Thanks

quantamc commented 6 years ago

Great work... been palying around with it but i get this error

Traceback (most recent call last): File "train.py", line 270, in <module> initial_weights=initial_weights) File "train.py", line 235, in train test_xs, test_ys = unzip(list(read_data("test/*.png"))[:50]) File "train.py", line 69, in read_data yield im, code_to_vec(p, code) File "train.py", line 59, in code_to_vec c = numpy.vstack([char_to_vec(c) for c in code]) File "train.py", line 59, in <listcomp> c = numpy.vstack([char_to_vec(c) for c in code]) File "train.py", line 56, in char_to_vec y[common.CHARS.index(c)] = 1.0 ValueError: substring not found

KaushikNathMIT commented 6 years ago

@sourabh2k15 Can you please share the wieghts.npz for indian license plate

sourabh2k15 commented 6 years ago

hello, I no longer have those trained weights but I do have one thing to say. After you put all the hardwork, make the code work and get the weights this thing is incredibly slow. It takes a good 1 minute for recognition and that it never is 100% accurate, reason being the fancy fonts used in Indian number plates.

If you use opencv's default russian number plate cascade classifier to crop out the number plate part, resize into various scales and then apply these weights it still takes about 10 seconds which is not good enough.

The solution we went for in our final project was to write our own plate extraction ( localization algorithm ) and then pass it to tesseract ocr which does the job with okayish performance for good images ( contrast brightness etc )

https://github.com/sourabh2k15/ANPR

ashish1405 commented 6 years ago

Thanks sourabh, for your inputs. But seeing the scenario for indian number plates(with hand painted/fancy fonts) will it not be advisable to use neural network like TF rather than Tesseract to do ocr?

BTW I checked the github code, in that you are saving possibleplate, but that variable is no were assigned.

sourabh2k15 commented 6 years ago

well it would be advisable to use TF or any other ML stuff if you get hold of a good dataset containing labelled indian license plate images. There isn't a ready one available afaik and the one randomly generated by gen.py isn't perfect for that task

ashish1405 commented 6 years ago

That is true, we need huge data set (more than 1 lac). Also another problem is AFAIK deep-anpr would not be able to do ocr for two line number plates (e.g. rickshaw). The only thing I've found so far is to use the FSNS model of TF. But to generate dataset for that is also a problem, coz it is using TFRecords.

chingjunehao commented 6 years ago

Hi guys, is there anyone fixed the problem of WARNING:tensorflow:From C:\Users\User\Anaconda3\envs\py35\lib\site-packages\tensorflow\python\util\tf_should_use.py:107: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02. Instructions for updating: Use tf.global_variables_initializer instead. 2018-01-06 15:21:41.717437: I C:\tf_jenkins\home\workspace\rel-win\M\windows\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2 Traceback (most recent call last): File "./train.py", line 266, in initial_weights=initial_weights) File "./train.py", line 232, in train test_xs, test_ys = unzip(list(read_data("test/*.png"))[:50]) File "./train.py", line 68, in read_data yield im, code_to_vec(p, code) File "./train.py", line 58, in code_to_vec c = numpy.vstack([char_to_vec(c) for c in code]) File "./train.py", line 58, in c = numpy.vstack([char_to_vec(c) for c in code]) File "./train.py", line 55, in char_to_vec y[common.CHARS.index(c)] = 1.0 ValueError: substring not found

Seems like there's no one reply regarding the thread of this problem...

IamSierraCharlie commented 6 years ago

Hi guys, is there anyone fixed the problem of WARNING:tensorflow:From C:\Users\User\Anaconda3\envs\py35\lib\site-packages\tensorflow\python\util\tf_should_use.py:107: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02. Instructions for updating: Use tf.global_variables_initializer instead. 2018-01-06 15:21:41.717437: I C:\tf_jenkins\home\workspace\rel-win\M\windows\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2 Traceback (most recent call last): File "./train.py", line 266, in initial_weights=initial_weights) File "./train.py", line 232, in train test_xs, test_ys = unzip(list(read_data("test/*.png"))[:50]) File "./train.py", line 68, in read_data yield im, code_to_vec(p, code) File "./train.py", line 58, in code_to_vec c = numpy.vstack([char_to_vec(c) for c in code]) File "./train.py", line 58, in c = numpy.vstack([char_to_vec(c) for c in code]) File "./train.py", line 55, in char_to_vec y[common.CHARS.index(c)] = 1.0 ValueError: substring not found

Seems like there's no one reply regarding the thread of this problem...

Have a look at the function 'def read_data()' function in train.py

I changed mine from this: def read_data(img_glob): for fname in sorted(glob.glob(img_glob)): im = cv2.imread(fname)[:, :, 0].astype(numpy.float32) / 255. code = fname.split("/")[1][9:16] p = fname.split("/")[1][17] == '1' yield im, code_to_vec(p, code)

To this: def read_data(img_glob): for fname in sorted(glob.glob(img_glob)): # fname is filename - absolute path print('fname', fname) im = cv2.imread(fname)[:, :, 0].astype(numpy.float32) / 255. # read the image as im this is read code = fname.split("/")[0][14:21] print('code is', code) p = fname.split("/")[0][22] == '1' print('pp', p) yield im, code_to_vec(p, code)

the numbers in the line code = ..... and p = ..... refer the letters/ numbers in your file name for the test data. I left the print statements in there on purpose so one might get an understanding for what the code returns to the variable. I am not an expert in python, but these changes worked for me running anaconda on windows 10.

Apologies the code indenting is not right - I'm not familiar with how to fix that on this website. I'll aim to do better in future.