Someone who have good ideas on detecting two layers license plate?

[TyroneLi]

Some times, we have to be capable of detecting and recogniting two layers car license plate.This method does a good job on single layer car plate?But when it comes to two layer, it does badly.So are there anybody tackling this issue using deep learning method?

[TyroneLi]

Hi~Could you release the second network training code?the Wpod_net training code? @sergiomsilva

[sergiomsilva]

@TyroneLi I already released, see train-detector.py

[TyroneLi]

@sergiomsilva Hi~Is that possible that I adjust the last ocr-network output size from 30x10 to 15x10, then I do transfer learning using fine-tune.So that the ocr-network can segment and recognize two layers ocr plate?

[ThorKing01]

@TyroneLi I also test the issue about two layers or two lines license palte. In the theory of YOLO, it's possible detect the two layers license plate, but it does badly.

[sergiomsilva]

Have you tried to just change the input resolution of the OCR net in the .cfg file? For a single layer, it is 240x80, but maybe for two layers something like 160x160 could do the job, you will just need to manage the order of the detections.

[ThorKing01]

@sergiomsilva I have tried to test the two layers or two rows license plate. The method could well detect the vechile plate and well recognize the second layer, bud it did badly on the first layer. And the phenomenon may be caused by the few test data and the input resolution. I will try both of them. Thank you！

[TyroneLi]

@sergiomsilva Yeap,it may work and I will try. @ThorKing01 Hi~how do you adjust the ocr-net?Just change the input resolution from 240x80 to 240x40 or something like that?Have you re-train your ocr-net in order to detect and recognize two layer plate?

[simonpunk]

@sergiomsilva Hi, thanks for this great open source project. I have some issues detecting two layers license plate too, I have a sample like this:

|------MU-----| |----12-34----|

I have change the height=80 to 160 in ocr-net.cfg, it is able to detect the plate with two layers. However, the ocr result turns out to 12MU34, so I think it is not scanning from top layer the bottom layers but from left to right instead. May I have some suggests of how to fine tune the ocr config for this issue? Many thanks.

[TyroneLi]

@sergiomsilva Hi, thanks for this great open source project. I have some issues detecting two layers license plate too, I have a sample like this:

|------MU-----| |----12-34----|

I have change the height=80 to 160 in ocr-net.cfg, it is able to detect the plate with two layers. However, the ocr result turns out to 12MU34, so I think it is not scanning from top layer the bottom layers but from left to right instead. May I have some suggests of how to fine tune the ocr config for this issue? Many thanks.

Hi~Can you share some training details in training ocr-net to detect two layer plate?Just adjust the last output layer height from 80 to 160 and then fine tune?Plus which dataset do you use?

[simonpunk]

@TyroneLi I haven't try to train the ocr yet, the pre-trained model seems to produce a good result already, but changing the height from 80 to 160 in ocr-net.cfg and also the license-plate-detection.py with this line Llp,LlpImgs,_ = detect_lp(wpod_net,im2single(Ivehicle),bound_dim,2**4,(240,160),lp_threshold) will slightly improve the success rate for two layer plate. And I have figured out how to rearrange the ocr output into a proper order for two layer plate, in license-plate-ocr.py L = dknet_label_conversion(R,width,height) L = nms(L,.45) here L will contain a list of coordinates of the ocr output, so you can try your own way to sort the order by the Y axis first, and then the X axis.

[TyroneLi]

@TyroneLi I haven't try to train the ocr yet, the pre-trained model seems to produce a good result already, but changing the height from 80 to 160 in ocr-net.cfg and also the license-plate-detection.py with this line Llp,LlpImgs,_ = detect_lp(wpod_net,im2single(Ivehicle),bound_dim,2**4,(240,160),lp_threshold) will slightly improve the success rate for two layer plate. And I have figured out how to rearrange the ocr output into a proper order for two layer plate, in license-plate-ocr.py L = dknet_label_conversion(R,width,height) L = nms(L,.45) here L will contain a list of coordinates of the ocr output, so you can try your own way to sort the order by the Y axis first, and then the X axis.

I got it.But merely adjusting the height from 80 to 160 in ocr-net.cfg and the corresponding code seems help slightly,since the the original model was trained in 80 from scratch.And I want to re-train the ocr-net for two layer plate,but I have some troubles in collecting such two layers plate dataset.As the original code released by author suppose each letter and digit has the same region area.In order to detect one layer plate and two layers plate together,I am trying to use FPN in yolov3 to attack this issue.But haven't try it cause lack of data....And I think to re-order your two layer plate's letter or digit, you can also try to judge them by x-axis and y-axis value.

[PhilipsKoshy]

I created some two-line LPs with a small OpenCV program. And, I also collected some real two-line LPs. With the small changes suggested by Simonpunk above, and sorting the results vertically and then horizontally, I find it detects OpenCV generated good-looking twoline plates. But, it fails with some characters of real twoline plates. So, I guess the issue is - the slight change in shape of the characters (because of the reorganized dimensions of the LP of the two-line plates), is never learnt by the OCR. So, perhaps we have to retrain the OCR for whatever new LP dimension we want. Any thoughts?

[IbrahimBond]

you guys can just use the darknet framework to retrain the OCR. just use the same cfg file included in this repository. you can even use the same weights in this repository and perform transfer learning on them.

[RohitMidha23]

@PhilipsKoshy can you share the code snippet you used for two line detection?

[PhilipsKoshy]

you guys can just use the darknet framework to retrain the OCR. just use the same cfg file included in this repository. you can even use the same weights in this repository and perform transfer learning on them.

Theoretically, yes, we can use the darknet framework. But, I had to abandon the work due to the following reason. I think the darknet training {1. expects a very very large number of input data set 2. does online data-augmentation, which can mess up the data}. I collected about 400 real two line plates, and I generated another 400 plates. I tried offline data-augmentation by some translation, rotation, noising etc. Even after exploding my data set, I fell short of the required numbers to even kick-off training. The amount of effort to prepare such a very huge data set is enormous. Then, I kind of felt that, perhaps, YOLOv2 which is used for OCR, is known to be good for large object detection and probably not the best for small character recognition. Still looking for a good character recognizing solution. I tried tesseract, and I found it's good only for printed text images (e.g. a scanned pdf document), and terrible for license plates.

[Xiaotong713986]

you guys can just use the darknet framework to retrain the OCR. just use the same cfg file included in this repository. you can even use the same weights in this repository and perform transfer learning on them.

Theoretically, yes, we can use the darknet framework. But, I had to abandon the work due to the following reason. I think the darknet training {1. expects a very very large number of input data set 2. does online data-augmentation, which can mess up the data}. I collected about 400 real two line plates, and I generated another 400 plates. I tried offline data-augmentation by some translation, rotation, noising etc. Even after exploding my data set, I fell short of the required numbers to even kick-off training. The amount of effort to prepare such a very huge data set is enormous. Then, I kind of felt that, perhaps, this OCR (YOLOv2) is meant for large object recognition and probably not the best for small character recognition. Still looking for a good character recognizing solution. I tried tesseract, and it's good only for pdf images.

Hi，what do you mean that tesseract is good only for pdf image? Is it perform bad for jpg image?

[PhilipsKoshy]

you guys can just use the darknet framework to retrain the OCR. just use the same cfg file included in this repository. you can even use the same weights in this repository and perform transfer learning on them.

Theoretically, yes, we can use the darknet framework. But, I had to abandon the work due to the following reason. I think the darknet training {1. expects a very very large number of input data set 2. does online data-augmentation, which can mess up the data}. I collected about 400 real two line plates, and I generated another 400 plates. I tried offline data-augmentation by some translation, rotation, noising etc. Even after exploding my data set, I fell short of the required numbers to even kick-off training. The amount of effort to prepare such a very huge data set is enormous. Then, I kind of felt that, perhaps, this OCR (YOLOv2) is meant for large object recognition and probably not the best for small character recognition. Still looking for a good character recognizing solution. I tried tesseract, and it's good only for pdf images.

Hi，what do you mean that tesseract is good only for pdf image? Is it perform bad for jpg image?

Thanks for pointing out, I edited my response.

[linnnff]

@TyroneLi我还没有尝试训练ocr，预训练的模型似乎已经产生了很好的结果，但是在ocr-net.cfg以及license-plate-detection.py中将高度从80更改为160。用这条线 Llp,LlpImgs,_ = detect_lp(wpod_net,im2single(Ivehicle),bound_dim,2**4,(240,160),lp_threshold) 会稍微提高两层板的成功率。而且我已经弄清楚了如何将ocr输出重新排列为两层板的正确顺序，在license-plate-ocr.py中 L = dknet_label_conversion(R,width,height) L = nms(L,.45) ，L将包含ocr输出的坐标列表，因此您可以尝试使用自己的方式进行排序首先是Y轴，然后是X轴。

you idear is wrong,its fool.

[Djvnit]

@TyroneLi我还没有尝试训练ocr，预训练的模型似乎已经产生了很好的结果，但是在ocr-net.cfg以及license-plate-detection.py中将高度从80更改为160。用这条线 Llp,LlpImgs,_ = detect_lp(wpod_net,im2single(Ivehicle),bound_dim,2**4,(240,160),lp_threshold) 会稍微提高两层板的成功率。而且我已经弄清楚了如何将ocr输出重新排列为两层板的正确顺序，在license-plate-ocr.py中 L = dknet_label_conversion(R,width,height) L = nms(L,.45) ，L将包含ocr输出的坐标列表，因此您可以尝试使用自己的方式进行排序首先是Y轴，然后是X轴。

you idear is wrong,its fool.

Dear if you have any idea that works better please help us.

[thangdc94]

I tried to implement formatter as @simonpunk 's answer with a simple algorithm.

I create 2 arrays: 1 for 1st line and the other for 2nd line. I separate 2 lines by comparing top left of all characters with the highest character. You guys can try it.

def format_pl(L):
    l_highest = min(L, key=lambda x: x.tl()[1])
    first_line = []
    second_line = []
    y_threshold = (l_highest.bl()[1] - l_highest.tl()[1]) / 2
    for l in L:
        if l.tl()[1] - l_highest.tl()[1] < y_threshold:
            first_line.append(l)
        else:
            second_line.append(l)
    first_line.sort(key=lambda x: x.tl()[0])
    second_line.sort(key=lambda x: x.tl()[0])    
    license_plate_str = "".join([chr(i.cl()) for i in first_line])
    if len(second_line) > 0:  # if license plate has 2 lines
        license_plate_str += "-" + "".join([chr(i.cl()) for i in second_line])
    return license_plate_str

Then call format_pl() after these lines

...
L = dknet_label_conversion(R,width,height)
L = nms(L,.45)

# Format license plate
format_pl(L)
...

[Djvnit]

I too used the same logic and solved the problem a way back. I had forgot about the issue raised here else I'd have proposed my solution here just by today's I reminded. (Thanks though!!!) Happy Learning.