search

ljmerza / frigate_plate_recognizer

Identify license plates via Plate Recognizer and add them as sublabels to Frigate
106 stars 13 forks source link

Watched Plates #30

Closed gadget-man closed 5 months ago

gadget-man commented 6 months ago

Hi all,

I'm working on a new PR to allow a list of 'watched plates' to be added to the config. When a plate is recognised by either PlateRecognizer or CP.AI, the code would then compare the recognised plate against the list of watched_plate to see if there's a close match (firstly based on the candidates/predictions returned by the engine, and if still no match then by doing some fuzzy / difflib matching to allow for minor differences.

I'd appreciate feedback on the best way to return this information once a match is found.

Should I replace the recognised plate with the matched plate in the sub-label and matt response (and save_image filename), perhaps with some additional info added to MQTT e.g. is_watched = true, original_plate = XXXXX and fuzzy_score = 0.9 - this option would mean that the sub-label shown in Frigate would be the 'corrected' plate, and the same in the save_image filename.

Alternatively, would people prefer it if I leave the 'recognised' plate as-is, but add an additional tag to mqtt to also identify the associated watched_plate (which could then be picked up by HA automations, even though the sub-label in Frigate would be the originally recognized (i.e. wrong) plate?

I propose to add the minimum fuzzy score into the config so that people could tailor the level of accuracy they are prepared to accept based on criticality / confidence.

kyle4269 commented 6 months ago

If fuzzy is as accurate as I've been reading. I say replace the recognized plate with the correct fuzzy one. I would definitely like to see the fuzzy threshold in the config, with lpr every setup is different and requires specific tweaking. Would be nice to have a description with the watched plates

Ex:

watched:
  - plate: 123
  - description: package thief
gadget-man commented 6 months ago

If fuzzy is as accurate as I've been reading. I say replace the recognized plate with the correct fuzzy one. I would definitely like to see the fuzzy threshold in the config, with lpr every setup is different and requires specific tweaking. Would be nice to have a description with the watched plates

Ex:

watched:
  - plate: 123
  - description: package thief

Please could you share the response json from a CP.AI successful recognition so that I can check I'm referencing the candidates correctly in the script?

kyle4269 commented 6 months ago

Looking at the API, I don't think CP.AI has candidates. 

{
   "confidence":0.8886001706123352,
   "label":"Plate: 4260",
   "plate":"4260",
   "x_min":1217,
   "y_min":501,   
   "x_max":1392,
   "y_max":618
}
gadget-man commented 6 months ago

I think the code is just picking the first prediction in the list. If you set debugging to DEBUG, what do you get in the logs for response?

kyle4269 commented 6 months ago

I think the code is just picking the first prediction in the list. If you set debugging to DEBUG, what do you get in the logs for response?

If you're looking to grab the plate number.. This is what I use. If not, give me like 20 min and i'll get the whole thing

    plate_x_min = response['predictions'][0].get('x_min')
    plate_y_min = response['predictions'][0].get('y_min')
    plate_x_max = response['predictions'][0].get('x_max')
    plate_y_max = response['predictions'][0].get('y_max')
gadget-man commented 6 months ago

No I want the whole response - what you are returning is index[0] of response['predictions'] which I think is the top scoring plate. What I want to see are all the other predictions - we will check those first for a watched plate, and if no joy, will then resort to fuzzy matching as a last resort.

kyle4269 commented 6 months ago

oops sorry! Did paste the wrong thing.. But here is the response

"response":{
   "success":true,
   "predictions":[
      {
         "confidence":0.8632612228393555,
         "label":"Plate: 4269",
         "plate":"4269",
         "x_min":1230,
         "y_min":500,
         "x_max":1394,
         "y_max":605
      }
   ],
   "message":"Found Plate: 4269",
   "processMs":180,
   "inferenceMs":166,
   "moduleId":"ALPR",
   "moduleName":"License Plate Reader",
   "code":200,
   "command":"alpr",
   "executionProvider":"CPU",
   "canUseGPU":false,
   "analysisRoundTripMs":207,
   "processedBy":"localhost"
}
gadget-man commented 6 months ago

oops sorry! Did paste the wrong thing.. But here is the response

"response":{
   "success":true,
   "predictions":[
      {
         "confidence":0.8632612228393555,
         "label":"Plate: 4269",
         "plate":"4269",
         "x_min":1230,
         "y_min":500,
         "x_max":1394,
         "y_max":605
      }
   ],
   "message":"Found Plate: 4269",
   "processMs":180,
   "inferenceMs":166,
   "moduleId":"ALPR",
   "moduleName":"License Plate Reader",
   "code":200,
   "command":"alpr",
   "executionProvider":"CPU",
   "canUseGPU":false,
   "analysisRoundTripMs":207,
   "processedBy":"localhost"
}

Ok great thanks. So in this example there was only 1 prediction, but it's an array so if CP.AI finds a few options, they would be returned here for processing. I'll include for that in the script.

gadget-man commented 6 months ago

OK, first half decent working version is ready for testing. Have a look at https://github.com/gadget-man/frigate_plate_recognizer/tree/watched-plates

Assuming a plate is recognised, firstly it will check to see if it's one of the matched plates. If it already is, no further work required.

Failing that, it will check to see if one of the other candidates returned by plate-recogniser / CP.AI matches a watched plate. If it does, then it will use that candidate instead of the one with the highest score.

If that doesn't work, then next it will test the recognized plate against each of the watched plates in turn using fuzzy matching. If the highest scoring match exceeds the threshold in config, it will then update the response with that value.

I've updated the readme.md with basic instructions, but haven't yet done anything with the tests (as I haven't worked out how they actually work yet!)

I've not done anything as yet to add a 'name' tag to the MQTT output if it's a watched plate - not really sure what you'd use that for?

Feel free to give it a test and let me know what error messages you get back.

kyle4269 commented 6 months ago

@gadget-man I like it! But got an error.

  watched_plates:
    -  4269
    -  DEF456
  fuzzy_match: 0.8
frigate_plate_test  | Traceback (most recent call last):
frigate_plate_test  |   File "/usr/src/app/./index.py", line 549, in <module>
frigate_plate_test  |     main()
frigate_plate_test  |   File "/usr/src/app/./index.py", line 545, in main
frigate_plate_test  |     run_mqtt_client()
frigate_plate_test  |   File "/usr/src/app/./index.py", line 504, in run_mqtt_client
frigate_plate_test  |     mqtt_client.loop_forever()
frigate_plate_test  |   File "/usr/local/lib/python3.9/site-packages/paho/mqtt/client.py", line 1756, in loop_forever
frigate_plate_test  |     rc = self._loop(timeout)
frigate_plate_test  |   File "/usr/local/lib/python3.9/site-packages/paho/mqtt/client.py", line 1164, in _loop
frigate_plate_test  |     rc = self.loop_read()
frigate_plate_test  |   File "/usr/local/lib/python3.9/site-packages/paho/mqtt/client.py", line 1556, in loop_read
frigate_plate_test  |     rc = self._packet_read()
frigate_plate_test  |   File "/usr/local/lib/python3.9/site-packages/paho/mqtt/client.py", line 2439, in _packet_read
frigate_plate_test  |     rc = self._packet_handle()
frigate_plate_test  |   File "/usr/local/lib/python3.9/site-packages/paho/mqtt/client.py", line 3033, in _packet_handle
frigate_plate_test  |     return self._handle_publish()
frigate_plate_test  |   File "/usr/local/lib/python3.9/site-packages/paho/mqtt/client.py", line 3327, in _handle_publish
frigate_plate_test  |     self._handle_on_message(message)
frigate_plate_test  |   File "/usr/local/lib/python3.9/site-packages/paho/mqtt/client.py", line 3570, in _handle_on_message
frigate_plate_test  |     on_message(self, self._userdata, message)
frigate_plate_test  |   File "/usr/src/app/./index.py", line 438, in on_message
frigate_plate_test  |     plate_number, plate_score, watched_plate, fuzzy_score = get_plate(snapshot, after_data)
frigate_plate_test  |   File "/usr/src/app/./index.py", line 382, in get_plate
frigate_plate_test  |     plate_number, plate_score, watched_plate, fuzzy_score = code_project(snapshot)
frigate_plate_test  |   File "/usr/src/app/./index.py", line 99, in code_project
frigate_plate_test  |     watched_plate, watched_score, fuzzy_score = check_watched_plates(plate_number, response['predictions'])   
frigate_plate_test  |   File "/usr/src/app/./index.py", line 146, in check_watched_plates
frigate_plate_test  |     config_watched_plates = [x.lower() for x in config_watched_plates] #make sure watched_plates are all lower case
frigate_plate_test  |   File "/usr/src/app/./index.py", line 146, in <listcomp>
frigate_plate_test  |     config_watched_plates = [x.lower() for x in config_watched_plates] #make sure watched_plates are all lower case
frigate_plate_test  | AttributeError: 'int' object has no attribute 'lower'
gadget-man commented 5 months ago

Ah - I forgot that python would treat number based plates as numbers not strings. I've corrected that now, please can you try again?

kyle4269 commented 5 months ago

Ah - I forgot that python would treat number based plates as numbers not strings. I've corrected that now, please can you try again?

Did a bunch of testing with a clip that comes back wrong 80% of the time.. The corrected plate is being set properly. Only because I'm OCD can we set the sublabel back to to UPPER? the plate is "EN 167" it's getting set as En167, spacing doesn't matter. Other than that I think its a GREAT beginning. I've also sent you a new link via email to the clip that comes back wrong for your testing if you'd like it.

gadget-man commented 5 months ago

I've now updated that. I've also added some code which seeks to overcome the bug that an entire event will be ignored if the top_score occurs prior to the object entering a zone monitored by frigate_plate_recognizer. Still needs quite a lot of tidying up, and I need to work out test.py works to make the updates but any interim feedback welcome!

kyle4269 commented 5 months ago

I've now updated that. I've also added some code which seeks to overcome the bug that an entire event will be ignored if the top_score occurs prior to the object entering a zone monitored by frigate_plate_recognizer. Still needs quite a lot of tidying up, and I need to work out test.py works to make the updates but any interim feedback welcome!

I have been watching your branch and have to say the fuzzy scoring is pretty accurate. I have a tough time with accurate plates at night because I have ZERO light in my driveway, which is about 250' long. My LPR camera is zoomed in to about the middle of the driveway and is 99% accurate during the day. So fuzzy has been correcting my matched plates very well. I like it!

For saved plates, I'd suggest only keeping x amount of days worth of images and deleting the oldest first. I'd help you with test.py, but it's very confusing to me.

ljmerza commented 5 months ago

let me know when the PR is settled and i can take a look at the tests when i get time

gadget-man commented 5 months ago

Thanks. I'll continue to monitor over the next few days to check it's not throwing out any funny responses, and will then prepare a PR.

gadget-man commented 5 months ago

For saved plates, I'd suggest only keeping x amount of days worth of images and deleting the oldest first.

@DrSpaldo proposed a script which moves snapshots into monthly folders. It should be straightforward to create a similar script that deletes snapshots older than a certain date. This should be run as a separate task, not part of the frigate_plate_recognizer script. https://github.com/ljmerza/frigate_plate_recognizer/issues/21#issuecomment-1893140046

kyle4269 commented 5 months ago

For saved plates, I'd suggest only keeping x amount of days worth of images and deleting the oldest first.

@DrSpaldo proposed a script which moves snapshots into monthly folders. It should be straightforward to create a similar script that deletes snapshots older than a certain date. This should be run as a separate task, not part of the frigate_plate_recognizer script. #21 (comment)

I actually created a pretty simple function that I can add in after.

kyle4269 commented 5 months ago

@gadget-man I noticed a bug while testing your latest commit. the vehicle plate reads "EN 167" In watched_plates its set to "- EN 167". Every time the plate is detected, it finds the match with fuzzy matching. 

frigate_plate_test  | 2024-01-30 20:02:14,494 - __main__ - DEBUG - Getting snapshot for event: 1706662934.139337-8bfi6c, Crop: True
frigate_plate_test  | 2024-01-30 20:02:14,494 - __main__ - DEBUG - event URL: http://192.168.1.32:5000/api/events/1706662934.139337-8bfi6c/snapshot.jpg
frigate_plate_test  | 2024-01-30 20:02:14,548 - __main__ - DEBUG - Getting plate for event: 1706662934.139337-8bfi6c
frigate_plate_test  | 2024-01-30 20:02:14,758 - __main__ - DEBUG - response: {'success': True, 'predictions': [{'confidence': 0.979057788848877, 'label': 'Plate: EN 167', 'plate': 'EN 167', 'x_min': 587, 'y_min': 946, 'x_max': 789, 'y_max': 1051}], 'message': 'Found Plate: EN 167', 'processMs': 176, 'inferenceMs': 154, 'moduleId': 'ALPR', 'moduleName': 'License Plate Reader', 'code': 200, 'command': 'alpr', 'executionProvider': 'CPU', 'canUseGPU': False, 'statusData': {'successfulInferences': 1085, 'failedInferences': 0, 'numInferences': 1085, 'numItemsFound': 254, 'averageInferenceMs': 101.87926267281107}, 'analysisRoundTripMs': 195, 'processedBy': 'localhost'}
frigate_plate_test  | 2024-01-30 20:02:14,758 - __main__ - DEBUG - No Watched Plates found from AI candidates
frigate_plate_test  | 2024-01-30 20:02:14,759 - __main__ - DEBUG - Best fuzzy_match: en 167 (1.0)
frigate_plate_test  | 2024-01-30 20:02:14,759 - __main__ - INFO - Watched plate found from fuzzy matching: en 167 with score 1.0
frigate_plate_test  | 2024-01-30 20:02:14,760 - __main__ - INFO - Storing plate number in database: en 167 with score: 0.979057788848877
frigate_plate_test  | 2024-01-30 20:02:14,771 - __main__ - DEBUG - sublabel: en 167
frigate_plate_test  | 2024-01-30 20:02:14,771 - __main__ - DEBUG - sublabel url: http://192.168.1.32:5000/api/events/1706662934.139337-8bfi6c/sub_label
frigate_plate_test  | 2024-01-30 20:02:14,779 - __main__ - INFO - Sublabel set successfully to: EN 167 with 97.9% confidence
frigate_plate_test  | 2024-01-30 20:02:14,780 - __main__ - DEBUG - Sending MQTT message: {'plate_number': 'EN 167', 'score': 0.979057788848877, 'frigate_event_id': '1706662934.139337-8bfi6c', 'camera_name': 'test', 'start_time': '2024-01-30 20:02:14', 'fuzzy_score': 1.0, 'original_plate': 'EN 167'}
frigate_plate_test  | 2024-01-30 20:02:14,786 - __main__ - DEBUG - Getting snapshot for event: 1706662934.139337-8bfi6c, Crop: False
frigate_plate_test  | 2024-01-30 20:02:14,786 - __main__ - DEBUG - event URL: http://192.168.1.32:5000/api/events/1706662934.139337-8bfi6c/snapshot.jpg
frigate_plate_test  | 2024-01-30 20:02:14,867 - __main__ - DEBUG - Drawing Plate Box: (586.0, 936.0, 773.0000000000001, 1050.0)
frigate_plate_test  | 2024-01-30 20:02:14,869 - __main__ - INFO - Saving image with path: /plates/EN 167_test_2024-01-30_20-02.png
gadget-man commented 5 months ago

Looks like it's a lowercase/uppercase issue - PlateRecognizer always returns matched plates in lowercase, and the script was designed around this. It looks as though CP.AI is uppercase (which quite frankly makes more sense, I don't know a single country in the world that uses lower case plates!). Code has been tweaked to correct for this.

kyle4269 commented 5 months ago

Thank you! I'll give it a test in a little bit.

kyle4269 commented 5 months ago

All good now!

gadget-man commented 5 months ago

I've been testing this for the last few days and everything seems to be working well. I'll submit as a PR. @ljmerza I'd be grateful if you could help with understanding of how the test script works / needs to be updated.