Save Images to Server

[gadget-man]

Hi,

Is there a way to save the image sent to PlateRecognizer (showing the numberplate detected), either when a plate is detected, or optionally on every detection request (even if no plate recognised)? There is a similar feature on https://github.com/robmarkcole/HASS-plate-recognizer but frigate_plate_recognizer seems much more efficient (and quick) so I'm not really using the HA version anymore, but really missing having a folder with all the saved plate data on it!

[DrSpaldo]

Can there be a consideration to appending the registration to the end of the file?

Maybe if it’s not too hard having file naming option/s?

[ljmerza]

I was hesitant to go down the rabbit hole of configuring the file name so kept it static.

[kyle4269]

@gadget-man After thinking about this for a while.. I looked at the response data from CP.AI and below is what I have.

{
   "confidence":0.8886001706123352,
   "label":"Plate: 4260",
   "plate":"4260",
   "x_min":1217,
   "y_min":501,   "x_max":1392,
   "y_max":618
}

CP.AI already has the coordinates of the plate. I changed your code to:

    vehicle = (
        x_min,
        y_min,
        x_max,
        y_max
    )

Here is the result:

Plate Recognizer has a similar response.. https://guides.platerecognizer.com/docs/snapshot/api-reference/#read-number-plates-from-an-image

EDIT: Saving the image in get_plate, the bounding box seems to be 99% accurate with my testing.

    img_bytes = requests.get(f"{snapshot_url}").content
    img = Image.open(io.BytesIO(img_bytes))

    if config.get('plate_recognizer'):
        plate_number, plate_score = plate_recognizer(snapshot)
        img.save("/config/latest.png")
    elif config.get('code_project'):
        plate_number, plate_score = code_project(snapshot)
        img.save("/config/latest.png")
    else:
        _LOGGER.error("Plate Recognizer is not configured")
        return None, None

[DrSpaldo]

449 snapshots yesterday! Might need to come up with a better filing system than per month.

Many are cropped and are missing the plate, would be great to get the full shot...

[kyle4269]

449 snapshots yesterday! Might need to come up with a better filing system than per month.

Many are cropped and are missing the plate, would be great to get the full shot...

Wow, you're getting a good amount of plates! I put in a PR for this. You'll need to add "crop_image: false" to your config under frigate once this is approved.

[gadget-man]

I've been working on this, and by repeating the call for a snapshot and then immediately using the API to get the event data, the box dimensions are MUCH more accurate. I've also changed it so that it saves a full snapshot, not just the crop. Will push a PR for this shortly. @kyle4269 I think this would negate the need to have a separate flag for crop_image - because the box dimensions published by frigate are for the full snapshot (not the crop), we're going back to saving the full snapshot.

[kyle4269]

I've been working on this, and by repeating the call for a snapshot and then immediately using the API to get the event data, the box dimensions are MUCH more accurate. I've also changed it so that it saves a full snapshot, not just the crop. Will push a PR for this shortly. @kyle4269 I think this would negate the need to have a separate flag for crop_image - because the box dimensions published by frigate are for the full snapshot (not the crop), we're going back to saving the full snapshot.

Heck of a job man!! I'm gonna try to find some time this weekend to test.

[kyle4269]

@gadget-man Did a little testing with your 1.8.9 build. See the below image, It's a good distance off from the actual plate.

[gadget-man]

Yeah, I was getting an almost perfect match. Maybe I was just lucky! From the logs, can you see time time difference between getting the updated snapshot and drawing the box?

[gadget-man]

I've had another thought - plate_recognizer also returns box dimensions (not sure if CodeProject.ai also does the same?). Perhaps we can use those to draw the box, rather than the data from Frigate? The issue would be the crop - unless we can reliably get the dimensions used to create the crop, we'd need to send the full image to platerecognizer to get consistent co-ordinates back....

[kyle4269]

See my post https://github.com/ljmerza/frigate_plate_recognizer/issues/21#issuecomment-1897644310 about using that. I think if we use CPAI and Plate Recognizer coordinates after the image was processed, we'd then have exact coordinates for the plate.

[gadget-man]

See my post #21 (comment) about using that. I think if we use CPAI and Plate Recognizer coordinates after the image was processed, we'd then have exact coordinates for the plate.

Sorry I'd totally missed that! Had you changed the code to send the un-cropped image for recognition in your updates?

[kyle4269]

All good man! Yeah, I don't send the images cropped. But thinking about it.. It shouldn't matter if we send cropped images or not. If we aren't using Frigate coordinates. We send the image to be processed and receive the coordinates for either CPAI or Plate Recognizer. Those should be exact to whatever we send. I think?

[gadget-man]

Yes, but if you send a cropped image for processing, if you choose to save the snapshot that would have to be cropped as well. I'm not sure why the original code only sends the cropped image - presumably for speed of processing / accuracy improvements? Personally, I'd prefer to have the final snapshot that is saved as the uncropped version.

[kyle4269]

That's what I figured cropping it was for. I do like having the uncropped version also. What about if the snapshot is saved then sent, instead of sending it directly from Frigate? In theory we'd be 100% accurate with the coordinates on every image. Wouldn't matter them if its cropped or not.. My only concern would on a busy street, would Plate Recognizer be able to keep up?

[kyle4269]

I turned on Plate Recognizer to get the response data.

{
   "processing_time":52.266,
   "results":[
      {
         "box":{
            "xmin":1454,
            "ymin":277,
            "xmax":1553,
            "ymax":322
         },
         "plate":"4269",
         "region":{
            "code":"us-ri",
            "score":0.103
         },
         "score":0.9,
         "candidates":[
            {
               "score":0.9,
               "plate":"4269"
            }
         ],
         "dscore":0.86,
         "vehicle":{
            "score":0.392,
            "type":"Big Truck",
            "box":{
               "xmin":1182,
               "ymin":1,
               "xmax":1912,
               "ymax":431
            }
         }
      }
   ],
   "filename":"1805_APZEs_upload.jpg",
   "version":1,
   "camera_id":"None",
   "timestamp":"2024-01-19T18:05:14.280348Z"
}

[gadget-man]

Ok I've got some feedback from Nick over at Frigate. We might be better using latest.jpg for the identified camera, rather than snapshot.jpg.

More work required! I'll try and find time to have another play at this over the weekend.

[gadget-man]

Based on the feedback back from Nick, I think the best approach would be as follows:

1. On mqtt message being received, grab the latest image using GET /api/<camera_name>/latest.jpg (note this returns a full size image, no option to get it cropped).
2. Based on the box dimensions of the object (or plate) create our own crop and pass this to plate_recognizer / CP.AI.
3. If a plate is returned, save the original image with the box dimensions of the plate from plate_recognizer / CP.AI overlaid. Note because we've created our own crop and only have 1 original image from Frigate, we can work out how to transpose the coordinates from the cropped image back onto the full size image and save it.

Bit of re-working the current code, but sounds like it might be the best solution. As a starting point, I'd probably try and run a parallel routine to just grab latest.jpg and save the Frigate 'box' coordinates onto it, just to see how close they are.

Any thoughts?

[gadget-man]

@kyle4269 looking at my snapshots from today, I've not had very many, but every one has been an EXACT match between the red box we've drawn and the original plate. Can I just check: are you using a test script/video for the examples above, and if so are you sure that the resolution of the saved clip matches the resolution of the 'detect' camera in your Frigate configuration? I had an issue the other day where I downloaded a clip to use for testing, forgetting that my clips are saved at a higher resolution than the detect stream, resulting in all my boxes being out of alignment....

[kyle4269]

@kyle4269 looking at my snapshots from today, I've not had very many, but every one has been an EXACT match between the red box we've drawn and the original plate. Can I just check: are you using a test script/video for the examples above, and if so are you sure that the resolution of the saved clip matches the resolution of the 'detect' camera in your Frigate configuration? I had an issue the other day where I downloaded a clip to use for testing, forgetting that my clips are saved at a higher resolution than the detect stream, resulting in all my boxes being out of alignment....

I am using a clone from your GitHub for testing. The clip im using is a frigate event clip of a vehicle coming into the driveway. Also for the "test" camera config I used was exactly what I have for my LPR. So I guess it's possible that when downloading the clip from frigate it plays with the resolution.

Edit: Just checked the file and it's the same resolution

[gadget-man]

OK how strange. And your -detect and -record streams in your Frigate config.yml are the same stream? It would be very interesting to know if you have the same issue on your 'real' camera.

[kyle4269]

OK how strange. And your -detect and -record streams in your Frigate config.yml are the same stream? It would be very interesting to know if you have the same issue on your 'real' camera.

That's correct. My record and detect are both on my main stream. I will try it on my real camera tonight. Have to wait for the kid to go to bed because I need to go drive up and down a few time haha

[kyle4269]

@gadget-man Here is an example from my real camera. I also noticed the plates folder has a bunch of the same snapshots seconds apart.

[gadget-man]

I’ve had a couple of detections today and they are a perfect match. Yours always seem to be offset by a very similar amount which is also a bit strange. I’m still suspicious that it’s a resolution issue. What resolution have you specified for Detect in your Frigate config.yml?

[kyle4269]

Running at 1920x1080. It's very odd. If everyone else is getting perfect matches, it's a me issue haha. Im perfectly fine changing the code on new releases to grab the coordinates from CPAI. So don't keep banging away at the keyboard for me. I appreciate all the work you are putting in. Wish I was as knowledgeable as you in python. Learning a lot from you and Leonardo.

[gadget-man]

Can you paste the extract from your Frigate config.yml for that camera (obfuscate any passwords etc)?

[kyle4269]

  LPR:
    ffmpeg:
      inputs:
      - path: rtsp://554/cam/realmonitor?channel=1&subtype=1
        roles:
        - audio
      - path: rtsp://554/cam/realmonitor?channel=1&subtype=0
        roles:
        - record
        - detect
      output_args:
        record: preset-record-generic-audio-aac
    detect:
      width: 1920
      height: 1080
      fps: 15
      stationary:
        interval: 10
    objects:
      track:
      - person
      - car
      - face
      - license_plate
         # - amazon
         # - ups
         # - fedex
      filters:
       car:
         min_score: .50
         threshold: .80
       license_plate:
         min_score: .6
    motion:
      threshold: 85
      improve_contrast: true
      contour_area: 30
    zones:
      Lp:
        coordinates: 0,382,0,1080,1920,1080,1920,400
        objects:
        - car
        - person
        - face
        - license_plate
    snapshots:
      enabled: true
      clean_copy: true
      bounding_box: false
      crop: false
      required_zones:
      - Lp
    mqtt:
      bounding_box: false
      crop: false
      height: 1080
      quality: 100
    record:
      enabled: true
      events:
        required_zones:
        - Lp

[gadget-man]

Doesn’t really make sense does it? In my latest code, if you add a line to log image_width and image_height, does it correctly return 1920 and 1080? And if you interrogate the saved snapshot jpg file, has it also saved at those dimensions?

[gadget-man]

Could you DM me your test .mp4 file so that I can play with it here? I’ll set up a test camera using your settings and see if I can work it out!

[kyle4269]

Doesn't make sense at all. It does say 1920x1080 for my snapshot. I'll put it on my google drive and send you a link when I get home. Where do you want me to dm it?

[gadget-man]

If you're happy to share it, just past the public google drive link here. Otherwise, you can email me at sales at iparcelbox dot com

[gadget-man]

Hi @kyle4269, so I just tried using the mp4 file you emailed me and the Frigate config you posted above (the only change being that I changed the input to the following for test purposes: ` inputs:

• path: /config/TundraFrontDay.mp4 input_args: -re -stream_loop -1 -fflags +genpts ` The plate was detected successfully, and this is the file that was saved by frigate_plate_recognizer:

I allowed the script to repeat quite a few times - each time the coordinates were very slightly different, but each time right smack bang in the middle of the plate.

Please can you check you're using the very latest version of the script from my forked repo, and try creating a test camera in Frigate replicating the above, to see if you can get similar results?

Please could you also check and confirm what version of Frigate you are currently running?

[kyle4269]

I'm using 0.13.0-rc1 and am using your latest fork. I will give it a try again and see my results. It looks like you only changed the stream loop to 1?

[gadget-man]

I'm on 0.13.0-c35c7da. Let me also update to rc1 just to make sure that doesn't make a difference...!

Update: Confirmed all good with Frigate rc1. Plate boxes and detection are absolutely spot on.

[kyle4269]

@gadget-man Can confirm that it's working!

[gadget-man]

Excellent! Did you have to change any settings to get it working? If so, perhaps there is something we could add to the readme to help others?

[kyle4269]

I didn't change anything other than in my testing camera 0 to -1.. But that doesn't matter for my main camera.

[gadget-man]

Ok great. Glad it's working at least! Next mini-project is to try and implement some kind of 'watched plates' function.

[kyle4269]

Ok great. Glad it's working at least! Next mini-project is to try and implement some kind of 'watched plates' function.

I like that! Can have it send a mqtt message if the plate matches. This was HA can grab that and send out notifications?

[gadget-man]

Ok great. Glad it's working at least! Next mini-project is to try and implement some kind of 'watched plates' function.

I like that! Can have it send a mqtt message if the plate matches. This was HA can grab that and send out notifications?

I haven't really thought too much about it yet. I know Platerecognizer returns 'candidate' plates as well as the best match, not sure of CP.AI does the same. So we could test those against a list of watched plates. Or do some kind of fuzzy matching. If a matched plate is found, it could either overwrite the recognised plate with the match, or perhaps add an additional tag to the existing MQTT message with the watched plate.

[kyle4269]

I looked into the API for CP.AI and I don't see any reference to possible matches.. I do have a clip of a plate that rarely comes back to the correct plate.. So when/if you get to that point of testing matching, I can give you that clip.

[gadget-man]

In 1.8.12, I'm finding that the red bounding box with the recognised plate is not longer appearing. I do see a small red dot in the top left corner.

Looking at the code and your latest PR @ljmerza, I note that you converted the initial dimensions to int() before multiplying by the image width / height. I suspect this may be the cause - the returned value is expected to be between 0 and 1 so this may be the issue?

e.g. dimension_1 = int(final_attribute[0]['box'][0]) << this is almost certainly returning either 0 or 1.

I'm not at home to test it, but I think the int() on these 4 lines was perhaps added in error as part of your review of the PR?

[ljmerza]

ah youre right was trying to make sense of the math when making tests. fixed in 1.8.13