Cropping and dynamic cropping

[ItayNaumann]

Is your feature request related to a problem? Please describe. There are certain regions within the image that tags won't apear, and attempting to detect them in those areas adversely impacts performance efficiency.

Describe the solution you'd like The ability to crop the image, preferably dynamic with nt entries for both x and y crop axis

[mcm001]

I think this wouldn't be terribly hard to do in software using a submatrix of the overall camera image. Not sure about in hardware via libcamera. From an image processing perspective, this just means transforming pixels from cropped image space into full sensor region space. I'm not sure how much complexity this would add to do elegantly, but I look forward to an implementation proposal

[ItayNaumann]

We don't want to change the hardware frame because it's not deterministic, and it may change the aspect ratio. We only want to change the region of interest for the detection, which is the most time-consuming part.

[mcm001]

Yep, I was referring to scaler_crop. Not sure how the pi isp handles this but it might be worth investigating

[Alextopher]

I'm interested in picking this up.

[Alextopher]

Here is my vision for implementing this:

• Add a 'crop' tab (pipe) to each pipeline.
• Crop parameters are x, y, width, and height as expected.
  • These parameters be set via sliders in the client or nt topics.
• An additional boolean parameter "fallback". Where if an object (colored shape, tag, object, etc) isn't found in the crop it will retry the detection over the entire image.
• Add some kind of crop hits & misses reporting. Maybe just 1 bit in pipeline result?

[mcm001]

So is dynamic cropping in response to targets being seen a hard requirement? Or how do teams expect to need to use this?

[Alextopher]

I think there is value in static cropping and dynamic cropping.

I think most teams could use static cropping today if we shipped it. You could remove a portion of the frame from calculations (say your robot bumpers covers a good chunk of your FOV).

Dynamic cropping opens up a wider space of strategies. From something as relatively simple as "have a NxM box follow the previous result" to "use pose estimates to predict the tag's location in frame".

I think dynamic cropping is fairly valuable.

[Alextopher]

My vision:

1. A static crop (yellow box). Pixels outside of which are never evaluated.
2. A dynamic crop (red box) that can be controlled via robot code or the webui.
3. A checkbox such that when checked, whenever a dynamic crop fails to find a target PV will immediately retry with the static crop

[ItayNaumann]

My vision:

1. A static crop (yellow box) that is never evaluated
2. A dynamic crop (red box) that is evaluated and can be changed via robot code
3. A checkbox such that when checked, whenever a dynamic crop fails to find a target PV will immediately retry with the static crop

I think that this solution takes the best of both worlds and lets the most customization for each team