On-Device 180-Degree Image Rotation (i.e. Camera Op/Down Flip) for megaAI/OAK-1

[Luxonis-Brandon]

Start with the why:

In some physical installations it is advantages (or necessary) to install megaAI upside-down, primarily because with the default settings, the image is upside down when the device is oriented such that the cable is facing down.

In these cases, the resultant output image needs to be 'flipped' (i.e. rotated 180 degrees). This can be done via cv2.flip on the host, but doing so adds unnecessary host-burden as since megaAI/DepthAI have direct access to the image sensors settings, it is possible to reconfigure the image sensors such that there is a 'no-compute-cost' 180-degree rotation of the images coming off of the image sensors.

And such a host solution is not possible for embedded applications, where the host (which could be, for example, and ATTiny8) is not capable of doing such operations.

And even more importantly, for operations that occur on-device (such as object detection, facial-landmark detections, etc.) often the network performance degrades (or is completely non-functional) if the input image is 'upside down' of how the network was trained.

Move to the how:

Implement sensor-level image flipping (i.e. 180 rotation) by reconfiguring the sensor output format and expose this capability through the DepthAI API.

Move to the what:

Provide a no-compute-load capability for the images coming off the image sensors to be rotated 180-degrees to allow for 'upside-down' installation of megaAI.

Make this the default setting for megaAI / OAK-1 in Gen2, such that the USB port facing downwards results in a right-side-up image.

Note that this issue is only for megaAI / OAK-1. If you would like this for DepthAI / OAK-D, please do leave a comment!

[Luxonis-Brandon]

From internal notes:

We could configure this directly in the camera sensor. It's straightforward for Mono (1 or 2 register sets), but will require tweaking homography matrix for StereoDepth. And for Color we need also to change the Bayer order in ISP.

And as a note, we should definitely implement this for megaAI/OAK-1 only to start (i.e. no grayscale/homography support) as the only requests we have gotten for this feature is for megaAI / OAK-1 since the color sensor is effective upside down on that unit.

[Luxonis-Brandon]

I think this is working as of today. Checking with the team on it.

[Luxonis-Brandon]

Discuss.luxonis.com comment on this for reference later: https://discuss.luxonis.com/d/71-upside-down-image-from-oak-1-camera

[alex-luxonis]

Done for Gen1 on https://github.com/luxonis/depthai/pull/297

Also done for Gen2, but needs first this CI pipeline to finish: https://github.com/luxonis/depthai-python/runs/1654428842 When done, the depthai library including the fix can be installed with: python3 -m pip install depthai==0.0.2.1+a20c2ded10e920367834c5c4d3cedf502ff6ec07 --extra-index-url https://artifacts.luxonis.com/artifactory/luxonis-python-snapshot-local/

[Luxonis-Brandon]

Thanks!

[chris-piekarski]

So to confirm, the default orientation was changed but the ability to configure the final install orientation is still pending?

[alex-luxonis]

@chris-piekarski On Gen1 we only changed the default orientation for OAK-1 / BW1093.

But on Gen2 it is configurable: https://github.com/luxonis/depthai-shared/pull/15

To rotate 180 degrees from the default, use this API:

cam_rgb = pipeline.createColorCamera()
...
cam_rgb.setImageOrientation(dai.CameraImageOrientation.NORMAL)  # For OAK-1, as AUTO defaults to ROTATE_180_DEG
cam_rgb.setImageOrientation(dai.CameraImageOrientation.ROTATE_180_DEG)  # For OAK-D

[chris-piekarski]

Got it. Thanks @alex-luxonis.

[rexn8r]

cam_rgb.setImageOrientation method should not only be restricted to 180 but should have been extended to 90 and 270 degree as well.

In real world solution, the camera might need to be mounted at 90/270 degree as well. For example, OAK-D is to be part of a custom digital display hardware where it is to be placed inside the custom bezel area/section of the screen either on left side or right side and the screen orientation is potrait. The width of bezel wouldn't allow camera to be fixed at 0 degree and has to be rotated.

I have been fighting since last 3-4 months to implement 270 degree (-90) rotation with 3stage NN with ImageManip rotation methods and struggling to build a script which is stable and accurate.

Would it not have made life easy if 90/270 degree rotation options were also implemented

Just asking as I am a newbie

[alex-luxonis]

@rexn8r For this setting we directly configure the camera sensor readout order, which exposes flags like horizontal-mirror and vertical-flip (and when both are combined, the resulting image is 180-degree rotated). These are achieved at no extra computation cost. But options to rotate by 90 or 270 degrees are generally not available in sensor hardware. https://docs.luxonis.com/projects/api/en/latest/references/python/?#depthai.CameraImageOrientation

ImageManip should allow to rotate by any angle, it uses the Warp hardware block of the VPU. Please let us know what issues you have encountered, and try also with the latest depthai release at the moment 2.17.0.0 - ImageManip was refactored there and potential stability issues may have been fixed.

API and examples that showcase ImageManip rotation:

• setWarpTransformFourPoints as in https://github.com/luxonis/depthai-python/blob/675891e/examples/ImageManip/rgb_rotate_warp.py#L49-L51
• setCropRotatedRect as in https://github.com/luxonis/depthai-python/blob/e96993e/examples/ImageManip/image_manip_rotate.py

[rexn8r]

Hi @alex-luxonis

https://github.com/luxonis/depthai-experiments/issues/377#issuecomment-1186491461

If you look at the pastebin link in the last post, i modified gen2 age gender demo to rotate cam at -90 degree, but there is no detection

Thanks

[Erol444]

I also wanted to mention that the upcoming SDK refactor will expose a simple API for changing the camera orientation (90/270 degree rotation).

[samialperen]

@alex-luxonis is there a way to rotate 180 degrees from the ROS side? I could not find a ros parameter for CameraImageOrientation https://docs.luxonis.com/projects/api/en/latest/references/python/#depthai.CameraImageOrientation.HORIZONTAL_MIRROR

[saching13]

@samialperen The current examples don't have camera Orientation Parameterized. We are working on a new one which contains it. For a quick change, it is best to add a line in the pipeline here to the required orientation. But note that you might also need to change the camera Info.

[samialperen]

@saching13 Thanks a lot man! If you don't mind, could you give a full example how to do that just to be sure? I am using Oak-D POE version.

[saching13]

@samialperen which launch file are you using ? and which camera do you want to mirror ?

[samialperen]

@saching13 I am using stereo_inertial_node.launch. My goal is to mount the camera upside down so that I will have easy access to the PoE switch from the top, so I want to mirror all the cameras. As far as I understand, if I do 180-degree mirroring, it is not gonna cost extra computational power.

[saching13]

@samialperen it is more complicated than that. When you flip the device upside down. Left Camera becomes right and right camera becomes left. So it requires swapping the cameras on the pipeline and intrinsics needs to be changed which we don't have it in the fw. So at this moment if you still want to do it. It requires

1. Added the Orientation Change
2. Changing the connection to ste reo.
3. Recalibrating with inverted mode. I wouldn't suggest this. Rather flip the depth map or disparity map on the host using opencv.

We will add this feature to handle internally on the fw sometime soon.

[samialperen]

@saching13 Oh I see. It seems far more complicated than I imagined. I will wait for the update from your side then. How can I stay updated on the newest fw for oak-d? Do you guys have a roadmap?

[saching13]

We have releases here Which will contains new features and fw upgrades.

[samialperen]

@saching13 Thank you very much, man! Highly appreciated.