[BUG] Camera Node setSize() not set Resolution and ISP Scale as expected

[borongyuan]

Describe the bug I want color camera to output 640×360 low resolution video. So I use setSize(640, 360) in Camera Node. I expected it to automatically set resolution to THE_1080_P and ISP Scale numerator to 1 and denominator to 3. But the actual obtained image resolution is 1352×1012. It seems that it calculates the ISP Scale correctly but uses a resolution of THE_12_MP.

If I setSize(1280, 720), I end up with 1352×1012 as well. Can't seem to get lower resolution images. If there are problems with the automatic configuration of setSize(), I hope you can continue to provide manual configuration methods like those in ColorCamera Node.

[moratom]

Thanks for the bug report @borongyuan.

We will look into the issue, please share also which device are you using and the depthai version?

[borongyuan]

We will look into the issue, please share also which device are you using and the depthai version?

I have multiple devices, including OAK-D-S2, OAK-D-Pro-W, and OAK-D-Pro-W-PoE. Their main cameras are all IMX378, and they all have this issue. I’m using depthai v2.24.0. I'm using v2.24.0, but this issue should have been there since the Camera Node was added.

[zrezke]

Thanks for the report, at the moment the camera node only finds the sensor resolution that fits best to the specified size, no ISP scaling is done. The tricky thing here is that the user could specify an illegal size which can't be achieved on the isp output. We will come up with a solution / behaviour for this.

In the end I think we need to implement the logic as in this table: https://docs.google.com/spreadsheets/d/153yTstShkJqsPbkPOQjsVRmM8ZO3A6sCqm7uayGF-EE/edit#gid=0

With a way to let the user know what the actual size of the isp output will be, before running the pipeline.

[borongyuan]

Thank you for this information and I'm looking forward to future updates.

Sachin suggested me to use generic Camera Node as I need its rectification feature. But I noticed that it doesn't actually automatically undistort images with small FOV. I need to setMeshSource(dai::CameraProperties::WarpMeshSource::CALIBRATION) explicitly. This part is not mentioned in the documentation, but it seems to be consistent with the handling of useHomographyRectification() in StereoDepth Node. Image will be automatically undistorted only when FOV is over 85°. I'm not sure why it's handled this way by default. Is it related to the previous calibration method? In my opinion, images should be undistorted by default in most applications, otherwise it will reduce the depth measurement accuracy and affect depth-color alignment. I just received a product that used your new calibration method recently, and it seems that the depth measurement accuracy is much better than before.

In addition, the document says that the default preview output is RGB. NeuralNetwork generally requires planar BGR input. Camera Node also lacks methods such as setColorOrder() and setInterleaved().

Please also check these issues. Should we track them here or open new issues for them?

[zrezke]

You are correct. For cameras with FOV < 85° the image will not be distorted automatically. For now we will just mention this behaviour in the docs: https://github.com/luxonis/depthai-python/pull/962.

On the preview output type: The Camera node is worked on atm to include methods: setPreviewType(RawImgFrame::Type t) and setVideoType(RawImgFrame::Type t) where you will be able to specify "any" type, so you will easily be able to set BGR, RGB, planar, interleaved... other..

[zrezke]

Also as a note: the Camera node lacks setters for color order and interleaved/planar, but you can edit the properties if need may be. For the time being you can use cameraNode->properties.inteleaved = false; and cameraNode->properties.colorOrder = dai::CameraProperties::ColorOrder::BGR in order to get planar BGR for example.

[borongyuan]

@zrezke Thanks for the note. I also tested that ISP scale can be enabled via setVideoSize() and configuring ispScale in properties. But I'm having a new problem, that is, once the rgb image is undistort in Camera Node, an error will occur when using YoloSpatialDetectionNetwork.

[19443010A106A22E00] [1.5] [1.444] [system] [critical] Fatal error. Please report to developers. Log: 'ImgPreproc' '562'

[zrezke]

Ah, I think we've hit a similar bug before. I think the automatic undistortion may be a little broken on Camera node. Can you please open a new issue with a MRE so we can take a look?

[diablodale]

device-side scaling has been borked for 2+ years. related bugs https://github.com/luxonis/depthai-core/issues/320 https://github.com/luxonis/depthai-core/issues/418

[borongyuan]

Sorry for the late reply. This is easy to reproduce. Just add the following line to any of your nn examples. cam->setMeshSource(dai::CameraProperties::WarpMeshSource::CALIBRATION);

[VojtaBTL]

Hi, are there any updates to this topic? I used the newest version of depthai-core 2.28.0, but did not find any table specification of possible resolutions for Camera node. My problem is I need to use undistortion, which Camera node is capable of, but needs to also specify some resolution, but I do not know, from which I can choose of.

Thanks for the report, at the moment the camera node only finds the sensor resolution that fits best to the specified size, no ISP scaling is done. The tricky thing here is that the user could specify an illegal size which can't be achieved on the isp output. We will come up with a solution / behaviour for this.

In the end I think we need to implement the logic as in this table: https://docs.google.com/spreadsheets/d/153yTstShkJqsPbkPOQjsVRmM8ZO3A6sCqm7uayGF-EE/edit#gid=0

With a way to let the user know what the actual size of the isp output will be, before running the pipeline.