Problems with the use of OAK-FFC-4P

[YaoYaoYuWuQi]

I have recently purchased the OAK-FFC-4P product but I am having some problems using it and would like to ask for advice. When I am using Depthai-viewer for display, I found that two of the cameras are displaying normally, but the images of the remaining two cameras keep on having coloured grid lines as shown in the picture, I don't know how to proceed to solve this problem.

When I reduce the resolution of the image, the display image is normal, but at 1200P screen there are colourful streaks, I don't know how to fix it.

[zrezke]

Hmm - @YaoYaoYuWuQi would you mind running cam_test.py and posting the images you get out of it, so we can see if it's a viewer or depthai issue.

To try cam_test.py:

• Clone https://github.com/luxonis/depthai-python
• Install requirements: python3 examples/install_requirements.py
• Run cam test: python3 utilities/cam_test.py -cams cama,c camd,c -cres 1200

[jakaskerl]

Adding link to discussion as well: https://discuss.luxonis.com/d/3519-problems-with-the-use-of-oak-ffc-4p/3

[zrezke]

Thanks @jakaskerl, ok so if it works on cam test it's likely a viewer issue then. @YaoYaoYuWuQi Regarding the low FPS from the discussion, it's because the default settings of the viewer run stereo depth adn an AI model too, both require a lot of compute, therefore the fps drops, if you disable the AI model and stereo in the configuration panel to the right of the screen, you should be able to hit ~60fps. Edit: Answering from question regarding exposure time in discussion: depthai-viewer does not support setting manual exposure time (You may find cam_test.py more useful in this case)

[YaoYaoYuWuQi]

I want to use the four cameras of the OAK-FFC-4P for color video display, with exposure control for the video, and depth estimation for each of the two cameras to get a depth map. How should I make the modification.

[zrezke]

@YaoYaoYuWuQi You could try to use the default-cam-test branch on depthai-python to run cam_test.py with the --stereo option to get the stereo depth output stream too, lmk if it works. Edit: branch name default-cam-test

[YaoYaoYuWuQi]

Could you tell me more about how to do this, I can't find the option.

[zrezke]

@YaoYaoYuWuQi sure, no problem. Here are the steps:

• Change directory into depthai-python
• Run: git pull
• Run: git fetch
• Run: git checkout default-cam-test

• Now you should be able to run cam_test.py with the stereo option like so:

  • python3 utilities/cam_test.py -cams cama,c camd,c --stereo That should display camera streams from CAM_A and CAM_D, and also a colorized disparity output.

  You may also want to look at some examples in the depthai-python/examples/StereoDepth and modify one of those to fit your needs.

[YaoYaoYuWuQi]

Sorry, I am having problems running your steps and need your help. When I run git checkout default-cam-test, I have the following error reporting problem. error: pathspec 'default-cam-test' did not match any file(s) known to git. when I run python3 utilities/cam_test.py -cams cama,c camd,c --stereo the system does not react, I don't get the result of the run. When I ran cam_test.py before, it was a downloaded python file that ran in python to get 4 coloured video results.

[zrezke]

@YaoYaoYuWuQi sorry, we just merged default-cam-test into develop yesterday (and deleted the default-cam-test branch) These are the new steps now:

• Run: git checkout develop
• Run: git pull Now you should be able to run cam_test.py with the stereo option like so: python3 utilities/cam_test.py -cams cama,c camd,c -stereo

[YaoYaoYuWuQi]

Sorry, there is still a problem, the error message is error: pathspec 'develop' did not match any file(s) known to git.

[zrezke]

oh right, I think you will first need to run cd depthai-python

[YaoYaoYuWuQi]

Thank you for pointing out the problem, I made the correction. However, when I run python3 utilities/cam_test.py -cams cama,c camd,c -stereo, the system does not respond.

[zrezke]

Perhaps try to do git pull, and then run again.. I've also seen problems with the git bash terminal on windows where cam_test.py just hangs, maybe you can try to use the windows CMD terminal instead if the cam_test still doesn't work after git pull

[YaoYaoYuWuQi]

Typed git pull into the Git Bash software but cam_test still didn't work so I ran it in cmd but could only get colour video for both cama and camd cameras, no stereo results.

[zrezke]

Can you see a print in the terminal, that the StereoDepth node could not be created? The default cam test relies on the presence of stereoRectificationData.leftCameraSocket in the calibration:

left, right = eeprom.stereoRectificationData.leftCameraSocket, eeprom.stereoRectificationData.rightCameraSocket

So if this is missing in the calibration the stereo node will not be created...

print("Couldn't create stereo depth node. Device has invalid calibration.")

This is the print you will probably see..

You could try to modify line 389 in depthai-python/utilities/cam_test.py to hardcode CAM_A, CAM_D as left and right cameras... perhaps it will work then.

[YaoYaoYuWuQi]

I did get the error "Couldn't create stereo depth node. Device has invalid calibration." but I had my setup calibrated by the service person once before I bought it, so do I need to re-calibrate it again? 389 lines of code is saturation = 0 is that where the change is?

[zrezke]

No need to recalibrate, the modification to cam_test.py will likely solve your problem.

            left, right = eeprom.stereoRectificationData.leftCameraSocket, eeprom.stereoRectificationData.rightCameraSocket

This is the line you should change to:

left, right = dai.CameraBoardSocket.CAM_A, dai.CameraBoardSocket.CAM_D

Perhaps you are looking at the wrong file? For me the line 389 in depthai-python/utilities/cam_test.py looks like this:

[YaoYaoYuWuQi]

Because I am using 4 colour camera model AR0234, his resolution is 1920*1200, so when I change the code to left, right = dai.CameraBoardSocket.CAM_A, dai.CameraBoardSocket.CAM_D, the program still doesn't get the depth map. The cmd prompts Couldn't create depth: can't create stereo depth with left cam width > 1280. Use --isp-downscale to downscale the image. When I try to downscale the resolution, run python utilities/cam_test.py -cams cama,c camd,c -cres 1080, cmd prompts Unsupported resolution set for detected camera AR0234, needs 1200_P, defaulting to it. Do I need to do isp-downscale?

[YaoYaoYuWuQi]

I still have a question, is it not possible to run the cam_test file directly in python, I see that the program will prompt AttributeError: module 'depthai' has no attribute 'ColorCameraProperties', is it necessary to add the formal parameter to run?

[YaoYaoYuWuQi]

Because I bought 4 lenses, two of them are LN001 and the other two are LN003, so I want to display two depth maps, but I see that I can only display the depth maps obtained from CAM_A and CAM_D.

[jakaskerl]

@YaoYaoYuWuQi Modular devices like FFC-4P need recalibration every time the camera setup changes (either in camera sensors or their positions). I would recommend recalibrating the device using the guide here.

But first try this:

Make sure you have checked out to the correct branch (of depthai-python) and ran:

• git submodule update --init --recursive and
• git pull --recurse-submodules then
• python3 examples/install_requirements.py

Then proceed with cam_test and if needed, do the recalibration.

[YaoYaoYuWuQi]

I can get the depth map by lowering the isp-downscale. Is the code not applicable to AR0234?

[YaoYaoYuWuQi]

I want to recalibrate afterwards, and I want to get two depth maps first. How should I make the changes?

[zrezke]

@jakaskerl we tried this locally and it worked right?

[jakaskerl]

Tried AR0234 on viewer and experienced no issue. But I did not calibrate the device and thus could not view depth. Perhaps the issue arises when depth is also being calculated. Hard to say for sure, but the streams looked OK. @zrezke

WRT double depth, you'd have to calibrate each pair separately, then the device should correctly combine the two when all 4 sockets are filled.

[YaoYaoYuWuQi]

I would like to ask if it is possible to display 4 RGB videos after distortion correction in CAM_test and how to call the relevant nodes. I tried to run it in rgb_undistort.py but I can only see one video after correction, how can I display 4 color videos?

[zrezke]

@YaoYaoYuWuQi Probably the easiest way to achieve that would be to use Camera node.

You can take a look at the example here: https://docs.luxonis.com/projects/api/en/latest/samples/Camera/camera_undistort/?highlight=undistort

The code will be largely the same for each camera, copy paste it then change the camera sockets and stream names.

[YaoYaoYuWuQi]

When I run this code, I find that I don't notice a very noticeable effect of distortion correction, is this correct?

[zrezke]

hmm, doesn't look right, you should try to explicitly set the dewarp mesh. The camera node should do it automatically for cameras with hfov > 85°, but to be on the safe side, it's better to set explicitly...

cam_node.setMeshSource(dai.CameraProperties.WarpMeshSource.CALIBRATION)

[YaoYaoYuWuQi]

I have encountered a problem. When I show 4 distortion corrected color cameras, I find that a fixed orb-like object appears in the image of one of the videos, making the resulting image problematic. How should this be fixed?

[YaoYaoYuWuQi]

I still want to display the distortion corrected image in cam-test, so I tried to modify it in cam_test, but it didn't work. There are two main problems, firstly it can't display the live image continuously, it stops displaying after displaying a few frames of the image. The second problem is that one of the videos has small chunks that can't be removed, as shown in my last message. I'll list the changes I made and you can see them. Line 336, change as follows.

Line 479, modify to q[c] = device.getOutputQueue(name="video "+c, maxSize=4, blocking=False) Line 567, modify to cv2.imshow(q[c].get().getCvFrame())

[YaoYaoYuWuQi]

If you see and have a relevant solution, your response would be greatly appreciated.

[zrezke]

Hmm, I do not see anything relevant, perhaps you could try to modify the rgb_undistort.py example you were working with previously to include all cameras.. Otherwise please open a ticket on https://discuss.luxonis.com/ if you have some more issues, (to not get too far of the viewer related issue here)