Shadow Effect on Depth Stream (D410 connected through network)

[saulcarvalho]

Required Info
Camera Model	D410
Firmware Version	2.53.1
Operating System & Version	Win 11
Platform	PC
SDK Version	2.53.1
Language	Python 3.11.9 (MINGW64)
Segment	Robot

Issue Description

I am accessing color and depth streams through GStreamer and I have noticed my depth stream has a shadow effect on the left side of every object. I know this is probably the result of baseline differences between the sensors and I need to perform a calibration.

Thing is, I can't use Intel Software to update firmwave, nor to see streams with the Viewer , nor reconfigure the sensors, because the network functionality has been disabled and the camera is integrated inside a Kinova Gen3 robot. I can't use pyrealsense2 either.

The only thing that I have available are functions from Kinova's API to set/get intrinsics and extrinsics, but for some reason the set functions do not seem to much for some reason ...

Any idea how to approach this problem?

My GStreamer pipelines are the following:

color_pipeline_str = (
        "rtspsrc location=rtsp://192.168.8.10/color latency=30 ! "
        "rtph264depay ! avdec_h264 ! videoconvert ! "
        "video/x-raw,format=BGR ! appsink name=color_sink"
    )
    depth_pipeline_str = (
        "rtspsrc location=rtsp://192.168.8.10/depth latency=30 ! "
        "rtpgstdepay ! videoconvert ! "
        "video/x-raw,format=GRAY16_LE ! appsink name=depth_sink"
    )

Additional info: The left side in black corresponds to the invalid depth band while the side on the right is an object that I have near the robot, below the camera's minimum Z value, so just ignore those.

[MartyG-RealSense]

Hi @saulcarvalho I note that you have a previous Kinova Gen3 case at https://github.com/IntelRealSense/librealsense/issues/12643

Do you have another computer that you can update the firmware version on, please? The firmware is stored in the camera hardware and not on the computer, so when the firmware is updated it is saved in the camera and carried over to whatever machine the camera is plugged into.

---

In regard to the black silhouette around the body section, as you correctly mentioned it is due to the positional difference between sensors and is called occlusion. Human body parts such as arms and hands are especially likely to have this silhoutte. Calibration may help a little but there is not much else that can be done to reduce the occlusion. The RealSense Viewer automatically applies 'occlusion invalidation' to the 3D pointcloud to reduce the impact of occlusion, though you mention that you cannot access the camera via the Viewer due to the network being disabled.

Moving the camera closer to the body part can cause the size of the shadow to shrink but this would not be practical if you need to capture an entire body part area within the camera's view.

[saulcarvalho]

At the moment I can't do much more than accessing the streams directly or by using functions from:

https://github.com/Kinovarobotics/Kinova-kortex2_Gen3_G3L/blob/master/api_python/doc/markdown/summary_pages/VisionConfig.md

Ideally I would like to update the firmware on the camera and eventually try the software. Since the camera can only be connected to via network, I tried to build the SDK from source from version 2.53.1, the most recent version with the network functionality available. Unfortunately, didn't have success doing this, using either CMake or MSBuild for building the source code.

[MartyG-RealSense]

The librealsense networking feature requires connecting to the computer / robot that has the camera attached to it from another computer using an SSH connection. Would you be able to connect to the robot with SSH if networking on the robot is disabled?

Did you disable networking on the Kinova Gen3 or did the robot come with networking disabled by default? The instructions below from the Kinova Gen3 user manual describe how to enable networking.

[MartyG-RealSense]

Hi @saulcarvalho Do you require further assistance with this case, please? Thanks!