Poor quality streams and depth measurement R200/F200 legacy Python UP2

[AdamMiltonBarker]

---

Required Info
Camera Model	R200
Firmware Version	(Open RealSense Viewer --> Click info)
Operating System & Version	Linux (Ubuntu 18.04)
Kernel Version (Linux Only)	4.15
Platform	UP2
SDK Version	legacy
Language	python }
Segment	others

Issue Description

Hi guys I am doing a couple of legacy projects to make use of my R200 and F200. The projects do facial recognition and calculate the distance using depth.

On the F200 the RGB streams are not amazing but can detect my face about a meter away, however I cannot get a measurement unless I am about 3-6cms away from the camera. When using in Skype on Windows the RGB stream is a million times better and same with the depth stream,

On the R200, the RGB stream is terrible on Linux, really bitty and it is very hard for it to detect my face, the depth measurements don't work at all when it does recognize my face. On Windows, the IR and depth streams are really bad, but the RGB stream is a lot better.

I am figuring as they work a lot better on Windows, that it must be possible to fine tune them but I have played around with the device options for the cameras and am getting no where. Could you suggest the best settings to get these cameras working at least like they are on Windows, if I can get that quality on the UP2 it will be a lot better.

TIA.

[MartyG-RealSense]

Hi @AdamMiltonBarker I would not honestly recommend using librealsense1 (Legacy Librealsense) for facial landmark detection as it was not designed to support that feature. At the time that this SDK was in common use, dedicated facial detection and recognition features were provided by Intel's RealSense SDK For Windows, also known as the '2016 SDK'. The '2016 R2' version of this is compatible with F200 and R200.

For face processing applications for RealSense users who could not use the Windows SDK, use of third-party libraries such as dlib was recommended instead. So I would recommend exploring whether dlib can provide better results if you are not using dlib already.

https://github.com/IntelRealSense/librealsense/issues/456

Some historical background: around the time of 2016-2017, the Windows SDK was Intel's official closed-source RealSense SDK and the original librealsense was an open-source SDK. There was also a third Linux-based Intel SDK for the ZR300 stereo camera model called 'Intel RealSense SDK For Linux' that added a range of extra features not available in librealsense1. When the 400 Series camera generation debuted, librealsense (now known as librealsense2 / RealSense SDK 2.0) became the single official SDK.

[AdamMiltonBarker]

Hi mate thanks for the reply. I don't use the librealsense for the actual facial recognition or identification I am using dlib/Facenet and NCS1. The issue I am trying to solve is the very low quality RGB stream and the really bad depth measurement.

[MartyG-RealSense]

I get the sense that the cameras are being treated by dlib as ordinary RGB webcams rather than using RealSense depth functions. Your mention of the very short range of detection for F200 reminds me of using a crude detection technique in 2015 called blob tracking with the F200. You had to virtually press your face or hand up against the camera to get it to detect and track the features when using that mode, whereas dedicated RealSense hand joint and face landmark functions could detect from a meter away.

If the cameras are operating in terms of RGB webcams instead of RealSense cameras, that may provide some insight about why there is a quality difference between Windows and Linux.

[AdamMiltonBarker]

Hi mate, Dlib is not used with the depth stream, the raw stream without anything is red as in the photo, on both cameras even before I add the facial rec in (which only uses the RGB stream) they are terrible, you have to be within 3-6 cm on the F200 to get any depth. I took a photo the other day where there was nothing at all but writing the depth frames to screen:

This is also the same if I dont merge the two streams into one image.

I will try break it down in simpler terms mate.

UP2 without additional processing, out of the box grab the frames and display with opencv:

• F200 without any additional processing at all, depth stream is bright red (First photo), you have to be within 3-6cms before the sensors pick you up.
• F200 without any additional processing at all, RGB stream is pretty bitty (First photo), not the best quality but a lot better than R200.
• R200 without any processing at all depth stream is bright red with dots everywhere, doesn't pic me up sat in front of it, hand or face has to be very very close.
• R200 without any additional processing at all, RGB stream is terrible, as in the image above.

UP2 with facial recognition (dlib) and identification (Facenet/NCS1) only on RGB stream, coordinates come from RGB stream and are used to index into the depth camera readings:

• F200, depth stream is bright red (First photo), you have to be with 3-6cms before the sensors pick you up.
• F200, RGB stream is pretty bitty (First photo), not the best quality but a lot better than R200.
• R200 depth stream is bright red with dots everywhere, doesn't pic me up sat in front of it, hand or face has to be very very close, although you will see some shapes on the red screen, cannot get measurements always 0.0.
• R200, RGB stream is terrible, as in the image above.

In short I can survive with the F200 I was just looking to improve it as when I plug both cameras into windows and use them in skype they are a million times better. Code for F200 is here: https://github.com/AdamMiltonBarker/TassAI/tree/master/UP2/NCS1/Realsense/F200

R200 is the major problem, for reasons I mentioned above. Hope this helps mate.

[MartyG-RealSense]

I tested the RGB of the R200 and F200 in the Windows Camera app to compare the quality to your own images. The first image is R200 and the second is F200. They seem to have about the same quality and are similar to your facial image in the opening comment

The link below demonstrates a typical depth map quality that can be expected from an R200 when used with OpenCV:

https://answers.opencv.org/question/167445/r200-depth-stream-on-opencv/?answer=167458

Something may therefore be incorrect in the code that is being used in your project to capture the depth frame in OpenCV.

[AdamMiltonBarker]

Hi thanks, the code form the hand is literally the provided code from the example, there is no difference between the hand example and the version I have integrated the facial recognition and identification in (regarding quality) . Also when using in Windows they are pretty much perfect, I will take screen shots now of that. Thanks for the replies will check your links out, but the point I was trying to make was that the issues exist out of the box with both cameras.

[AdamMiltonBarker]

So the F200 is undoubtedly a lot better, with both depth and RGB on Windows.

The R200 mostly has not improved on Windows, but the RGB stream is a million times better, again, if I were to use the example code on the UP2 with both cameras I would experience the same issues. Are there any specific options that can be set to improve the issues I am having ?

[AdamMiltonBarker]

Just to mention this is the code used in the hand example above: https://pyrealsense.readthedocs.io/en/master/examples.html

[MartyG-RealSense]

I went over your project again carefully but I cannot see much potential for improving the results. The R200 does have a calibrator app for Windows though (the F200 does not have an equivalent app), so you may be able to achieve some improvement if you use it on your R200.

https://downloadcenter.intel.com/download/24958/Intel-RealSense-Camera-Calibrator-for-Windows-

[AdamMiltonBarker]

Hi thanks for the reply. So the bitty images shown in the first post and the requirement to be within 6cms of both cameras to detect depth are expected results ? Doesn't feel right to me especially when the cameras are better in Windows.

[MartyG-RealSense]

The R200 is nearly six years old now so its hardware capabilities and supported resolution modes are very limited. Having said that, it is certainly capable of better depth images, like the one that I linked to earlier, and it is capable of depth sensing up to 4 meters range (the F200 depth-senses over about 1 meter range).

In Windows there is a Virtual Driver that enables the camera to be accessed by non-RealSense applications, such as Skype. I am not certain how access is handled on Linux, but I recall it may be based on V4L2 / Video4Linux. If that is the case, that may account for some of the difference in rendering between Windows and Linux.

[AdamMiltonBarker]

OK thanks I will leave the R200 out of the projects. It was just to make use of them really. Thanks for the help Marty.